Paper presentation topics

1
Paper presentation topics
1. Segmentation
2. More on feature detection and descriptors
3. Shape and Matching
4. Indexing and Retrieval
5. More on 3D reconstruction
2
Depth from disparity
input image (1 of 2)
3
Real-time stereo
Nomad robot searches for meteorites in
Antartica http//www.frc.ri.cmu.edu/projects/meteo
robot/index.html

• Used for robot navigation (and other tasks)
• Several software-based real-time stereo
  techniques have been developed (most based on
  simple discrete search)

4
Stereo reconstruction pipeline

• Steps
• Calibrate cameras
• Rectify images
• Compute disparity
• Estimate depth

• Camera calibration errors
• Poor image resolution
• Occlusions
• Violations of brightness constancy (specular
  reflections)
• Large motions
• Low-contrast image regions

What will cause errors?
5

Spacetime Stereo
Li Zhang, Noah Snavely, Brian Curless, Steven
Seitz CVPR 2003, SIGGRAPH 2004
6

Stereo
7

Stereo
8
Marker-based Face Capture
The Polar Express, 2004
The largest intractable problem with The Polar
Express is that the motion-capture technology
used to create the human figures has resulted in
a film filled with creepily unlifelike beings.
New York Times Review, Nov 2004
9

Stereo
10
Stereo
Frame-by-Frame Stereo W?H 15?15 Window
A Pair of Videos 640?480_at_60fps Each
Inaccurate Jittering
11

Spacetime Stereo
3D Surface
12
Spacetime Stereo
3D Surface
Time
13
Spacetime Stereo
3D Surface
Time
14
Spacetime Stereo
3D Surface
Time
15
Spacetime Stereo
Surface Motion
Time
16
Spacetime Stereo
Surface Motion
Time0
17
Spacetime Stereo
Surface Motion
Time1
18
Spacetime Stereo
Surface Motion
Time2
19
Spacetime Stereo
Surface Motion
Time3
20
Spacetime Stereo
Surface Motion
Time4
21

Spacetime Stereo
Key ideas

• Matching Volumetric Window

• Affine Window Deformation

Surface Motion
Time
22

Spacetime Stereo
Time
23

Spacetime Stereo
Time
24

Spacetime Stereo
25
Spacetime Stereo
A Pair of Videos 640?480_at_60fps Each
Spacetime Stereo W?H?T 9?5?5 Window
26
Frame-by-Frame vs. Spacetime Stereo
Spacetime Stereo W?H?T 9?5?5 Window
Frame-by-Frame W?H 15?15 Window
Spatially More Accurate Temporally More Stable
27
Spacetime Face Capture System
Black White Cameras
Color Cameras
Video Projectors
28
System in Action
29
Input Videos (640?480, 60fps)
30
Spacetime Stereo Reconstruction
31
Creating a Face Database
32
Creating a Face Database
Zhang et al. SIGGRAPH04
33
Application 1 Expression Synthesis
A New Expression
Zhang et al. SIGGRAPH04
34
Application 2 Facial Animation
Zhang et al. SIGGRAPH04
35
Keyframe Animation
36
Some Applications
Entertainment Games Movies
Medical Practice Prosthetics
37
Some books on linear algebra
Linear Algebra, Serge Lang, 2004
Finite Dimensional Vector Spaces, Paul R. Halmos,
1947
Matrix Computation, Gene H. Golub, Charles F. Van
Loan, 1996
Linear Algebra and its Applications, Gilbert
Strang, 1988
38
Multiview Stereo
39
Choosing the stereo baseline
all of these points project to the same pair of
pixels
width of a pixel
Large Baseline
Small Baseline

• Whats the optimal baseline?
• Too small large depth error
• Too large difficult search problem

40
The Effect of Baseline on Depth Estimation
41
pixel matching score
1/z
42
(No Transcript)
43
Multibaseline Stereo

• Basic Approach
• Choose a reference view
• Use your favorite stereo algorithm BUT
• replace two-view SSD with SSD over all baselines
• Limitations
• Must choose a reference view (bad)
• Visibility!

44
MSR Image based Reality Project
http//research.microsoft.com/larryz/videoviewint
erpolation.htm