Stereo

1
Stereo

• Frank Dellaert
• Slides adapted from Jim Rehg

2
Stereo vision

• Triangulate on two images of the same point to
  recover depth.
• Feature matching across views
• Calibrated cameras

Left
Right
Matching correlation windows across scan lines
3
Outline

• Pin-hole camera model
• Basic stereo equations
• SSD image matching
• Basic stereo algorithm

4
Pinhole Camera Model
Image plane
Focal length f
Center of projection
5
Pinhole Camera Model
Image plane
Virtual Image
6
Basic Stereo Derivations
7
Basic Stereo Derivations
disparity
8
Stereo Vision
Z(x, y) is depth at pixel (x, y) d(x, y) is
disparity
Left
Right
Matching correlation windows across scan lines
9
Stereo Requirements

• Matching or scoring function
• Sum of squared (pixel) differences (SSD)
• Equivalent to normalized correlation
• Constraints
• Rectified images
• Match order constraint
• Search algorithm
• Dynamic programming

10
Stereo Correspondence

• Search over disparity to find correspondences
• Range of disparities to search over can change
  dramatically within a single image pair.

11
Correspondence Using Correlation
Left
Right
scanline
SSD error
disparity
Left
Right
12
Sum of Squared (Pixel) Differences
Left
Right
13
Image Normalization

• Even when the cameras are identical models, there
  can be differences in gain and sensitivity.
• The cameras do not see exactly the same surfaces,
  so their overall light levels can differ.
• For these reasons and more, it is a good idea to
  normalize the pixels in each window

14
Images as Vectors
Left
Right
Unwrap image to form vector, using raster scan
order
row 1
row 2
Each window is a vectorin an m2
dimensionalvector space.Normalization
makesthem unit length.
row 3
15
Image Metrics
(Normalized) Sum of Squared Differences
Normalized Correlation
16
Correspondence Using Correlation
Left
Disparity Map
Images courtesy of Point Grey Research
Left
Right
17
Stereo Results
Images courtesy of Point Grey Research
18
Epipolar Geometry

• The epipolar geometry is the fundamental
  constraint in stereo.
• Rectification aligns epipolar lines with scanlines

Epipolar plane
Epipolar line for p
Epipolar line for p
19
Correspondence

• It is fundamentally ambiguous, even with stereo
  constraints

Ordering constraint
and its failure
20
Stereo Correspondences
Left scanline
Right scanline
21
Stereo Correspondences
Left scanline
Right scanline
22
Search Over Correspondences
Left scanline
Right scanline
Disoccluded Pixels

• Three cases
• Sequential cost of match
• Occluded cost of no match
• Disoccluded cost of no match

23
Stereo Matching with Dynamic Programming
Left scanline
Start

• Dynamic programming yields the optimal path
  through grid. This is the best set of matches
  that satisfy the ordering constraint

Dis-occluded Pixels
Right scanline
End
24
Dynamic Programming

• Efficient algorithm for solving sequential
  decision (optimal path) problems.

1
1
1
1

2
2
2
2
3
3
3
3
How many paths through this trellis?
25
Dynamic Programming
1
1
1
2
2
2
States
3
3
3
Suppose cost can be decomposed into stages
26
Dynamic Programming
1
1
1
2
2
2
3
3
3
Principle of Optimality for an n-stage assignment
problem
27
Dynamic Programming
1
1
1
2
2
2
3
3
3
28
Memoization
Principle of Optimality for an n-stage assignment
problem

• -Code algorithm naively
• Just memoize C(t,j)
• Data structure is image

29
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
30
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
31
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
32
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
33
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
34
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
35
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
36
Computing Correspondence

• Another approach is to match edges rather than
  windows of pixels
• Which method is better?
• Edges tend to fail in dense texture (outdoors)
• Correlation tends to fail in smooth featureless
  areas

37
Computing Correspondences

• Both methods fail for smooth surfaces
• There is currently no good solution to the
  correspondence problem

38
Segmentation-based Stereo
39
Another Example