Stanford CS223B Computer Vision, Winter 200809 Lecture 7 Stereo

1
Stanford CS223B Computer Vision, Winter
2008/09Lecture 7 Stereo

• Professor Sebastian Thrun
• CAs Ethan Dreyfuss, Young Min Kim, Alex Teichman
  

2
Vocabulary Quiz

• Baseline
• Epipole
• Fundamental Matrix
• Essential Matrix
• Stereo Rectification
• Sprites

3
Why Stereo Vision?
P
Q
PQ
O
center of projection

• 2D images project 3D points into 2D

4
Stereo Vision Illustration
http//www.well.com/user/jimg/stereo/stereo_list.h
tml
5
Stereo Example (Stanley Robot)
Disparity map
6
Stereo Example
Credit Ben Wegbreit
7
Autostereograms

• Depth perception from one image
• Viewing trick the brain by focusing at the plane
  behind - match can be established perception of
  3D

8
Stereo Vision Outline

• Basic Equations
• Correspondence
• Epipolar Geometry
• Image Rectification
• Layered Stereo
• Smoothing

9
Pinhole Camera Model
Image plane
Focal length f
Center of projection
10
Pinhole Camera Model
Image plane
11
Basic Stereo Derivations
12
Basic Stereo Derivations
13
Stereo Vision Outline

• Basic Equations
• Correspondence
• Epipolar Geometry
• Image Rectification
• Layered Stereo
• Smoothing

14
Correspondence
15
Correspondence via Correlation
Left
Right
Rectified images
(Same as max-correlation / max-cosine for
normalized image patch)
16
Images as Vectors
Left
Right
Each window is a vectorin an m2
dimensionalvector space.Normalization
makesthem unit length.
17
Correspondence Metrics
(Normalized) Sum of Squared Differences
Normalized Correlation
18
Correspondence Using Correlation
Left
Disparity Map
Images courtesy of Point Grey Research
19
Window size

• Effect of window size

• Better results with adaptive window
• T. Kanade and M. Okutomi, A Stereo Matching
  Algorithm with an Adaptive Window Theory and
  Experiment,, Proc. International Conference on
  Robotics and Automation, 1991.
• D. Scharstein and R. Szeliski. Stereo matching
  with nonlinear diffusion. International Journal
  of Computer Vision, 28(2)155-174, July 1998

(S. Seitz)
20
Stereo results

• Data from University of Tsukuba

Ground truth
Scene
(Seitz)
21
Results with window correlation
Window-based matching (best window size)
Ground truth
(Seitz)
22
Results with better method
State of the art
Ground truth
Boykov et al., Fast Approximate Energy
Minimization via Graph Cuts, International
Conference on Computer Vision, September 1999.
(Seitz)
23
Correspondence By Features
RIGHT IMAGE
LEFT IMAGE

• Search in the right image the disparity (dx, dy)
  is the displacement when the similarity measure
  is maximum

24
Whats the best way to correspond?

• By individual image area (feature)?
• By groups of image areas (features?)

25
Stereo Correspondences
Left scanline
Right scanline
26
Stereo Correspondences
Left scanline
Right scanline
27
Search Over Correspondences
Left scanline
Right scanline
Disoccluded Pixels

• Three cases
• Sequential cost of match
• Occluded cost of no match
• Disoccluded cost of no match

28
Stereo Matching with Dynamic Programming
Left scanline
Start

• Find the shortest path through grid, moving
  only diagonal (match), right or down
  (occlusion/disocclusion)

Dis-occluded Pixels
Right scanline
End
29
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
30
Stereo Matching with Dynamic Programming
Left scanline
Dynamic programming yields the optimal path
through grid. This is the best set of matches
that satisfy the ordering constraint
Dis-occluded Pixels
Right scanline
Terminal
31
Dynamic Programming (Ohta and Kanade, 1985)
Reprinted from Stereo by Intra- and
Intet-Scanline Search, by Y. Ohta and T. Kanade,
IEEE Trans. on Pattern Analysis and
Machine Intelligence, 7(2)139-154 (1985). ? 1985
IEEE.
32
Dynamic Programming (DP)for Correspondence

• Does this always work?
• When would it fail?
• Failure Example 1
• Failure Example 2
• Failure Example 3

33
Correspondence Problem 1

• It is fundamentally ambiguous, even with stereo
  constraints

Figure from Forsyth Ponce
Ordering constraint
and its failure
34
Correspondence Problem 2

• Correspondence fail for smooth surfaces
• There is currently no good solution to the
  correspondence problem

35
Correspondence Problem 3

• Regions without texture
• Highly Specular surfaces
• Translucent objects

36
Stereo Vision Outline

• Basic Equations
• Correspondence
• Epipolar Geometry
• Image Rectification
• Layered Stereo
• Smoothing

37
What If?
38
What If?
39
Goal Rectification
P
Pl
Pr
Yr
p
p
l
r
Yl
Xl
Zl
Zr
T
Ol
Or
Xr
40
Epipolar Rectified Images
Source A. Fusiello, Verona, 2000
41
Stereo Rectification (nonlinear)
Marc Pollefeys
42
Epipolar Geometry
P
Pl
Pr
Yr
p
p
r
l
Yl
Zl
Zr
Xl
fl
fr
Ol
Or
Xr
43
Epipolar Geometry
P
Pl
Pr
Epipolar Plane
Epipolar Lines
p
p
l
r
Ol
el
er
Or
Epipoles
44
Epipolar Geometry

• Epipolar plane plane going through point P and
  the centers of projection (COPs) of the two
  cameras
• Epipoles The image in one camera of the COP of
  the other
• Epipolar Constraint Corresponding points must
  lie on epipolar lines

45
Essential Matrix
P
Pl
Pr
p
p
r
l
Ol
el
er
Or
46
Essential Matrix
P
Pl
Pr
p
p
r
l
Ol
el
er
Or
47
Fundamental Matrix

• Same as Essential Matrix in Camera Pixel
  Coordinates

Pixel coordinates
Intrinsic parameters
48
Computing F

• How many points do we need?

49
Computing F The Eight-Point Algorithm

• Problem Recover F (3-3 matrix of rank 2)
• Idea Get 8 points
• Minimize
• Notice Argument linear in coefficients of F

50
Computing F The Eight-Point Algorithm

• Run Singular Value Decomposition of A
• Appendix A.6, page 322-325
• See also G. Strang Linear algebra and its
  applications
• Least squares solution column of V corresponding
  to the smallest eigenvalue of A

51
Computing F The Eight-Point Algorithm

• Idea Compile points into matrix A

52
Computing F The Eight-Point Algorithm

• Decompose A via SVD
• Solution F is column of V corresponding to the
  smallest eigenvector of A
• In practice F will be of rank 3, not 2. Correct
  by
• SVD decomposition of F
• Set smallest eigenvalue to 0
• Reconstruct F

53
Computing F The Eight-Point Algorithm

• Input n point correspondences ( n gt 8)
• Construct homogeneous system Ax 0 from
• x (f11,f12, ,f13, f21,f22,f23 f31,f32, f33)
  entries in F
• Each correspondence give one equation
• A is a nx9 matrix
• Obtain estimate F by SVD of A
• x (up to a scale) is column of V corresponding to
  the least singular value
• Enforce singularity constraint since Rank (F)
  2
• Compute SVD of F
• Set the smallest singular value to 0 D -gt D
• Correct estimate of F
• Output the estimate of the fundamental matrix
  F
• Similarly we can compute E given intrinsic
  parameters

54
Stereo Vision Outline

• Basic Equations
• Correspondence
• Epipolar Geometry
• Image Rectification
• Layered Stereo
• Smoothing

55
Recitification

• Idea Align Epipolar Lines with Scan Lines.
• Question What type transformation?

56
Locating the Epipoles
P
Pl
Pr
p
p
r
l
Ol
el
er
Or
57
Stereo Rectification (see Trucco)
P
Pl
Pr
Yr
p
p
l
r
Yl
Xl
Zl
Zr
T
Ol
Or
Xr

• Stereo System with Parallel Optical Axes
• Epipoles are at infinity
• Horizontal epipolar lines

58
Reconstruction (3-D) Idealized
Pl
Pr
P
p
p
r
l
Ol
Or
59
Reconstruction (3-D) Real
Pl
Pr
P
p
p
r
l
Ol
Or
See Trucco/Verri, pages 161-171
60
Epipolar Rectified Images
Epipolar line
61
Epipolar Rectified Images
Source A. Fusiello, Verona, 2000
62
Image Normalization

• Even when the cameras are identical models, there
  can be differences in gain and sensitivity.
• The cameras do not see exactly the same surfaces,
  so their overall light levels can differ.
• For these reasons and more, it is a good idea to
  normalize the pixels in each window

63
Stereo Vision Outline

• Basic Equations
• Correspondence
• Epipolar Geometry
• Image Rectification
• Layered Stereo
• Smoothing

64
Layered Stereo

• Assign pixel to different layers (objects,
  sprites)

65
Layered Stereo

• Track each layer from frame to frame, compute
  plane eqn. and composite mosaic
• Re-compute pixel assignment by comparing original
  images to sprites

66
Layered Stereo

• Re-synthesize original or novel images from
  collection of sprites

67
Layered Stereo

• Advantages
• can represent occluded regions
• can represent transparent and border (mixed)
  pixels (sprites have alpha value per pixel)
• works on texture-less interior regions
• Limitations
• fails for high depth-complexity scenes

68
Fitting Planar Surfaces (with EM)
69
Expectation Maximization

• 3D Model

Distance point-surface
70
Mixture Measurement Model

• Case 1 Measurement zi caused by plane qj

• Case 2 Measurement zi caused by something else

71
Measurement Model with Correspondences
72
Expected Log-Likelihood Function
73
The EM Algorithm

• E-step given plane params, compute
• M-step given expectations, compute

74
Choosing the Right Number of Splines AIC
J2
J3
J5
J0
J1
J4
75
Determining Number of Sprites
76
Layered Stereo

• Resulting sprite collection

77
Layered Stereo

• Estimated depth map

78
Stereo Vision Outline

• Basic Equations
• Correspondence
• Epipolar Geometry
• Image Rectification
• Layered Stereo
• Smoothing

79
Smoothing Motivation and Goals
James Diebel
80
Motivation and Goals
James Diebel
81
Network of Constraints (Markov Random Field)
James Diebel
82
MRF Approach to Smoothing

• Potential function contains a sensor-model term
  and a surface prior
• The edge potential is important!
• Minimize ? by conjugate gradient
• Optimize systems with tens of thousands of
  parameters in just a couple seconds
• Time to converge is O(N), between 0.7 sec (25,000
  nodes in the MRF) and 25 sec (900,000 nodes)

Diebel/Thrun, 2006
83
Possible Edge Potential Functions
L2
L1
84
Results Smoothing
James Diebel
85
Results Smoothing
James Diebel
86
Results Smoothing
James Diebel
87
Results Smoothing
James Diebel
88
Movies
89
Summary Stereo Vision

• Correspondence feature- or region-based solved
  using dynamic programming
• Epipolar Geometry Corresponding points lie on
  epipolar line
• Essential/Fundamental matrix Defines this line
• Eight-Point Algorithm Recovers Fundamental
  matrix
• Rectification Epipolar lines parallel to
  scanlines
• Reconstruction Minimize quadratic distance
• Layered stereo extracts small number of sprites
  (panaer regions)
• Smoothing Use Markov Random Fields