Stanford CS223B Computer Vision, Winter 200809 Lecture 9 Structure From Motion

1
Stanford CS223B Computer Vision, Winter
2008/09Lecture 9Structure From Motion

• Professor Sebastian Thrun
• CAs Ethan Dreyfuss, Young Min Kim, Alex Teichman
  

2
Summary SFM

• Problem
• Determine feature locations (structure)
• Determine camera extrinsic (motion)
• Two Principal Solutions
• Bundle adjustment (nonlinear least squares, local
  minima)
• SVD (through orthographic approximation, affine
  geometry)
• Correspondence
• (RANSAC)
• Expectation Maximization

3
Structure From Motion
Recover structure (feature locations), motion
(camera extrinsics)
4
Recovery Problems
5
Structure From Motion (1)
Tomasi Kanade 92
6
Structure From Motion (2)
Tomasi Kanade 92
7
Structure From Motion (3)
Tomasi Kanade 92
8
Structure From Motion (4a) Images
Marc Pollefeys
9
Structure From Motion (4b)
Marc Pollefeys
10
Structure From Motion (5)
http//www.cs.unc.edu/Research/urbanscape
11
SFM Holy Grail of 3D Reconstruction

• Take movie of object
• Reconstruct 3D model

live.com
12
Structure From Motion

• Problem 1
• Given n points pij (xij, yij) in m images
• Reconstruct structure 3-D locations Pj (xj, yj,
  zj)
• Reconstruct camera positions (extrinsics) Mi(Ai,
  bi)
• Problem 2
• Establish correspondence c(pij)

13
Structure From Motion
Recover structure (feature locations), motion
(camera extrinsics)
14
SFM General Formulation
15
SFM Bundle Adjustment
16
Bundle Adjustment

• SFM Nonlinear Least Squares problem
• Minimize through
• Gradient Descent
• Conjugate Gradient
• Gauss-Newton
• Levenberg Marquardt is common method
• Prone to local minima

17
Count Constraints vs Unknowns

• m camera poses
• n points
• 2mn point constraints
• 6m3n unknowns
• Suggests need 2mn ? 6m 3n
• But Can we really recover all parameters???

18
How Many Parameters Cant We Recover?
We can recover all but
m camera poses n feature points
Place Your Bet!
19
Count Constraints vs Unknowns

• m camera poses
• n points
• 2mn point constraints
• 6m3n unknowns
• Suggests need 2mn ? 6m 3n
• But Can we really recover all parameters???
• Cant recover origin, orientation (6 params)
• Cant recover scale (1 param)
• Thus, we need 2mn ? 6m 3n - 7

20
Are we done?

• No, bundle adjustment has many local minima.

21
The Trick Of The Day

• Replace Perspective by Orthographic Geometry
• Replace Euclidean Geometry by Affine Geometry
• Solve SFM linearly via PCA (closed form,
  globally optimal)
• Post-Process to make solution Euclidean
• Post-Process to make solution perspective
• By Tomasi and Kanade, 1992

22
Orthographic Camera Model

• Orthographic Limit of Pinhole Model

23
Orthographic Projection
Limit of Pinhole Model
Orthographic Projection
24
The Orthographic SFM Problem
subject to
25
The Affine SFM Problem
subject to
26
Count Constraints vs Unknowns

• m camera poses
• n points
• 2mn point constraints
• 8m3n unknowns
• Suggests need 2mn ? 8m 3n
• But Can we really recover all parameters???

27
How Many Parameters Cant We Recover?
We can recover all but
Place Your Bet!
28
The Answer is (at least) 12
29
Points for Solving Affine SFM Problem

• m camera poses
• n points
• Need to have 2mn ? 8m 3n-12

30
Where do those additional 5 DOFs come from?

• Anisotropic scaling (2 dim)
• Sheering (3dim)

31
Affine SFM
32
The Rank Theorem
2m elements
n elements
33
Singular Value Decomposition
34
Affine Solution to Orthographic SFM
Gives also the optimal affine reconstruction
under noise
35
Back To Orthographic Projection
Find C for which constraints are met Search in
9-dim space (instead of 8m 3n-12)
36
Back To Projective Geometry
Orthographic (in the limit)
Projective
37
Back To Projective Geometry
Optimize
Using orthographic solution as starting point
38
The Trick Of The Day

• Replace Perspective by Orthographic Geometry
• Replace Euclidean Geometry by Affine Geometry
• Solve SFM linearly via PCA (closed form,
  globally optimal)
• Post-Process to make solution Euclidean
• Post-Process to make solution perspective
• By Tomasi and Kanade, 1992

39
Structure From Motion

• Problem 1
• Given n points pij (xij, yij) in m images
• Reconstruct structure 3-D locations Pj (xj, yj,
  zj)
• Reconstruct camera positions (extrinsics) Mi(Ai,
  bi)
• Problem 2
• Establish correspondence c(pij)

40
The Correspondence Problem
View 1
View 3
View 2
41
Correspondence Solution 1

• Track features (e.g., optical flow)
• but fails when images taken from widely
  different poses

42
Correspondence Solution 2

• Start with random solution A, b, P
• Compute soft correspondence p(cA,b,P)
• Plug soft correspondence into SFM
• Reiterate
• See Dellaert/Seitz/Thorpe/Thrun, Machine Learning
  Journal, 2003

43
Example
44
Results Cube
45
Animation
46
Tomasis Benchmark Problem
47
Reconstruction with EM
48
3-D Structure
49
Correspondence Alternative Approach

• Ransac Random sampling and consensus
• Fisher/Bolles

50
Summary SFM

• Problem
• Determine feature locations (structure)
• Determine camera extrinsic (motion)
• Two Principal Solutions
• Bundle adjustment (nonlinear least squares, local
  minima)
• SVD (through orthographic approximation, affine
  geometry)
• Correspondence
• (RANSAC)
• Expectation Maximization