Structure from Motion

1
Structure from Motion

• (With a digression on SVD)

2
Structure from Motion

• For now, static scene and moving camera
• Equivalently, rigidly moving scene andstatic
  camera
• Limiting case of stereo with many cameras
• Limiting case of multiview camera calibration
  with unknown target
• Given n points and N camera positions, have 2nN
  equations and 3n6N unknowns

3
Approaches

• Obtaining point correspondences
• Optical flow
• Stereo methods correlation, feature matching
• Solving for points and camera motion
• Nonlinear minimization (bundle adjustment)
• Various approximations

4
Orthographic Approximation

• Simplest SFM case camera approximated by
  orthographic projection

Perspective
Orthographic
5
Weak Perspective

• An orthographic assumption is sometimes well
  approximated by a telephoto lens

Weak Perspective
6
Consequences ofOrthographic Projection

• Scene can be recovered up to scale
• Translation perpendicular to image planecan
  never be recovered

7
Orthographic Structure from Motion

• Method due to Tomasi Kanade, 1992
• Assume n points in space p1 pn
• Observed at N points in time at image coordinates
  (xij, yij)
• Feature tracking, optical flow, etc.

8
Orthographic Structure from Motion

• Write down matrix of data

Points ?
Frames ?
9
Orthographic Structure from Motion

• Step 1 find translation
• Translation parallel to viewingdirection can not
  be obtained
• Translation perpendicular to viewing direction
  equals motion of average position of all points

10
Orthographic Structure from Motion

• Subtract average of each row

11
Orthographic Structure from Motion

• Step 2 try to find rotation
• Rotation at each frame defines local coordinate
  axes , , and
• Then

12
Orthographic Structure from Motion

• So, can write where R is a
  rotation matrix and S is a shape matrix

13
Orthographic Structure from Motion

• Goal is to factor
• Before we do, observe that rank( ) 3(in
  ideal case with no noise)
• Proof
• Rank of R is 3 unless no rotation
• Rank of S is 3 iff have noncoplanar points
• Product of 2 matrices of rank 3 has rank 3
• With noise, rank( ) might be gt 3

14
DigressionSingular Value Decomposition (SVD)

• Handy mathematical technique that has application
  to many problems
• Given any m?n matrix A, algorithm to find
  matrices U, V, and W such that
• A U W VT
• U is m?n and orthonormal
• V is n?n and orthonormal
• W is n?n and diagonal

15
SVD

• Treat as black box code widely available
  (svd(A,0) in Matlab)

16
SVD

• The wi are called the singular values of A
• If A is singular, some of the wi will be 0
• In general rank(A) number of nonzero wi
• SVD is mostly unique (up to permutation of
  singular values, or if some wi are equal)

17
SVD and Inverses

• Why is SVD so useful?
• Application 1 inverses
• A-1(VT)-1 W-1 U-1 V W-1 UT
• This fails when some wi are 0
• Its supposed to fail singular matrix
• Pseudoinverse if wi0, set 1/wi to 0 (!)
• Closest matrix to inverse
• Defined for all (even non-square) matrices

18
SVD and Least Squares

• Solving Axb by least squares
• xpseudoinverse(A) times b
• Compute pseudoinverse using SVD
• Lets you see if data is singular
• Even if not singular, ratio of max to min
  singular values (condition number) tells you how
  stable the solution will be
• Set 1/wi to 0 if wi is small (even if not exactly
  0)

19
SVD and Eigenvectors

• Let AUWVT, and let xi be ith column of V
• Consider ATA xi
• So elements of W are squared eigenvalues and
  columns of V are eigenvectors of ATA

20
SVD and Eigenvectors

• Eigenvectors of ATA turned up in finding
  least-squares solution to Ax0
• Solution is eigenvector corresponding to smallest
  eigenvalue

21
SVD and Matrix Similarity

• One common definition for the norm of a matrix is
  the Frobenius norm
• Frobenius norm can be computed from SVD
• So changes to a matrix can be evaluated by
  looking at changes to singular values

22
SVD and Matrix Similarity

• Suppose you want to find best rank-k
  approximation to A
• Answer set all but the largest k singular values
  to zero
• Can form compact representation by eliminating
  columns of U and V corresponding to zeroed wi

23
SVD and PCA

• Principal Components Analysis (PCA)
  approximating a high-dimensional data set with a
  lower-dimensional subspace

Original axes
24
SVD and PCA

• Data matrix with points as rows, take SVD
• Columns of Vk are principal components
• Value of wi gives importance of each component

25
SVD and Orthogonalization

• The matrix U is the closest orthonormal matrix
  to A
• Yet another useful application of the
  matrix-approximation properties of SVD
• Much more stable numerically thanGraham-Schmidt
  orthogonalization
• Find rotation given general affine matrix

26
End of Digression

• Goal is to factor into R and S
• Lets apply SVD!
• But should have rank 3 ?all but 3 of the wi
  should be 0
• Extract the top 3 wi, together with the
  corresponding columns of U and V

27
Factoring for Orthographic Structure from Motion

• After extracting columns, U3 has dimensions 2N?3
  (just what we wanted for R)
• W3V3T has dimensions 3?n (just what we wanted for
  S)
• So, let RU3, SW3V3T

28
Affine Structure from Motion

• The i and j entries of R are not, in general,
  unit length and perpendicular
• We have found motion (and therefore shape)up to
  an affine transformation
• This is the best we could do if we didntassume
  orthographic camera

29
Ensuring Orthogonality

• Since can be factored as R S, it can also
  be factored as (RQ)(Q-1S), for any Q
• So, search for Q such that R R Q has the
  properties we want

30
Ensuring Orthogonality

• Want or
• Let T QQT
• Equations for elements of T solve byleast
  squares
• Ambiguity add constraints

31
Ensuring Orthogonality

• Have found T QQT
• Find Q by taking square root of T
• Cholesky decomposition if T is positive definite
• General algorithms (e.g. sqrtm in Matlab)

32
Orthogonal Structure from Motion

• Lets recap
• Write down matrix of observations
• Find translation from avg. position
• Subtract translation
• Factor matrix using SVD
• Write down equations for orthogonalization
• Solve using least squares, square root
• At end, get matrix R R Q of camera
  positionsand matrix S Q-1S of 3D points

33
Results

• Image sequence

Tomasi Kanade
34
Results

• Tracked features

Tomasi Kanade
35
Results

• Reconstructed shape

Front view
Top view
Tomasi Kanade