Projective structure from motion

1
Projective structure from motion

• Marc Pollefeys
• COMP 256

Some slides and illustrations from J. Ponce,
2
Last class Affine camera

• The affine projection equations are

how to find the origin? or for that matter a 3D
reference point?
affine projection preserves center of gravity
3
Orthographic factorization

• The orthographic projection equations are
• where

Note that P and X are resp. 2mx3 and 3xn matrices
and therefore the rank of x is at most 3
4
Orthographic factorization

• Factorize m through singular value decomposition
• An affine reconstruction is obtained as follows

Closest rank-3 approximation yields MLE!
5
Tentative class schedule
6
Further Factorization work

• Factorization with uncertainty
• Factorization for indep. moving objects
• Factorization for dynamic objects
• Perspective factorization
• Factorization with outliers and missing pts.

(Irani Anandan, IJCV02)
(Costeira and Kanade 94)
(Bregler et al. 2000, Brand 2001)
(Sturm Triggs 1996, )
(Jacobs 1997 (affine), Martinek and Pajdla
2001, Aanaes 2002 (perspective))
7
Structure from motion of multiple moving objects
one object
multiple objects
8
SfM of multiple moving objects
9
Structure from motion of deforming objects
(Bregler et al 00 Brand 01)

• Extend factorization approaches to deal with
  dynamic shapes

10
Representing dynamic shapes
(fig. M.Brand)
represent dynamic shape as varying linear
combination of basis shapes
11
Projecting dynamic shapes
(figs. M.Brand)
Rewrite
12
Dynamic image sequences
One image
(figs. M.Brand)
Multiple images
13
Dynamic SfM factorization?
Problem find J so that M has proper structure
14
Dynamic SfM factorization
(Bregler et al 00)
Assumption SVD preserves order and orientation
of basis shape components
15
Results
(Bregler et al 00)
16
Dynamic SfM factorization
(Brand 01)
constraints to be satisfied for M
constraints to be satisfied for M, use to compute
J
hard!
(different methods are possible, not so simple
and also not optimal)
17
Non-rigid 3D subspace flow
(Brand 01)

• Same is also possible using optical flow in stead
  of features, also takes uncertainty into account

18
Results
(Brand 01)
19
Results
(Brand 01)
20
Results
(Bregler et al 01)
21
PROJECTIVE STRUCTURE FROM MOTION

• The Projective Structure from Motion Problem
• Elements of Projective Geometry
• Projective Structure and Motion from Two Images
• Projective Motion from Fundamental Matrices
• Projective Structure and Motion from Multiple
  Images

Reading Chapter 13.
22
The Projective Structure-from-Motion Problem
Given m perspective images of n fixed points P
we can write
j
2mn equations in 11m3n unknowns
Overconstrained problem, that can be solved using
(non-linear) least squares!
23
The Projective Ambiguity of Projective SFM
When the intrinsic and extrinsic parameters are
unknown
and Q is an arbitrary non-singular 4x4 matrix.
Q is a projective transformation.
24
Projective Spaces (Semi-Formal) Definition
25
A Model of P( R )
3
26
Projective Subspaces and Projective Coordinates
27
Projective Subspaces and Projective Coordinates
P
Projective coordinates
28
Projective Subspaces
Given a choice of coordinate frame
Line
Plane
29
Affine and Projective Spaces
30
Affine and Projective Coordinates
31
Cross-Ratios
Collinear points
Pencil of planes
Pencil of coplanar lines
32
Cross-Ratios and Projective Coordinates

Along a line equipped with the basis


In a plane equipped with the basis
In 3-space equipped with the basis
33
Projective Transformations
Bijective linear map
Projective transformation ( homography )
Projective transformations map projective
subspaces onto projective subspaces and preserve
projective coordinates.
Projective transformations map lines onto lines
and preserve cross-ratios.
34
Perspective Projections induce projective
transformations between planes.
35
Geometric Scene Reconstruction
Idea use (A,O,O,B,C) as a projective basis.
36
Reprinted from Relative Stereo and Motion
Reconstruction, by J. Ponce, T.A. Cass and D.H.
Marimont, Tech. Report UIUC-BI-AI-RCV-93-07,
Beckman Institute, Univ. of Illinois (1993).
37
Motion estimation from fundamental matrices
Q
Once M and M are known, P can be computed with
LLS.
Facts
b can be found using LLS.
38
Projective Structure from Motion and Factorization
Factorization??

• Algorithm (Sturm and Triggs, 1996)
• Guess the depths
• Factorize D
• Iterate.

Does it converge? (Mahamud and Hebert, 2000)
39
Bundle adjustment - refining structure and motion

• Minimize reprojection error
• Maximum Likelyhood Estimation (if error
  zero-mean Gaussian noise)
• Huge problem but can be solved efficiently
  (exploit sparseness)

40
Bundle adjustment

• Developed in photogrammetry in 50s

41
Non-linear least squares

• Linear approximation of residual
• allows quadratic approximation of sum-
• of-squares

Minimization corresponds to finding zeros of
derivative
N
Levenberg-Marquardt extra term to deal with
singular N (decrease/increase l if
success/failure to descent)
(extra term descent term)
42
Bundle adjustment

• Jacobian of has sparse block structure
• cameras independent of other cameras,
• points independent of other points

im.pts. view 1
Needed for non-linear minimization
43
Bundle adjustment

• Eliminate dependence of camera/motion parameters
  on structure parameters
• Note in general 3n gtgt 11m

Allows much more efficient computations
e.g. 100 views,10000 points, solve
?1000x1000, not ?30000x30000
Often still band diagonal use sparse linear
algebra algorithms
44
Sequential SfM

• Initialize motion from two images
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

45
Initial projective camera motion

• Choose P and Pcompatible with F
• Reconstruction up to projective ambiguity

Same for more views?

• Initialize motion
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

different projective basis
46
Initializing projective structure

• Reconstruct matches in projective frame
• by minimizing the reprojection error

Non-iterative optimal solution

• Initialize motion
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

47
Projective pose estimation

• Infere 2D-3D matches from 2D-2D matches
• Compute pose from (RANSAC,6pts)

X
F
x

• Initialize motion
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

Inliers
48
Refining and extending structure

• Refining structure
• Extending structure
• 2-view triangulation

(Iterative linear)

• Initialize motion
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

49
Refining structure and motion

• use bundle adjustment

Also model radial distortion to avoid bias!
50
Hierarchical structure and motion recovery

• Compute 2-view
• Compute 3-view
• Stitch 3-view reconstructions
• Merge and refine reconstruction

F
T
H
PM
51
Metric structure and motion

• use self-calibration (see next class)

Note that a fundamental problem of the
uncalibrated approach is that it fails if a
purely planar scene is observed (in one or more
views) (solution possible based on model
selection)
52
Dealing with dominant planes
53
PPPgric
HHgric
54
Farmhouse 3D models
(note reconstruction much larger than camera
field-of-view)
55
Application video augmentation
56
Next class Camera calibration (and
self-calibration)
Reading Chapter 2 and 3