Projective structure from motion

1
Projective structure from motion

• Marc Pollefeys
• COMP 256

Some slides and illustrations from J. Ponce,
2
Last class Affine camera

• The affine projection equations are

how to find the origin? or for that matter a 3D
reference point?
affine projection preserves center of gravity
3
Orthographic factorization

• The orthographic projection equations are
• where

Note that P and X are resp. 2mx3 and 3xn matrices
and therefore the rank of x is at most 3
4
Orthographic factorization

• Factorize m through singular value decomposition
• An affine reconstruction is obtained as follows

Closest rank-3 approximation yields MLE!
5
Tentative class schedule
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources Shadows Color
Feb 6/8 Linear filters edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
6
PROJECTIVE STRUCTURE FROM MOTION

• The Projective Structure from Motion Problem
• Elements of Projective Geometry
• Projective Structure and Motion from Two Images
• Projective Motion from Fundamental Matrices
• Projective Structure and Motion from Multiple
  Images

Reading Chapter 13.
7
The Projective Structure-from-Motion Problem
Given m perspective images of n fixed points P
we can write
j
2mn equations in 11m3n unknowns
Overconstrained problem, that can be solved using
(non-linear) least squares!
8
The Projective Ambiguity of Projective SFM
When the intrinsic and extrinsic parameters are
unknown
and Q is an arbitrary non-singular 4x4 matrix.
Q is a projective transformation.
9
Projective Spaces (Semi-Formal) Definition
10
A Model of P( R )
3
11
Projective Subspaces and Projective Coordinates
12
Projective Subspaces and Projective Coordinates
P
Projective coordinates
13
Projective Subspaces
Given a choice of coordinate frame
Line
Plane
14
Affine and Projective Spaces
15
Affine and Projective Coordinates
16
Cross-Ratios
Collinear points
Pencil of planes
Pencil of coplanar lines
17
Cross-Ratios and Projective Coordinates

Along a line equipped with the basis


In a plane equipped with the basis
In 3-space equipped with the basis
18
Projective Transformations
Bijective linear map
Projective transformation ( homography )
Projective transformations map projective
subspaces onto projective subspaces and preserve
projective coordinates.
Projective transformations map lines onto lines
and preserve cross-ratios.
19
Perspective Projections induce projective
transformations between planes.
20
Geometric Scene Reconstruction
Idea use (A,O,O,B,C) as a projective basis.
21
Reprinted from Relative Stereo and Motion
Reconstruction, by J. Ponce, T.A. Cass and D.H.
Marimont, Tech. Report UIUC-BI-AI-RCV-93-07,
Beckman Institute, Univ. of Illinois (1993).
22
Motion estimation from fundamental matrices
Q
Once M and M are known, P can be computed with
LLS.
Facts
b can be found using LLS.
23
Projective Structure from Motion and Factorization
Factorization??

• Algorithm (Sturm and Triggs, 1996)
• Guess the depths
• Factorize D
• Iterate.

Does it converge? (Mahamud and Hebert, 2000)
24
Bundle adjustment - refining structure and motion

• Minimize reprojection error
• Maximum Likelyhood Estimation (if error
  zero-mean Gaussian noise)
• Huge problem but can be solved efficiently
  (exploit sparseness)

25
Bundle adjustment

• Developed in photogrammetry in 50s

26
Non-linear least squares

• Linear approximation of residual
• allows quadratic approximation of sum-
• of-squares

Minimization corresponds to finding zeros of
derivative
N
Levenberg-Marquardt extra term to deal with
singular N (decrease/increase l if
success/failure to descent)
(extra term descent term)
27
Bundle adjustment

• Jacobian of has sparse block structure
• cameras independent of other cameras,
• points independent of other points

im.pts. view 1
Needed for non-linear minimization
28
Bundle adjustment

• Eliminate dependence of camera/motion parameters
  on structure parameters
• Note in general 3n gtgt 11m

Allows much more efficient computations
e.g. 100 views,10000 points, solve
?1000x1000, not ?30000x30000
Often still band diagonal use sparse linear
algebra algorithms
29
Sequential SfM

• Initialize motion from two images
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

30
Initial projective camera motion

• Choose P and Pcompatible with F
• Reconstruction up to projective ambiguity

Same for more views?

• Initialize motion
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

different projective basis
31
Initializing projective structure

• Reconstruct matches in projective frame
• by minimizing the reprojection error

Non-iterative optimal solution

• Initialize motion
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

32
Projective pose estimation

• Infere 2D-3D matches from 2D-2D matches
• Compute pose from (RANSAC,6pts)

X
F
x

• Initialize motion
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

Inliers
33
Refining and extending structure

• Refining structure
• Extending structure
• 2-view triangulation

(Iterative linear)

• Initialize motion
• Initialize structure
• For each additional view
• Determine pose of camera
• Refine and extend structure
• Refine structure and motion

34
Refining structure and motion

• use bundle adjustment

Also model radial distortion to avoid bias!
35
Hierarchical structure and motion recovery

• Compute 2-view
• Compute 3-view
• Stitch 3-view reconstructions
• Merge and refine reconstruction

F
T
H
PM
36
Metric structure and motion

• use self-calibration (see next class)

Note that a fundamental problem of the
uncalibrated approach is that it fails if a
purely planar scene is observed (in one or more
views) (solution possible based on model
selection)
37
Dealing with dominant planes
38
PPPgric
HHgric
39
Farmhouse 3D models
(note reconstruction much larger than camera
field-of-view)
40
Application video augmentation
41
Next class Camera calibration (and
self-calibration)
Reading Chapter 2 and 3