Multiple View Geometry

1
Multiple View Geometry

• Marc Pollefeys
• University of North Carolina at Chapel Hill

Modified by Philippos Mordohai
2
Outline

• Stereo matching
• Self-calibration
• Chapter 11 and 18 of Multiple View Geometry in
  Computer Vision by Hartley and Zisserman

3
Stereo matching

• attempt to match every pixel
• use additional constraints

4
Exploiting motion and scene constraints

• Epipolar constraint (through rectification)

• Ordering constraint
• Uniqueness constraint
• Disparity limit
• Disparity continuity constraint

5
Ordering constraint
surface slice
surface as a path
6
5
occlusion left
4
3
1
2
4,5
6
1
2,3
2,3
4
5
6
occlusion right
1
3
6
2
1
4,5
6
Uniqueness constraint

• In an image pair each pixel has at most one
  corresponding pixel
• In general one corresponding pixel
• In case of occlusion there is none

7
Disparity constraint
surface slice
surface as a path
bounding box
disparity band
constant disparity surfaces
8
Disparity continuity constraint

• Assume piecewise continuous surface
• piecewise continuous disparity
• In general disparity changes continuously
• discontinuities at occluding boundaries

9
Stereo matching

• Constraints
• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
• Trade-off
• Matching cost (data)
• Discontinuities (prior)

(Cox et al. CVGIP96 Koch96 Falkenhagen97
Van Meerbergen,Vergauwen,Pollefeys,VanGool
IJCV02)
10
Hierarchical stereo matching
Allows faster computation Deals with large
disparity ranges
Downsampling (Gaussian pyramid)
Disparity propagation
(Falkenhagen97Van Meerbergen,Vergauwen,Pollefeys
,VanGool IJCV02)
11
Disparity map
image I(x,y)
image I(x,y)
Disparity map D(x,y)
(x,y)(xD(x,y),y)
12
Example reconstruct image from neighboring images
13
Multi-view depth fusion
(Koch, Pollefeys and Van Gool. ECCV98)

• Compute depth for every pixel of reference image
• Triangulation
• Use multiple views
• Up- and down sequence
• Use Kalman filter

Allows to compute robust texture
14
Point reconstruction
15
Linear triangulation
homogeneous
invariance?
Inhomogeneous is affine invariant
inhomogeneous
16
Geometric error
Can be compute using Levenberg-Marquadt (for 2 or
more points) or directly (for 2 points)
17
Geometric error

• Reconstruct matches in projective frame
• by minimizing the reprojection error

Non-iterative optimal solution
(see HartleySturm,CVIU97)
18
Reconstruction uncertainty
consider angle between rays
19
Outline

• Stereo matching
• Self-calibration
• Introduction
• Self-calibration
• Dual Absolute Quadric
• Critical Motion Sequences

20
Motivation

• Avoid explicit calibration procedure
• Complex procedure
• Need for calibration object
• Need to maintain calibration

21
Motivation

• Allow flexible acquisition
• No prior calibration necessary
• Possibility to vary intrinsics
• Use archive footage

22
Example
23
Projective ambiguity

• Reconstruction from uncalibrated images
• ? projective ambiguity on reconstruction

24
Stratification of geometry
Projective
Affine
Metric
15 DOF
7 DOF absolute conic angles, rel.dist.
12 DOF plane at infinity parallelism
More general
More structure
25
Constraints ?

• Scene constraints
• Parallellism, vanishing points, horizon, ...
• Distances, positions, angles, ...

Unknown scene ? no constraints

• Camera extrinsics constraints
• Pose, orientation, ...

Unknown camera motion ? no constraints

• Camera intrinsics constraints
• Focal length, principal point, aspect ratio skew

Perspective camera model too general ? some
constraints
26
Euclidean projection matrix
Factorization of Euclidean projection matrix
Intrinsics
(camera geometry)
Extrinsics
(camera motion)
Note every projection matrix can be factorized,
but only meaningful for euclidean projection
matrices
27
Constraints on intrinsic parameters

• Constant
• e.g. fixed camera
• Known
• e.g. rectangular pixels
• square pixels
• principal point known

28
Self-calibration

• Upgrade from projective structure to metric
  structure using constraints on intrinsic camera
  parameters
• Constant intrinsics
• Some known intrinsics, others varying
• Constraints on intrincs and restricted motion
• (e.g. pure translation, pure rotation, planar
  motion)

(Faugeras et al. ECCV92, Hartley93, Triggs97,
Pollefeys et al. PAMI99, ...)
(HeydenAstrom CVPR97, Pollefeys et al.
ICCV98,...)
(Moons et al.94, Hartley 94, Armstrong ECCV96,
...)
29
A counting argument

• To go from projective (15DOF) to metric (7DOF) at
  least 8 constraints are needed
• Minimal sequence length should satisfy
• Independent of algorithm
• Assumes general motion (i.e. not critical)

30
Self-calibrationconceptual algorithm
Given projective structure and motion Pj,Mi,
then the metric structure and motion can be
obtained as PjT-1,TMi, with
cost given constraints
function extracting intrinsics from projection
matrix
31
Conics Quadrics
conics
32
The Absolute Conic

• ?? is a specific imaginary conic on ??,
• for metric frame
• or
• Remember, the absolute conic is fixed under H
• if, and only if, H is a similarity transformation
  
• Image related to intrinsics

33
The Absolute Dual Quadric
(Triggs CVPR97)

• Degenerate dual quadric ??
• Encodes both absolute conic ?? and ??

??
??
??
for metric frame
34
Absolute Dual Quadric and Self-calibration

• Eliminate extrinsics from equation

Equivalent to projection of dual quadric
Abs.Dual Quadric also exists in projective world
Transforming world so that reduces ambiguity to
metric
35
Absolute Dual Quadric and Self-calibration
Projection equation

• Translate constraints on K through projection
  equation to constraints on ?

Absolute conic calibration object which is
always present but can only be observed through
constraints on the intrinsics
36
Constraints on ??
constraints
condition
constraint
type
37
Linear algorithm
(Pollefeys et al.,ICCV98/IJCV99)

• Assume everything known, except focal length

Yields 4 constraint per image Note that rank-3
constraint is not enforced
38
Linear algorithm revisited
(Pollefeys et al., ECCV02)
Weighted linear equations
assumptions
39
Projective to metric

• Compute T from
• using eigenvalue decomposition of
• and then obtain metric reconstruction as

40
Alternatives (Dual) image of absolute conic

• Equivalent to Absolute Dual Quadric
• Practical when H? can be computed first
• Pure rotation (Hartley94, Agapito et al.98,99)
• Vanishing points, pure translations, modulus
  constraint,

41
Note that in the absence of skew the IAC can be
more practical than the DIAC!
42
Kruppa equations

• Limit equations to epipolar geometry
• Only 2 independent equations per pair
• But independent of plane at infinity

43
Refinement

• Metric bundle adjustment

Enforce constraints or priors on intrinsics
during minimization (this is self-calibration
for photogrammetrist)