Multiple View Geometry

1
Multiple View Geometry

• Marc Pollefeys
• University of North Carolina at Chapel Hill

Modified by Philippos Mordohai
2
Outline

• Fundamental matrix estimation
• Image rectification
• Structure Computation
• Stereo
• Chapter 11 and 12of Multiple View Geometry in
  Computer Vision by Hartley and Zisserman

3
Automatic computation of F

• Interest points
• Putative correspondences
• RANSAC
• (iv) Non-linear re-estimation of F
• Guided matching
• (repeat (iv) and (v) until stable)

4
Feature points

• Extract feature points to relate images
• Required properties
• Well-defined
• (i.e. neigboring points should all be
  different)
• Stable across views

(i.e. same 3D point should be extracted as
feature for neighboring viewpoints)
5
Feature points
(e.g.HarrisStephens88 ShiTomasi94)
Find points that differ as much as possible from
all neighboring points
homogeneous
edge
corner
M should have large eigenvalues
Feature local maxima (subpixel) of F(?1, ? 2)
6
Feature points

• Select strongest features (e.g. 1000/image)

7
Feature matching

• Evaluate NCC for all features with
• similar coordinates

Keep mutual best matches Still many wrong matches!
8
Feature example
0.96 -0.40 -0.16 -0.39 0.19
-0.05 0.75 -0.47 0.51 0.72
-0.18 -0.39 0.73 0.15 -0.75
-0.27 0.49 0.16 0.79 0.21
0.08 0.50 -0.45 0.28 0.99
Gives satisfying results for small image motions
9
Wide-baseline matching

• Requirement to cope with larger variations
  between images
• Translation, rotation, scaling
• Foreshortening
• Non-diffuse reflections
• Illumination

geometric transformations
photometric changes
10
RANSAC

• Step 1. Extract features
• Step 2. Compute a set of potential matches
• Step 3. do
• Step 3.1 select minimal sample (i.e. 7 matches)
• Step 3.2 compute solution(s) for F
• Step 3.3 determine inliers
• until ?(inliers,samples)lt95

Step 4. Compute F based on all inliers Step 5.
Look for additional matches Step 6. Refine F
based on all correct matches
inliers 90 80 70 60 50
samples 5 13 35 106 382
11
Finding more matches
restrict search range to neighborhood of
epipolar line (?1.5 pixels) relax disparity
restriction (along epipolar line)
12
Degenerate cases

• Degenerate cases
• Planar scene
• Pure rotation
• No unique solution
• Remaining DOF filled by noise
• Use simpler model (e.g. homography)
• Model selection (Torr et al., ICCV98, Kanatani,
  Akaike)
• Compare H and F according to expected residual
  error (compensate for model complexity)

13
More problems

• Absence of sufficient features (no texture)
• Repeated structure ambiguity

• Robust matcher also finds
• support for wrong hypothesis
• solution detect repetition

(Schaffalitzky and Zisserman, BMVC98)
14
Two-view geometry

• geometric relation between two views is fully
• described by recovered 3x3 matrix F

15
Outline

• Fundamental matrix estimation
• Image rectification
• Structure Computation
• Stereo
• Chapter 11 and 12of Multiple View Geometry in
  Computer Vision by Hartley and Zisserman

16
Point reconstruction
17
Linear triangulation
homogeneous
invariance?
Inhomogeneous is affine invariant
inhomogeneous
18
Geometric error
Can be compute using Levenberg-Marquadt (for 2 or
more points) or directly by solving 6th degree
polynomial (for 2 points)
19
Reconstruction uncertainty
consider angle between rays
20
Outline

• Fundamental matrix estimation
• Image rectification
• Structure Computation
• Stereo
• Chapter 11 and 12of Multiple View Geometry in
  Computer Vision by Hartley and Zisserman

21
Image pair rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar
lines correspond to horizontal scanlines
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
22
Planar rectification
(standard approach)
Distortion minimization (uncalibrated)
Bring two views to standard stereo setup (moves
epipole to ?) (not possible when in/close to
image)
23
(No Transcript)
24
(No Transcript)
25
Polar rectification
(Pollefeys et al. ICCV99)
Polar re-parameterization around
epipoles Requires only (oriented) epipolar
geometry Preserve length of epipolar lines Choose
?? so that no pixels are compressed
original image
rectified image
Works for all relative motions Guarantees minimal
image size
26
polar rectification example
27
polar rectification example
28
Example Béguinage of Leuven
Does not work with standard Homography-based
approaches
29
Example Béguinage of Leuven
30
Outline

• Fundamental matrix estimation
• Image rectification
• Structure Computation
• Stereo
• Chapter 11 and 12of Multiple View Geometry in
  Computer Vision by Hartley and Zisserman

31
Stereo matching

• attempt to match every pixel
• use additional constraints

32
Exploiting motion and scene constraints

• Epipolar constraint (through rectification)

• Ordering constraint
• Uniqueness constraint
• Disparity continuity constraint

33
Ordering constraint
surface slice
surface as a path
6
5
occlusion left
4
3
1
2
4,5
6
1
2,3
2,3
4
5
6
occlusion right
1
3
6
2
1
4,5
34
Uniqueness constraint

• In an image pair each pixel has at most one
  corresponding pixel
• In general one corresponding pixel
• In case of occlusion there is none

35
Disparity continuity constraint

• Assume piecewise continuous surface
• piecewise continuous disparity
• In general disparity changes continuously
• discontinuities at occluding boundaries

36
Stereo matching

• Constraints
• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
• Trade-off
• Matching cost (data)
• Discontinuities (prior)

(Cox et al. CVGIP96 Koch96 Falkenhagen97
Van Meerbergen,Vergauwen,Pollefeys,VanGool
IJCV02)
37
Disparity map
image I(x,y)
image I(x,y)
Disparity map D(x,y)
(x,y)(xD(x,y),y)
38
Hierarchical stereo matching
Allows faster computation Deals with large
disparity ranges
Downsampling (Gaussian pyramid)
Disparity propagation
(Falkenhagen97Van Meerbergen,Vergauwen,Pollefeys
,VanGool IJCV02)