Multiple View Geometry

1
Multiple View Geometry

• Marc Pollefeys
• University of North Carolina at Chapel Hill

Modified by Philippos Mordohai
2
Tutorial outline

• Image formation
• 2-D and 3-D projective geometry and
  transformations
• Finite perspective camera model
• Parameter estimation and RANSAC
• Epipolar geometry and the fundamental matrix
• 3-D reconstruction (stratification)
• Image rectification
• Feature matching
• Self-calibration

3
Outline

• Perspective projection
• 3D projective geometry
• Parameter estimation
• Epipolar geometry and the fundamental matrix
• 3D reconstruction and self-calibration

4
Visual 3D models from images and video
unknown scene
What can be achieved?
unknown camera
Scene (static)
camera
unknown motion
automatic modelling
Visual model
5
(Pollefeys et al. ICCV98 Pollefeys et
al.IJCV04)
6
Perspective projection
Perspective projection
Linear equations (in homogeneous coordinates)
7
Homogeneous coordinates

• 2-D points represented as 3-D vectors (x y 1)T
• 3-D points represented as 4-D vectors (X Y Z
  1)T
• Equality defined up to scale
• (X Y Z 1)T (WX WY WZ W)T
• Useful for perspective projection ? makes
  equations linear

8
The pinhole camera
9
Effects of perspective projection

• Colinearity is invariant
• Parallelism is not preserved

10
Intrinsic parameters
or

• Camera deviates from pinhole
• s skew
• fx ? fy different magnification in x and y
• (cx cy) optical axis does not pierce image plane
  exactly at the center
• Usually
• rectangular pixels
• square pixels
• principal point known

11
Extrinsic parameters
Scene motion
Camera motion
12
Projection matrix

• Mapping from 2-D to 3-D is a function of internal
  and external parameters

13
Ideal points and the line at infinity
Intersections of parallel lines
Note that in P2 there is no distinction between
ideal points and others
14
Duality in 2D
15
Outline

• Perspective projection
• 3D projective geometry
• Parameter estimation
• Epipolar geometry and the fundamental matrix
• 3D reconstruction and self-calibration

16
3D Projective Geometry

• Points, lines, planes and quadrics
• Transformations
• ?8, ?8 and O 8

17
3D points
3D point
in R3
in P3
projective transformation
(4x4-115 dof)
18
Planes
3D plane
Dual points ? planes, lines ? lines
19
Planes from points
Or implicitly from coplanarity condition
20
Hierarchy of transformations
Projective 15dof
Intersection and tangency
Parallellism of planes, Volume ratios,
centroids, The plane at infinity p8
Affine 12dof
Similarity 7dof
The absolute conic O8
Euclidean 6dof
Volume
21
The plane at infinity
The plane at infinity p? is a fixed plane under
a projective transformation H iff H is an
affinity

• canical position
• contains directions
• two planes are parallel ? line of intersection in
  p8
• line // line (or plane) ? point of intersection
  in p8

22
The absolute conic
The absolute conic O8 is a (point) conic on p?.
In a metric frame
or conic for directions (with no real points)
The absolute conic O8 is a fixed conic under the
projective transformation H iff H is a similarity

• O8 is only fixed as a set
• Circles intersect O8 in two points
• Spheres intersect p8 in O8

23
Transformation of lines and conics
For a point transformation
Transformation for lines
24
Outline

• Perspective projection
• 3D projective geometry
• Parameter estimation
• Epipolar geometry and the fundamental matrix
• 3D reconstruction and self-calibration

25
Parameter estimation

• 2D homography
• Given a set of (xi,xi), compute H (xiHxi)
• 3D to 2D camera projection
• Given a set of (Xi,xi), compute P (xiPXi)
• Fundamental matrix
• Given a set of (xi,xi), compute F (xiTFxi0)
• Trifocal tensor
• Given a set of (xi,xi,xi), compute T

26
Number of measurements required

• At least as many independent equations as degrees
  of freedom required
• Example

2 independent equations / point 8 degrees of
freedom
4x28
27
Algebraic distance
DLT minimizes
residual vector
partial vector for each (xi?xi)
algebraic error vector
Not geometrically/statistically meaningfull, but
given good normalization it works fine and is
very fast (use for initialization)
28
Geometric distance
d(.,.) Euclidean distance (in image)
e.g. calibration pattern
29
Sampson error
between algebraic and geometric error
(Sampson error)
30
RANSAC

• Objective
• Robust fit of model to data set S which contains
  outliers
• Algorithm
• Randomly select a sample of s data points from S
  and instantiate the model from this subset.
• Determine the set of data points Si which are
  within a distance threshold t of the model. The
  set Si is the consensus set of samples and
  defines the inliers of S.
• If the subset of Si is greater than some
  threshold T, re-estimate the model using all the
  points in Si and terminate
• If the size of Si is less than T, select a new
  subset and repeat the above.
• After N trials the largest consensus set Si is
  selected, and the model is re-estimated using all
  the points in the subset Si

31
How many samples?

• Choose t so probability for inlier is a (e.g.
  0.95)
• Or empirically
• Choose N so that, with probability p, at least
  one random sample is free from outliers. e.g. p
  0.99

32
Outline

• Perspective projection
• 3D projective geometry
• Parameter estimation
• Epipolar geometry and the fundamental matrix
• 3D reconstruction and self-calibration

33
The fundamental matrix F
geometric derivation
mapping from 2-D to 1-D family (rank 2)
34
The fundamental matrix F
(note doesnt work for CC ? F0)
35
The fundamental matrix F
F is the unique 3x3 rank 2 matrix that satisfies
xTFx0 for all x?x

• Transpose if F is fundamental matrix for (P,P),
  then FT is fundamental matrix for (P,P)
• Epipolar lines lFx lFTx
• Epipoles on all epipolar lines, thus eTFx0, ?x
  ?eTF0, similarly Fe0
• F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)
• F is a correlation, projective mapping from a
  point x to a line lFx (not a proper
  correlation, i.e. not invertible)

36
Projective transformation and invariance
Derivation based purely on projective concepts
F invariant to transformations of projective
3-space
unique
not unique
canonical form
37
The singularity constraint
SVD from linearly computed F matrix (rank 3)
Compute closest rank-2 approximation
38
(No Transcript)
39
The NOT normalized 8-point algorithm
40
The normalized 8-point algorithm
Transform image to -1,1x-1,1
(700,500)
(0,500)
(0,0)
(700,0)
41
Gold standard
Maximum Likelihood Estimation
( least-squares for Gaussian noise)
Initialize normalized 8-point, (P,P) from F,
reconstruct Xi
Parameterize
(overparametrized FtxM)
Minimize cost using Levenberg-Marquardt (preferabl
y sparse LM, see book)
42
Automatic computation of F

• Interest points
• Putative correspondences
• RANSAC
• (iv) Non-linear re-estimation of F
• Guided matching
• (repeat (iv) and (v) until stable)

43
Image pair rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar
lines correspond to horizontal scanlines
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
44
Planar rectification
(standard approach)
Distortion minimization (uncalibrated)
Bring two views to standard stereo setup (moves
epipole to ?) (not possible when in/close to
image)
45
Polar rectification
(Pollefeys et al. ICCV99)
Polar re-parameterization around
epipoles Requires only (oriented) epipolar
geometry Preserve length of epipolar lines Choose
?? so that no pixels are compressed
original image
rectified image
Works for all relative motions Guarantees minimal
image size
46
Polar rectification example
47
polar rectification example
48
Outline

• Perspective projection
• 3D projective geometry
• Parameter estimation
• Epipolar geometry and the fundamental matrix
• 3D reconstruction and self-calibration

49
Reconstruction problem
given xi?xi , compute P,P and Xi
for all i
without additional information possible up to
projective ambiguity
50
Outline of reconstruction

• Compute F from correspondences
• Compute camera matrices from F
• Compute 3D point for each pair of corresponding
  points

computation of F use xiFxi0 equations, linear
in coeff. F 8 points (linear), 7 points
(non-linear), 8 (least-squares) (more on this
next class)
computation of camera matrices use
triangulation compute intersection of two
backprojected rays
51
Reconstruction ambiguity similarity
52
Reconstruction ambiguity projective
53
The projective reconstruction theorem
If a set of point correspondences in two views
determine the fundamental matrix uniquely, then
the scene and cameras may be reconstructed from
these correspondences alone, and any two such
reconstructions from these correspondences are
projectively equivalent

• along same ray of P2, idem for P2

two possibilities X2iHX1i, or points along
baseline
key result allows reconstruction from pair of
uncalibrated images
54
Stratified reconstruction

• Projective reconstruction
• Affine reconstruction
• Metric reconstruction

55
Projective to affine
(if D?0)
theorem says up to projective transformation,
but projective with fixed p8 is affine
transformation
can be sufficient depending on application, e.g.
mid-point, centroid, parallellism
56
Scene constraints
Parallel lines parallel lines intersect at
infinity reconstruction of corresponding
vanishing point yields point on plane at
infinity 3 sets of parallel lines allow to
uniquely determine p8
remark in presence of noise determining the
intersection of parallel lines is a delicate
problem
remark obtaining vanishing point in one image
can be sufficient
57
Affine to metric
identify absolute conic
transform so that
then projective transformation relating original
and reconstruction is a similarity transformation
in practice, find image of W8 image w8
back-projects to cone that intersects p8 in W8
note that image is independent of particular
reconstruction
58
Orthogonality
vanishing points corresponding to orthogonal
directions
vanishing line and vanishing point corresponding
to plane and normal direction
59
Known internal parameters
60
Same camera for all images
same intrinsics ? same image of the absolute conic
e.g. moving camera
given sufficient images there is in general only
one conic that projects to the same image in all
images, i.e. the absolute conic This approach is
called self-calibration
transfer of IAC
provides 4 constraints, one more needed
61
Constraints for Reconstruction

• Scene constraints
• Parallellism, vanishing points, horizon, ...
• Distances, positions, angles, ...

Unknown scene ? no constraints

• Camera extrinsics constraints
• Pose, orientation, ...

Unknown camera motion ? no constraints

• Camera intrinsics constraints
• Focal length, principal point, aspect ratio skew

Perspective camera model too general ? some
constraints
62
Constraints on intrinsic parameters

• Constant
• e.g. fixed camera
• Known
• e.g. rectangular pixels
• square pixels
• principal point known

63
Self-calibration

• Upgrade from projective structure to metric
  structure using constraints on intrinsic camera
  parameters
• Constant intrinsics
• Some known intrinsics, others varying
• Constraints on intrincs and restricted motion
• (e.g. pure translation, pure rotation, planar
  motion)

(Faugeras et al. ECCV92, Hartley93, Triggs97,
Pollefeys et al. PAMI99, ...)
(HeydenAstrom CVPR97, Pollefeys et al.
ICCV98,...)
(Moons et al.94, Hartley 94, Armstrong ECCV96,
...)
64
The Absolute Conic
?? is a specific imaginary conic on ??, for
metric frame or Remember, the absolute conic is
fixed under H if, and only if, H is a similarity
transformation Image related to intrinsics
65
The Absolute Dual Quadric
(Triggs CVPR97)
Degenerate dual quadric ?? Encodes both absolute
conic ?? and ??
??
??
??
for metric frame
66
Absolute Dual Quadric and Self-calibration
Eliminate extrinsics from equation
Equivalent to projection of dual quadric
Abs.Dual Quadric also exists in projective world
Transforming world so that reduces ambiguity to
metric
67
Constraints on ??
constraints
condition
constraint
type
68
Linear algorithm
(Pollefeys et al.,ICCV98/IJCV99)
Assume everything known, except focal length
Yields 4 constraint per image Note that rank-3
constraint is not enforced
69
Linear algorithm revisited
(Pollefeys et al., ECCV02)
Weighted linear equations
assumptions
70
Projective to metric
Compute T from using eigenvalue decomposition
of and then obtain metric reconstruction as