Multiple-view Reconstruction from Points and Lines

1
Multiple-view Reconstruction from Points and Lines
2
Single and Two view summary

• Uncalibrated case
• Recover Fundamental Matrix F
• Projective reconstruction
• Recover planar homography
• (no decomposition possible)
• Rotational homography (mosaics)
• Calibration with planar rig
• Calibration with 3D rig
• Single view
• Homography between 3D plane and image plane
  (rectification)
• Partial calibration using vanishing points
• Partial calibration and pose recovery using from
  single view/world homography

• Calibrated case
• Recover Essential Matrix decompose to R,T
• 3D reconstruction
• Planar case
• Recover homography H
• decompose to R,T,n,d

3
Problem formulation
Input Corresponding images (of features) in
multiple images. Output Camera motion, camera
calibration, object structure.
4
Affine/Orthographic projection model

• Full perspective projection model
• Affine camera projection model (in-homogenous
  coordinates)
• Good approximation if the distance from the scene
  gtgt scene depth variation
• Rigid body motion under affine projection model

5
Affine Multiview Factorization

• Problem given n correspondences in m views
• determine and 3D points
• Obtained by minimization of the following
  objective function
• Assume that the centroid of the structure is the
  origin of
• the coordinate frame and denote the
  centroids in the following way
• Minimize with respect to

6
Affine Multiview Factorization
The choice of the frame is arbitrary further
assume that The objective function then
becomes Writing all the constrains in the
matrix form
Drop for clarity
7
Affine Multiview Factorization
Measurement matrix
Motion matrix
Structure matrix

• Matrix W must have rank 3 (product of two rank 3
  matrices)
• In noise case we seek best rank 3 approximation
  of W matrix
• Given the actual measurement matrix compute
  SVD
• Best rank 3 approximation is

8
Affine Multiview Factorization

• Decomposition of is not unique

• Affine ambiguity Q

• The ambiguity can be resolved using rotation
  matrix constraints

where
Tomasi, KanadeIJCV 1992 Factorization approach
9
Projective Multiview Factorization
Measurement matrix
Motion matrix
Structure matrix

• Similar strategy expect now the scales are
  unknown
• We dont have the matrix W
• Matrix W is a product of two rank 4 matrices
  i.e. is rank 4
• How to apply the factorization idea to
  projective setting ?
• - need to compute scales (possible from two
  view reconstruction)
• - or initialize the scales and iterate

10
Projective Factorization Algorithm

• Given a set of image points in n views
• Compute the projective depths using two
  view methods or set
• Form the measurement matrix W and find the
  nearest rank 4 approximation using SVD
• Decompose into camera matrices and structure
• Optionally reproject the and iterate
• (i.e. given new motions, we can compute new
  scales)

11
Multi-view methods

• Advantages of multiview methods
• - more frames wider baseline better
  conditioned algorithms
• - additional complexity of matching
  (establishing
• correspondences) across multiple views
• How are multi-view and two view methods related ?
  
• Can we just run the two view algorithm for each
  pair of views
• and get better results ?
• What if we are trying to do reconstruction using
  lines ?
• Are there other constraints between multiple
  views then pairwise epipolar constraints ?
• Next brief tour of multilinear constraints

12
Traditional multifocal constraints
For images of the same 3-D point
(leading to the conventional approach)
Multilinear constraints among 2, 3, 4-wise views
13
Rank conditions for point feature
WLOG, choose camera frame 1 as the reference
Multiple-View Matrix
Lemma Rank Condition for Point Features
Let
then and are linearly dependent.
14
Rank conditions vs. multifocal constraints
15
Rank conditions vs. multi-focal constraints
16
Rank conditions vs. multifocal constraints

• These constraints are only necessary but NOT
  sufficient!
• However, there is NO further relationship among
  quadruple wise
• views. Quadrilinear constraints hence are
  redundant!

17
Point Features Uniqueness of the pre-image
bilinear constraints coplanarity constraints
Extend to three views
collinear optical centers
coplanar optical centers
18
Point Features Uniqueness of the pre-image
trilinear constraints
Given m vectors with respect
to m camera frames, They correspond to a unique
point in the 3D space if the rank of the Matrix
Mp is 1. If rank is 0, the point is determined
up to a line on which all optical centers must
lie .
19
Image of a line feature
Homogeneous representation of a 3-D line
Homogeneous representation of its 2-D co-image
Projection of a 3-D line to an image plane
20
Multiple-view matrix line vs. point
Point Features
Line Features
21
Rank conditions line vs. point
22
Multiple-view structure and motion recovery
Given images of points
23
SVD based 4-step algorithm for SFM
24
Utilizing all incidence relations
Three edges intersect at each vertex.
.
.
.
25
Example simulations
26
Example simulations
27
Example experiments
Errors in all right angles lt 1o
28
Summary

• Incidence relations ltgt rank conditions
• Rank conditions gt multiple-view factorization
• Rank conditions implies all multi-focal
  constraints
• Rank conditions for points, lines, planes, and
• (symmetric) structures.

29
Global multiple-view analysis examples
30
A family of intersecting lines
each can randomly take the image of any of the
lines
Nonlinear constraints among up to four views
.
.
.
31
Universal rank condition
Theorem The Universal Rank Condition for images
of a point on a line
32
Instances with mixed features
Examples
Case 1 a line reference
Case 2 a point reference

• All previously known constraints are the
  theorems instances.
• Degenerate configurations if and only if a drop
  of rank.

33
Generalization restriction to a plane
Homogeneous representation of a 3-D plane
Corollary Coplanar Features
Rank conditions on the new extended remain
exactly the same!
34
Generalization restriction to a plane
Given that a point and line features lie on a
plane in 3-D space
GENERALIZATION Multiple View Matrix Coplanar
Features