Title: Affine structure from motion
1Affine structure from motion
Some slides and illustrations from J. Ponce, A.
Zisserman, R. Hartley, Luc Van Gool,
2Last time Optical Flow
Ixu
Ix
u
Ixu- It
It
Aperture problem
- two solutions
- - regularize (smoothness prior)
- constant over window
- (i.e. Lucas-Kanade)
Coarse-to-fine, parametric models, etc
3Tentative class schedule
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources Shadows Color
Feb 6/8 Linear filters edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
4AFFINE STRUCTURE FROM MOTION
- The Affine Structure from Motion Problem
- Elements of Affine Geometry
- Affine Structure from Motion from two Views
- A Geometric Approach
- Affine Epipolar Geometry
- An Algebraic Approach
- Affine Structure from Motion from Multiple Views
- From Affine to Euclidean Images
- Structure from motion of multiple and deforming
object
Reading Chapter 12.
5Affine Structure from Motion
Reprinted with permission from Affine Structure
from Motion, by J.J. (Koenderink and A.J.Van
Doorn, Journal of the Optical Society of America
A, 8377-385 (1990). ? 1990 Optical Society of
America.
- Given m pictures of n points, can we recover
- the three-dimensional configuration of these
points? - the camera configurations?
(structure) (motion)
6Orthographic Projection
Parallel Projection
7Weak-Perspective Projection
Paraperspective Projection
8The Affine Structure-from-Motion Problem
Given m images of n fixed points P we can write
j
2mn equations in 8m3n unknowns
Overconstrained problem, that can be solved using
(non-linear) least squares!
9The Affine Ambiguity of Affine SFM
When the intrinsic and extrinsic parameters are
unknown
So are M and P where
i
j
and
Q is an affine transformation.
10Affine Spaces (Semi-Formal) Definition
112
Example R as an Affine Space
12In General
The notation
is justified by the fact that choosing some
origin O in X allows us to identify the point P
with the vector OP.
Warning Pu and Q-P are defined independently
of O!!
13Barycentric Combinations
NO!
we can define
14Affine Subspaces
15Affine Coordinates
- Coordinate system for YOU
16When do m1 points define a p-dimensional
subspace Y of an n-dimensional affine space X
equipped with some coordinate frame basis?
Rank ( D ) p1, where
Writing that all minors of size (p2)x(p2) of D
are equal to zero gives the equations of Y.
17Affine Transformations
- Bijections from X to Y that
- map m-dimensional subspaces of X onto
m-dimensional - subspaces of Y
- map parallel subspaces onto parallel subspaces
and - preserve affine (or barycentric) coordinates.
- Bijections from X to Y that
- map lines of X onto lines of Y and
- preserve the ratios of signed lengths of
- line segments.
3
In E they are combinations of rigid
transformations, non-uniform scalings and shears.
18Affine Transformations II
- Given two affine spaces X and Y of dimension m,
and two - coordinate frames (A) and (B) for these spaces,
there exists - a unique affine transformation mapping (A) onto
(B).
- Given an affine transformation from X to Y, one
can always write
- When coordinate frames have been chosen for X
and Y, - this translates into
19Affine projections induce affine transformations
from planes onto their images.
20Affine Shape
Two point sets S and S in some affine space X
are affinely equivalent when there exists an
affine transformation y X X such that X
y ( X ).
Affine structure from motion affine shape
recovery.
recovery of the corresponding motion
equivalence classes.
21Geometric affine scene reconstruction from two
images (Koenderink and Van Doorn, 1991).
22Affine Structure from Motion
Reprinted with permission from Affine Structure
from Motion, by J.J. (Koenderink and A.J.Van
Doorn, Journal of the Optical Society of America
A, 8377-385 (1990). ? 1990 Optical Society of
America.
(Koenderink and Van Doorn, 1991)
23The Affine Epipolar Constraint
Note the epipolar lines are parallel.
24Affine Epipolar Geometry
25The Affine Fundamental Matrix
where
26An Affine Trick..
Algebraic Scene Reconstruction
27The Affine Structure of Affine Images
Suppose we observe a scene with m fixed cameras..
The set of all images of a fixed scene is a 3D
affine space!
28has rank 4!
29From Affine to Vectorial Structure
Idea pick one of the points (or their center of
mass) as the origin.
30What if we could factorize D? (Tomasi and
Kanade, 1992)
Affine SFM is solved!
Singular Value Decomposition
We can take
31From uncalibrated to calibrated cameras
Weak-perspective camera
Calibrated camera
Problem what is Q ?
Note Absolute scale cannot be recovered. The
Euclidean shape (defined up to an arbitrary
similitude) is recovered.
32Reconstruction Results (Tomasi and Kanade, 1992)
Reprinted from Factoring Image Sequences into
Shape and Motion, by C. Tomasi and T. Kanade,
Proc. IEEE Workshop on Visual Motion (1991). ?
1991 IEEE.
33More examples
Tomasi Kanade92, Poelman Kanade94
34More examples
Tomasi Kanade92, Poelman Kanade94
35More examples
Tomasi Kanade92, Poelman Kanade94
36Further Factorization work
- Factorization with uncertainty
- Factorization for indep. moving objects (now)
- Factorization for articulated objects (now)
- Factorization for dynamic objects (now)
- Perspective factorization (next week)
- Factorization with outliers and missing pts.
(Irani Anandan, IJCV02)
(Costeira and Kanade 94)
(Yan and Pollefeys 05)
(Bregler et al. 2000, Brand 2001)
(Sturm Triggs 1996, )
(Jacobs 97 (affine), Martinek Pajdla01
Aanaes02 (perspective))
37Structure from motion of multiple moving objects
38Structure from motion of multiple moving objects
39Shape interaction matrix
- Shape interaction matrix for articulated objects
looses block diagonal structure
Costeira and Kanades approach is not usable for
articulated bodies (assumes independent motions)
40Articulated motion subspaces
Motion subspaces for articulated bodies intersect
(Yan and Pollefeys, CVPR05) (Tresadern and Reid,
CVPR05)
Joint (1D intersection)
(jointorigin)
(rank8-1)
Hinge (2D intersection)
(hingez-axis)
(rank8-2)
Exploit rank constraint to obtain better estimate
Also for non-rigid parts if
(Yan Pollefeys, 06?)
41Results
- Toy truck
- Segmentation
- Intersection
- Student
- Segmentation
- Intersection
42Articulated shape and motion factorization
(Yan and Pollefeys, 2006?)
- Automated kinematic chain building for
articulated non-rigid obj. - Estimate principal angles between subspaces
- Compute affinities based on principal angles
- Compute minimum spanning tree
43Structure from motion of deforming objects
(Bregler et al 00 Brand 01)
- Extend factorization approaches to deal with
dynamic shapes
44Representing dynamic shapes
(fig. M.Brand)
represent dynamic shape as varying linear
combination of basis shapes
45Projecting dynamic shapes
(figs. M.Brand)
Rewrite
46Dynamic image sequences
One image
(figs. M.Brand)
Multiple images
47Dynamic SfM factorization?
Problem find J so that M has proper structure
48Dynamic SfM factorization
(Bregler et al 00)
Assumption SVD preserves order and orientation
of basis shape components
49Results
(Bregler et al 00)
50Dynamic SfM factorization
(Brand 01)
constraints to be satisfied for M
constraints to be satisfied for M, use to compute
J
hard!
(different methods are possible, not so simple
and also not optimal)
51Non-rigid 3D subspace flow
(Brand 01)
- Same is also possible using optical flow in stead
of features, also takes uncertainty into account
52Results
(Brand 01)
53Results
(Brand 01)
54Results
(Bregler et al 01)
55Next class Projective structure from motion
Reading Chapter 13