Algorithms

1
Algorithms

• Point correspondences
• Salient point detection
• Local descriptors
• Matrix decompositions
• RQ decomposition
• Singular value decomposition - SVD
• Estimation
• Systems of linear equations
• Solving systems of linear equations
• Direct Linear Transform DLT
• Normalization
• Iterative error / cost minimization
• Outliers ? Robustness, RANSAC
• Pose estimation
• Perspective n-point problem PnP

2
Relevant Issues in Practice

• Poor condition of A ? Normalization
• Algebraic error vs.
• geometric error, ? Iterative minimization
• nonlinearities (lens dist.)
• Outliers ? Robust algorithms
• (RANSAC)

3
Normalization (1)
Homography H

• Entries of A are quadratic in point coordinates
• SVD is not robust against coordinate transform !
• change of coordinate system (translation,
  scaling) will influence result !
• algebraic vs. geometric error !
• Normalization recommended, e.g.
• translate origin (0,0,1) ? image center
• isotropic scaling such that
• either average distance to (0,0,1) is ,
• or average point is (1,1,1)

4
Normalization (2)
Fundamental Matrix F

• note some algorithms use
• eigenvalues of ATA instead
• of singular values (SVD) !

• poor condition of ATA
• Normalization is mandatory
• normalized 8-point algorithm to estimate F
  Hartley95 in defense of the 8-point algorithm

5
Iterative Minimization

• DLT minimizes algebraic error
• geometric distance is more complex
• Lens distortion is non-linear
• Standard approach
• estimate linear parameters by DLT ?
  initialization for
• subsequent iterative minimization over all
  parameters
• E.g. gold standard for estimation of H

6
Gold Standard for Estimation of H
(1)HartleyZisserman
DLT algorithm to estimate H

• Objective
• Given n4 2D to 2D point correspondences
  xi?xi, determine the 2D homography matrix H
  such that xiHxi
• Algorithm
• For each correspondence xi ?xi compute Ai.
  Usually only two first rows needed.
• Assemble n 2x9 matrices Ai into a single 2nx9
  matrix A
• Obtain SVD of A. Solution for h is last column of
  V
• Determine H from h

adapted from Pollefeys course
7
Gold Standard for Estimation of H
(2)HartleyZisserman
normalized DLT algorithm to estimate H

• Objective
• Given n4 2D to 2D point correspondences
  xi?xi, determine the 2D homography matrix H
  such that xiHxi
• Algorithm
• Normalize points
• Apply DLT algorithm to
• Denormalize solution

adapted from Pollefeys course
8
Gold Standard for Estimation of H
(3)HartleyZisserman

• Objective
• Given n4 2D to 2D point correspondences
  xi?xi, determine the Maximum Likelihood
  Estimation of H
• Algorithm
• Initialization compute an initial estimate using
  normalized DLT or RANSAC
• Geometric minimization of either Sampson error
• ? Minimize the Sampson error
• ? Minimize using Levenberg-Marquardt over 9
  entries of h
• or Gold Standard error
• ? compute initial estimate for subsidiary
• ? minimize cost
  over
• ? if many points, use sparse method

adapted from Pollefeys course
9
Robust Estimation (RANSAC) HartleyZisserman
Handling of outliers !
RANSAC RANdom Sample Consensus
10
RANSAC Algorithm HartleyZisserman

• Objective
• Robust fit of model to data set S which contains
  outliers
• Algorithm
• Randomly select a sample of s data points from S
  and instantiate the model from this subset.
• Determine the set of data points Si which are
  within a distance threshold t of the model. The
  set Si is the consensus set of samples and
  defines the inliers of S.
• If the subset of Si is greater than some
  threshold T, re-estimate the model using all the
  points in Si and terminate
• If the size of Si is less than T, select a new
  subset and repeat the above.
• After N trials the largest consensus set Si is
  selected, and the model is re-estimated using all
  the points in the subset Si

adapted From Pollefeys course
11
RANSAC Algorithm HartleyZisserman
sample size vs. proportion of outliers
adapted from Pollefeys course
12
More Problems

• critical cases ! e.g. TorrMurray, IJCV 1997

13
Pose Estimation

• Camera

• calibrated camera, known K

yC

• determine camera pose R, t

xC
R, t
C
zC
R, t
Z
yV
xV
Y
zV
X

• Visualization (screen, HMD)

14
Perspective n-Point Problem PnP (1)
Pa
pa
C
pb
dab
?

• Calibrated camera
• Known K
• Known points Pi in the scene
• Given n point correspondences
• pi ? Pi
• What can be measured with one calibrated camera?
• ? angle ? between two lines of sight

Pb
15
Perspective n-Point Problem PnP (2)
Pa
pa
C
pb
dab
?

• PnP uses just this information
• P3P will give up to 4 solutions
• P4P is already overdetermined
• Perform 4 x P3P
• Find consensus

Pb
16
Pose Estimation ? Tracking

• In theory, tracking is simple !
• Calibrate your camera (K)
• Measure some points Pi in the scene (ground
  truth)
• Perform pose estimation in real-time (for each
  frame)
• In practice, tracking is a hard problem !
• Point detection
• Correspondence
• Motion prediction
• Occlusion
• Unknown scene
• Many solutions have been proposed !

Tracking beyond 15 minutes of thought SIGGRAPH
2001 Turorial 15 Allen, Bishop, Welch
An introduction to the Kalman filter SIGGRAPH
2001 Turorial 8 Welch, Bishop
17
Tracking Systems (vision-based / hybrid)some of
my own contributions (1)
Hybrid inside out magnetic stereo
vision Auer 1999
18
Tracking Systems (vision-based / hybrid)some of
my own contributions (2)
stereo vision outside in Ribo ca. 2000
19
Tracking Systems (vision-based / hybrid)some of
my own contributions (3)
inertial inside out hybrid inertal
vision many 2000-2004
20
Tracking Systems (vision-based / hybrid)some of
my own contributions (4)
vision (stereo or mono) inside out speed solves
correspondence ! Mühlmann 2005
21
Our Current View Schweighofer 2008
stereo vision inside out structure and motion
22
Summary

• In these four lectures, I gave an introduction
  to
• Projective geometry
• Perspective cameras
• Homographies, camera projection matrices,
  fundamental and essential matrices
• Algorithms that are typically applied to solve
  for
• Camera calibration
• Stereo reconstruction
• Camera pose estimation
• I consider this the basis for further reading in
  topics including
• Vision-based pose tracking
• Structure and motion analysis (sometimes termed
  SLAM)
• Many aspects were, of course, not covered, but
  would also be important !

23
What could not be covered ?

• Self calibration (see Pollefeys, absolute
  conic,)
• Bundle adjustment
• Levenberg-Marquardt
• The full presentation of algorithms for the
  estimation of H, P, K, F,
• see the Hartley, Zisserman book for all about
  multiple view geometry
• Tracking in general, Kalman filter (two UNC
  Siggraph 2001Tutorials)
• Several prominent variants of vision-based
  tracking algorithms/systems
• KLT
• Rapid, RoRapid
• Condensation, ICondensation
• Lu, Hager
• Ansar, Daniilidis
• Wunsch, Hirzinger
• Klein, Murray

• Another reference to Pollefeys
• http//www.cs.unc.edu/marc/tutorial/node159.html

• interested in more detail ?
• 2 VO Image based measurement WS
• 1 LU Image based measurement SS
• seminar, project, bachelor, diploma, PhD,