3D Urban Modeling

1
3D Urban Modeling

• Marc Pollefeys
• Jan-Michael Frahm, Philippos Mordohai
• Fall 2006 / Comp 790-089
• Tue Thu 1400-1515
• SN252

2
3D Urban Modeling

• Basics sensor, techniques,
• Video, LIDAR, GPS, IMU,
• SfM, Stereo,
• Robot Mapping, GIS,
• Review state-of-the-art
• Video and LIDAR-based systems
• Projects to experiment
• Virtual UNC-Chapel Hill
• DARPA Urban Challenge

3
3D Urban Modeling course schedule(tentative)
Sep. 5, 7 Introduction Video-based Urban 3D Capture (Jan-Michael)
Sep. 12, 14 Cameras Epi. Geom. and Triangulation
Sep. 19, 21 Feature Tracking Matching (Sudipta) Stereo matching (Philippos)
Sep. 26, 28 GPS/INS/LIDAR (Brian) Project proposals
Oct. 3, 5
Oct. 10, 12
Oct. 17, 19 (fall break)
Oct. 24, 26 Project Status Update
Oct. 31, Nov. 2
Nov. 7, 9
Nov. 14, 16
Nov. 21, 23 (Thanksgiving)
Nov. 28, 30
Dec. 5 Project demonstrations (classes ended)
Note Dec. 3 is CVPR deadline
4
Course material

• Slides, notes, papers and references
• see course webpage/wiki (later)
• On-line shape-from-video tutorial
• http//www.cs.unc.edu/marc/tutorial.pdf
• http//www.cs.unc.edu/marc/tutorial/

5
Epipolar geometry?
courtesy Frank Dellaert
6
Epipolar geometry
Underlying structure in set of matches for rigid
scenes

1. Computable from corresponding points
2. Simplifies matching
3. Allows to detect wrong matches
4. Related to calibration

7
Epipolar geometry
8
The projective reconstruction theorem
If a set of point correspondences in two views
determine the fundamental matrix uniquely, then
the scene and cameras may be reconstructed from
these correspondences alone, and any two such
reconstructions from these correspondences are
projectively equivalent
allows reconstruction from pair of uncalibrated
images!
9
Computation of F

• Linear (8-point)
• Minimal (7-point)
• Robust (RANSAC)
• Non-linear refinement (MLE, )
• Practical approach

10
Epipolar geometry basic equation
separate known from unknown
(data)
(unknowns)
(linear)
11
the NOT normalized 8-point algorithm
12
the normalized 8-point algorithm

• Transform image to -1,1x-1,1

normalized least squares yields good results
(Hartley, PAMI97)
13
the singularity constraint
SVD from linearly computed F matrix (rank 3)
Compute closest rank-2 approximation
14
(No Transcript)
15
the minimum case 7 point correspondences
one parameter family of solutions
but F1lF2 not automatically rank 2
16
the minimum case impose rank 2
(obtain 1 or 3 solutions)
(cubic equation)
Compute possible l as eigenvalues of (only real
solutions are potential solutions)
Minimal solution for calibrated cameras 5-point
17
Robust estimation

• What if set of matches contains gross outliers?
• (to keep things simple lets consider line
  fitting first)

18
RANSAC(RANdom Sampling Consensus)

• Objective
• Robust fit of model to data set S which contains
  outliers
• Algorithm
• Randomly select a sample of s data points from S
  and instantiate the model from this subset.
• Determine the set of data points Si which are
  within a distance threshold t of the model. The
  set Si is the consensus set of samples and
  defines the inliers of S.
• If the subset of Si is greater than some
  threshold T, re-estimate the model using all the
  points in Si and terminate
• If the size of Si is less than T, select a new
  subset and repeat the above.
• After N trials the largest consensus set Si is
  selected, and the model is re-estimated using all
  the points in the subset Si

19
Distance threshold

• Choose t so probability for inlier is a (e.g.
  0.95)
• Often empirically
• Zero-mean Gaussian noise s then follows
• distribution with mcodimension of model

(dimensioncodimensiondimension space)
Codimension Model t 2
1 line,F 3.84s2
2 H,P 5.99s2
3 T 7.81s2
20
How many samples?

• Choose N so that, with probability p, at least
  one random sample is free from outliers. e.g.
  p0.99

proportion of outliers e proportion of outliers e proportion of outliers e proportion of outliers e proportion of outliers e proportion of outliers e proportion of outliers e
s 5 10 20 25 30 40 50
2 2 3 5 6 7 11 17
3 3 4 7 9 11 19 35
4 3 5 9 13 17 34 72
5 4 6 12 17 26 57 146
6 4 7 16 24 37 97 293
7 4 8 20 33 54 163 588
8 5 9 26 44 78 272 1177
Note Assumes that inliers allow to identify
other inliers
21
Acceptable consensus set?

• Typically, terminate when inlier ratio reaches
  expected ratio of inliers

22
Adaptively determining the number of samples

• e is often unknown a priori, so pick worst case,
  i.e. 0, and adapt if more inliers are found, e.g.
  80 would yield e0.2
• N8, sample_count 0
• While N gtsample_count repeat
• Choose a sample and count the number of inliers
• Set e1-(number of inliers)/(total number of
  points)
• Recompute N from e
• Increment the sample_count by 1
• Terminate

23
Other robust algorithms

• RANSAC maximizes number of inliers
• LMedS minimizes median error
• Not recommended case deletion, iterative
  least-squares, etc.

inlier percentile
100
50
residual (pixels)
1.25
24
Non-linear refinment
25
Geometric distance
Gold standard Symmetric epipolar distance
26
Gold standard
Maximum Likelihood Estimation
( least-squares for Gaussian noise)
Initialize normalized 8-point, (P,P) from F,
reconstruct Xi
Parameterize
(overparametrized)
Minimize cost using Levenberg-Marquardt (preferabl
y sparse LM, e.g. see HZ)
27
Gold standard
Alternative, minimal parametrization (with a1)
(note (x,y,1) and (x,y,1) are epipoles)

• problems
• a0

? pick largest of a,b,c,d to fix to 1

• epipole at infinity

? pick largest of x,y,w and of x,y,w
4x3x336 parametrizations!
reparametrize at every iteration, to be sure
28
ZhangLoops approach CVIU01
29
First-order geometric error (Sampson error)
(one eq./point ?JJT scalar)
(problem if some x is located at epipole)
advantage no subsidiary variables required
30
Symmetric epipolar error
31
Some experiments
32
Some experiments
33
Some experiments
34
Some experiments
Residual error
(for all points!)
35
Recommendations

1. Do not use unnormalized algorithms

• Quick and easy to implement 8-point normalized

• Better enforce rank-2 constraint during
  minimization

• Best Maximum Likelihood Estimation (minimal
  parameterization, sparse implementation)

36
The envelope of epipolar lines
What happens to an epipolar line if there is
noise?
Monte Carlo
n50
n25
n15
n10
37
Automatic computation of F

• Step 1. Extract features
• Step 2. Compute a set of potential matches
• Step 3. do
• Step 3.1 select minimal sample (i.e. 7 matches)
• Step 3.2 compute solution(s) for F
• Step 3.3 determine inliers
• until ?(inliers,samples)lt95

Step 4. Compute F based on all inliers Step 5.
Look for additional matches Step 6. Refine F
based on all correct matches
inliers 90 80 70 60 50
samples 5 13 35 106 382
38
Abort verification early
O O O O O I O O I O O O O
I I I O I I I I O O I O I I I I I I O I I I I I I
O I O O I I I I
O O O O O O O
O O O O I O O O O

• Assume percentage of inliers p, what is the
  probability P to pick n or more wrong samples out
  of m?
• stop if Plt0.05 (sample most probably contains
  outlier)
• (Pcum. binomial distr. funct. (m-n,m,p ))
• To avoid problems this requires to also verify
  at random!
• (but we already have a random sampler anyway)
• (inspired from Chum and Matas BMVC2002)

39
Finding more matches
restrict search range to neighborhood of
epipolar line (e.g. ?1.5 pixels) relax
disparity restriction (along epipolar line)
40
Degenerate cases

• Degenerate cases
• Planar scene
• Pure rotation
• No unique solution
• Remaining DOF filled by noise
• Use simpler model (e.g. homography)
• Solution 1 Model selection
• (Torr et al., ICCV98, Kanatani, Akaike)
• Compare H and F according to expected residual
  error (compensate for model complexity)
• Solution 2 RANSAC
• Compare H and F according to inlier count
• (see next slide)

41
RANSAC for quasi-degenerate cases

• Full model (8pts, 1D solution)
• Planar model (6pts, 3D solution)
• Accept if large number of remaining inliers
• Planeparallax model (plane2pts)

(accept inliers to solution F)
(accept inliers to solution F1,F2F3)
closest rank-6 of Anx9 for all plane inliers
Sample for out of plane points among outliers
42
More problems

• Absence of sufficient features (no texture)
• Repeated structure ambiguity

• Robust matcher also finds
• support for wrong hypothesis
• solution detect repetition

(Schaffalitzky and Zisserman, BMVC98)
43
RANSAC for ambiguous matching

• Include multiple candidate matches in set of
  potential matches
• Select according to matching probability (
  matching score)
• Helps for repeated structures or scenes with
  similar features as it avoids an early commitment

(Tordoff and Murray ECCV02)
44
two-view geometry

• geometric relations between two views is fully
• described by recovered 3x3 matrix F

45
Triangulation
x1
C1
L1
Triangulation

• calibration
• correspondences

46
Triangulation

• Backprojection

• Triangulation

Iterative least-squares

• Maximum Likelihood Triangulation

47
Optimal 3D point in epipolar plane

• Given an epipolar plane, find best 3D point for
  (m1,m2)

Select closest points (m1,m2) on epipolar
lines Obtain 3D point through exact
triangulation Guarantees minimal reprojection
error (given this epipolar plane)
48
Non-iterative optimal solution

• Reconstruct matches in projective frame by
  minimizing the reprojection error
• Non-iterative method
• Determine the epipolar plane for reconstruction
• Reconstruct optimal point from selected epipolar
  plane
• Note only works for two views

3DOF
(Hartley and Sturm, CVIU97)
(polynomial of degree 6)
1DOF