MultiView Stereo through Feature Matching and Expansion

1
Multi-View Stereo through Feature Matching and
Expansion

• Yasutaka FurukawaUniversity of Illinois at
  Urbana-Champaign
• Jean Ponce
• École Normale Supérieure

2
Multi-View Stereo Algorithms

• Takes a set of calibrated photographs
• Reconstruct
• An object
• A scene
• A movie-shot

3
Classification by Surface Representations

• Polygonal MeshesHernandez 2004, Furukawa 2006
• VoxelsFaugeras 1997, Seitz 1997, Pons 2005,
  Vogiatzis 2005, Tran 2006, Hornung 2006
• Multiple Depth MapsKolmogorov 2002, Goesele
  2006, Strecha 2006
• Surfels (Patches)Lhuillier 2005, Habbecke 2006,
  This work

4
Patch

• Patch p consists of its
• 3D coordinate c(p)
• Surface normal n(p)

5
Inputs Outputs

• Inputs
• Calibrated photographs
• Outputs
• Patches (Oriented point sets)
• Polygonal surface with a post-processing

48 images1750x1100
6
Patch Model
4 components Pp1,p2,p3, I(pi) Ir(pi) It(pi)
7
Patch Model

• Given a set of patches P p1, p2, ,pn
• P determinesI(pi) a set of images in which pi
  is visible
• pi has a reference image Ir(pi)
• Ir(pi) determinesIt(pi) Ir(pi) every image J
  in I(pi) that satisfies

8
Patch Model Objective

• Find PI(pi) and Ir(pi) It(pi) that maximize
  S(P) under some constraints (next slide)

9
Patch Model Constraints

• -
• N(P) number of occupied cells

Image0
Image1
Image2
10
How do you find the optimal solution for such a
problem?

• No, score is not optimal
• But, constraints are satisfied
• We use heuristics
• Overestimate I(pi) It(pi)Assume that a patch
  is visible in more images
• Use filtering to remove inconsistent images in
  I(pi) and It(pi) and patches

11
Algorithm

• Feature Detection (Harris DoG)
• Initial Feature Matching to generate patch
  candidates
• Patch Expansion and Filtering

12
Algorithm

• Feature Detection (Harris DoG)
• Initial Feature Matching to generate patch
  candidates
• Patch Expansion and Filtering

13
Feature Detection

• Extract local maxima of
• Harris Corner Detector (corners)
• Difference of Gaussian (blobs)

14
Algorithm

• Feature Detection (Harris DoG)
• Initial Feature Matching to generate patch
  candidates
• Patch Expansion and Filtering

15
Initial Feature Matching
16
Initial Feature Matching
17
Initial Feature Matching
18
Initial Feature Matching
19
Patch Optimization

• Patch Optimization
• 1 dof for depth with respect to Ir(pi)
• 2 dof for orientation
• Optimize depth and orientation while maximizing
  s(pi)

20
Algorithm

• Feature Detection (Harris DoG)
• Initial Feature Matching to generate patch
  candidates
• Patch Expansion and Filtering

21
Patch Expansion
22
Patch Expansion
23
Patch Expansion
Optimize
24
Patch Expansion
25
Filtering

• Omitting details
• A filter to
• remove inconsistent images from It(p)
• remove patches

26
Reconstructed Patches
16 images 640x480 35 minutes computation time
16 images 640x480 20 minutes computation time
Thanks to Steve Seitz, Brian Curless, James
Diebel, Daniel Scharstein, and Richard Szeliski
27
Reconstructed Patches
7 images 1500x1000 1 hour computation time Thanks
to Strecha http//homes.esat.kuleuven.be/ cstrec
ha/demos/3d/
28
From Patches to Polygonal Surfaces

• Manifold reconstruction from oriented point set
  Kazhdan 2005
• Voxel based approach Curless 1996, Goesele,
  2006
• Iterative deformation Furukawa 2006
• Direct triangulation

29
Part of Data Sets
30
36 images1700x2100
24 images2000x2000
31
More Results
32
7 images 1500x1000 Thanks to Strecha http//homes.
esat.kuleuven.be/cstrecha/demos/3d/
16 images 640x480 Thanks to Steve Seitz, Brian
Curless, James Diebel, Daniel Scharstein, and
Richard Szeliski
33
7 images 1500x1000
34
Works on Weak Features
13 images1500x1000 Thanks to Alex
Sutter and Industrial Light Magic
35
Results with Outliers
3 images 700x1000 Thanks to Strecha
http//homes.esat.kuleuven.be/ cstrecha/demos/3d/

36
Results with Obstacles
37
Results with Obstacles
38
ComparisonsLaser Range Scanner, Ours, Hernandezs
39
Conclusions

• Pros
• Can handle challenging shapes (sharp concavities
  with weak textures)
• Little regularization or smoothness constraints
• Does not requite multi-resolution framework
• Can handle obstacles and outliers very well
• No initialization needed
• Directly computing a global model
• No discretization errors (fully continuous
  optimization)
• Few false positives
• Cons
• Need one more step to obtain a mesh
• Little regularization or smoothness constraints

40
Future Work

• Better manifold generation algorithmfrom patches
• Enforce stronger regularization
• Multiple Synchronized Camerasdynamic scenes
• Marker-less facial mocap

41
Acknowledgements

• Steve Seitz, Brian Curless, James Diebel, Daniel
  Scharstein, and Richard Szeliski for datasets and
  evaluations
• Carlos Hernandez Esteban, Francis Schmitt, and
  Museum of Cherbourg for datasets and a laser
  scanned model
• Jean-Marc Lavest for a calibration software
• Alex Sutter and Industrial Light Magic for
  datasets
• National Science Foundation IIS-0312438

Thank You
42
(No Transcript)
43
(No Transcript)
44
Filtering Outer Erroneous Patches

• To resolve inconsistencies in I(p), either
• Remove a red patch
• Remove an image from It(p) and I(p) from each
  green patch
• Compare loss of s(p) and decide

45
Another Quantitative Evaluation
Ours Hernandezs
Ours Laser Scanner
Hernandezs Laser Scanner
46
Quantitative Evaluations
Thanks to Steve Seitz, Brian Curless, James
Diebel, Daniel Scharstein, and Richard Szeliski
47
Why Patches?

• Seems to work pretty well with weak textures

36 images1700x2100 Thanks toCarlos Hernandez
Esteban, Francis Schmitt
48 images1750x1100
48
(No Transcript)
49
Fitting Visual Hull to Patches

• Snapping a visual hull to patches
• Minimize distances between Visual Hull and
  reconstructed patches

Visual Hull
v
Patches
50
Fittint Visual Hull to Patches
51
(No Transcript)
52
Comparisons with Multiple-Depth Maps
Representation

• Efficient memory usage no need to store lots of
  information at each pixel
• Handle outliers easily
• Multiple Depth Maps requires 2image labels
• Running time of an algorithm is rather
  proportional to the size of an output (not
  proportional to the input size)

53
Why Patches?
36 images 1200x2800 Thanks toCarlos Hernandez
Esteban, Francis Schmitt
7 images 1500x1000
54
Part of Data Sets