Accurate, Dense and Robust Multi-View Stereopsis

1
Accurate, Dense and Robust Multi-View Stereopsis

• Yasutaka Furukawa and Jean Ponce

Presented by Rahul Garg and Ryan Kaminsky
2
Agenda

• Problem Statement
• Multi-view Stereo Taxonomy
• Algorithm
• Results
• Comparison to other works
• Questions

3
Problem Statement

• Multi-view Stereo
• Dense shape reconstruction from multiple views

4
Multi-View Stereo Taxonomy
S. M. Seitz, B. Curless, J. Diebel, D.
Scharstein, and R. Szeliski

• Scene Representation
• Photoconsistency Measure
• Visibility Model
• Shape Prior
• Reconstruction algorithm
• Initialization

5
Scene Representation

• Geometry on 3D grid
• Voxels, Level sets
• Polygon Mesh
• Set of planar facets
• Depth Map
• Image that stores depth
• per pixel

6
Photoconsistency Measure

• Definition Measures visual compatibility of
  reconstruction with input images
• Scene Space
• Project part of reconstruction into images,
  measure closeness
• Measures Variance , sum of squared distances,
  normalized cross-correlation
• Image Space
• Use scene geometry to transform image to
  different view, measure error of predicted vs.
  actual (prediction error)

7
Visibility Model

• Definition Views to consider when evaluating
  photo consistency
• Geometric
• Explicitly model geometry of the scene
• Quasi-Geometric
• Approximate geometric reasoning
• Outlier based approaches
• Treat occlusions as outliers

8
Shape Prior

• Definition Additional constraints or assumptions
  about reconstruction
• Minimal Surfaces
• Level sets, Min-cut
• Maximal Surfaces
• Voxel coloring, space carving
• Local Measures
• Assume local smoothness on nearby pixels

9
Reconstruction Algorithm

• Optimize cost function
• Voxels, graph cut, level sets, meshes
• A set of consistent depth maps
• Feature extraction, matching, surface fitting

10
Initialization

• Definition Constraints on scene geometry
• Bounding box or volume
• Visual hull
• Range of disparity

11
Overview of Algorithm
input image detected
reconstructed final patches polygonal
surface features
patches after after expansion
from reconstructed the initial
and filtering patches

matching


12
Algorithm Block Diagram
Initialization
Expansion
Filter
Reconstruction
Patch Model
Feature Detection
13
Init

• Detect features using Harris Corner and DoG
• Feature matching to generate sparse set of patches

14
Patch Models

• R(p) Most closely associated image with p
• S(p) Images where p should be visible
• T(p) Images where p is truly visible

15
c(p) from triangulation
n(p) Direction of optical ray from c(p) to O
ß pixels


















Epipolar line
16
Normalized Cross Correlation (NCC)
where   is the mean of the feature and     
is the mean of f(x,y) in the region under the
feature.
Optimization step Maximizing the average NCC
score
17
Patch Expansion

• Expand patches along tangential planes into empty
  areas.

• Optimize for normal and center and add if
  photometric constraints are satisfied in at least
  k images.

18
Filtering

• Analyzing visibility consistency

19
Filtering (Contd.)

• Local smoothness constraint Remove patches for
  which proportion of neighboring patches with
  tangential plane nearly parallel is less than e

20
Polygonal Surface Reconstruction

• Initialize using convex hull of patches
• Iteratively deform/snap to the patch model using
  two kinds of forces
• Smoothness term
• Photometric Consistency term

S Current surface S True surface n(v)
Normal at v ?(v) Set of patches compatible
with v d(v) Distance between S and S
21
Algorithm Taxonomy Categorization

• Scene Representation
• Depth Map Mesh
• Photoconsistency Measure
• NCC
• Shape Prior
• Assume local smoothness
• Reconstruction
• Feature extraction , depth maps, optimization
  over patches
• Initialization
• None

22
Results
Patch Model
Polygonal Surface Model
23
Results (Contd.)
24
Results (Contd.)

• Evaluation on vision.middlebury.edu

Temple ( of views) Temple ( of views) Temple ( of views) Dino ( of views) Dino ( of views) Dino ( of views)
Full (312) Ring (47) Sparse (16) Full (312) Ring (47) Sparse(16)
This paper 0.54 0.55 0.62 0.32 0.33 0.42
Goesele et. al. 0.42 0.61 0.87 0.46 0.46 0.56
Hernandez et. al. 0.36 0.52 0.75 0.49 0.45 0.60
Accuracy Measure Distance d in mm which brings
90 of the reconstruction within ground truth
Old Results
25
Results (Contd.)

• Handle occlusions/obstacles

26
Similar Approaches

• Setup similar to Goesele et al. (ICCV07)
  initialize patches, expand and optimize for
  position and normal

This Paper Goesele et. al.
Initialize patches using triangulated points Initialize using Structure from Motion features
Explicit occlusion handling Occlusion handling through outlier removal and view selection, prioritize patch candidates for expansion
27
Questions

• Pose the problem as an optimization problem
  simultaneously accounting for local smoothness,
  photo consistency, occlusion
• Convergence of Expand/Filter do more iterations
  lead to better reconstructions?
• Occlusion/Outlier handling results on more
  datasets
• Advantages of patch model Adaptive Resolution,
  generalizes to large number of object classes