Multi-view Stereo via Volumetric Graph-cuts

1
Multi-view Stereo via Volumetric Graph-cuts

• George Vogiatzis
• Roberto CipollaCambridge Univ. Engineering Dept.

Philip H. S. Torr Department of
ComputingOxford Brookes University
2
Multi-view Dense Stereo
Calibrated images of Lambertian scene
3D model of scene
3
Multi-view Dense Stereo

• Two main approaches
• Volumetric
• Disparity (depth) map

4
Dense Stereo reconstruction problem

• Two main approaches
• Volumetric
• Disparity (depth) map

Disparity-map
5
Shape representation

• Disparity-maps
• MRF formulation good optimisation techniques
  exist (Graph-cuts, Loopy BP)
• MRF smoothness is viewpoint dependent
• Disparity is unique per pixel only functions
  represented

6
Shape representation

• Volumetric e.g. Level-sets, Space carving etc.
• Able to cope with non-functions
• Levelsets Local optimization
• Space carving no simple way to impose surface
  smoothness

7
Our approach

• Cast volumetric methods in MRF framework
• Use approximate surface containing the real scene
  surface
• E.g. visual hull
• Benefits
• General surfaces can be represented
• No depth map merging required
• Optimisation is tractable (MRF solvers)
• Smoothness is viewpoint independent

8
Volumetric Graph cuts for segmentation
Boykov and Jolly ICCV 2001

• Volume is discretized
• A binary MRF is defined on the voxels
• Voxels are labelled as OBJECT and BACKGROUND
• Labelling cost set by OBJECT / BACKGROUND
  intensity statistics
• Compatibility cost set by intensity gradient

9
Volumetric Graph cuts for stereo

• Challenges
• What do the two labels represent
• How to define cost of setting them
• How to deal with occlusion
• Interactions between distant voxels

10
Volumetric Graph cuts
1. Outer surface
2. Inner surface (at constant offset)
3. Discretize middle volume
4. Assign photoconsistency cost to voxels
11
Volumetric Graph cuts
Source
Sink
12
Volumetric Graph cuts
cut ? 3D Surface S
Cost of a cut ? ?? ?(x) dS
Source
Boykov and Kolmogorov ICCV 2001
S
S
Sink
13
Volumetric Graph cuts
Minimum cut ? Minimal 3D Surface under
photo-consistency metric
Source
Boykov and Kolmogorov ICCV 2001
Sink
14
Photo-consistency

• Occlusion

1. Get nearest point on outer surface
2. Use outer surface for occlusions
2. Discard occluded views
15
Photo-consistency

• Occlusion

Self occlusion
16
Photo-consistency

• Occlusion

Self occlusion
17
Photo-consistency

• Occlusion

N
18
Photo-consistency
Normalised cross correlation Use all remaining
cameras pair wise Average all NCC scores

• Score

19
Photo-consistency
Average NCC C Voxel score ? 1 - exp(
-tan2?(C-1)/4 / ?2 )

• Score

0 ? ? ? 1 ? 0.05 in all experiments
20
Example
21
Example - Visual Hull
22
Example - Slice
23
Example - Slice with graphcut
24
Example 3D
25
Protrusion problem

• Balooning force
• favouring bigger volumes that fill the visual
  hull
• L.D. Cohen and I. Cohen. Finite-element methods
  for active contour models and balloons for 2-d
  and 3-d images. PAMI, 15(11)11311147, November
  1993.

26
Protrusion problem
?? ?(x) dS - ? ??? dV

• Balooning force
• favouring bigger volumes that fill the visual
  hull
• L.D. Cohen and I. Cohen. Finite-element methods
  for active contour models and balloons for 2-d
  and 3-d images. PAMI, 15(11)11311147, November
  1993.

27
Protrusion problem
28
Protrusion problem
29
Graph
wij 4/3?h2 (?i?j)/2
Boykov and Kolmogorov ICCV 2001
wb ?h3
i
j
30
Results

• Model House

31
Results

• Model House Visual Hull

32
Results

• Model House

33
Results

• Stone carving

34
Results

• Haniwa

35
Summary

• Novel formulation for multiview stereo
• Volumetric scene representation
• Computationally tractable global optimisation
  using Graph-cuts.
• Visual hull for occlusions and geometric
  constraint
• Occlusions approximately modelled

Questions ?