Reconstructing Relief Surfaces

1
Reconstructing Relief Surfaces

• George Vogiatzis, Philip Torr, Steven Seitz and
  Roberto Cipolla
• BMVC 2004

2
Stereo reconstruction problem

• Input
• Set of images of a scene II1,,IK
• Camera matrices P1,,PK
• Output
• Surface model

3
Shape parametrisation

• Disparity-map parametrisation
• MRF formulation good optimisation techniques
  exist (Graph-cuts, Loopy BP)
• MRF smoothness is viewpoint dependent
• Disparity is unique per pixel only functions
  represented

4
Shape parametrisation

• Volumetric parametrisation e.g. Level-sets,
  Space carving etc.
• Able to cope with non-functions
• Convergence properties not well understood, Local
  minima
• Memory intensive
• For Space carving, no simple way to impose
  surface smoothness

5
Solution ?

• Cast volumetric methods in MRF framework
• Key assumption Approximate scene geometry given
• Benefits
• General surfaces can be represented
• Optimisation is tractable (MRF solvers)
• Occlusions are approximately modelled
• Smoothness is viewpoint independent

6
MRFs

• The labelling problem

7
MRFs

• A set of random variables h1,,hM
• A binary neighbourhood relation N defined on the
  variables
• Each can take a label out of a set H1,,HL
• Ci(hi) (Labelling cost)
• Ci,j(hi,hj) for (i,j)?N (Compatibility cost)
• -log P(h1,,hM) ? Ci(hi) ? Ci,j(hi,hj)

8
MRF inference

• Minimise ? Ci(hi) ? Ci,j(hi,hj)
• Not in polynomial time in general case
• Special cases (e.g. no loops or 2 label MRF)
  solved exactly
• General cases solved approximately via Graph-cuts
  or Loopy Belief Propagation. Approx. 10-15mins
  for MRF with 250,000 nodes.

9
Relief Surfaces

• Approximate base surface
• Triangulated feature matches
• Visual hull from silhouettes
• Initialised by hand

10
Relief Surfaces
11
Relief Surfaces
Xihini
ni
Xi
Ci(hi)photoconsistency(Xihini)
12
Relief Surfaces
Compatibility cost
Xjhjnj
Low cost
Xihini
nj
Xj
ni
Xi
13
Relief Surfaces
Neighbour cost
Xihini
High cost
Xjhjnj
ni
Xi
Ci,j(hi, hj) (Xihini)-(Xjhjnj)
14
Relief Surfaces

• Base surface is the occluding volume
• If base surface contains true surface (e.g.
  visual hull) then
• Points on the base surface Xi are not visible by
  cameras they shouldnt be Kutulakos, Seitz 2000
• Approximation
• Visibility is propagated from Xi to Xihini

15
Loopy Belief Propagation
min ? Ci(hi) ? Ci,j(hi,hj)

• Iterative message passing algorithm
• m(t)i,j (hj) is the message passed from i to j at
  time step t
• It is a L-dimensional vector
• Represents what node i believes about the true
  state of node j.

16
Loopy Belief Propagation

• Message passing rule
• After convergence, optimal state is given by

17
Loopy Belief Propagation

• O(L2) to compute a message (L is number of
  allowable heights)
• Message passing schedule can be asynchronous
  which can accelerate convergence Tappen
  Freeman ICCV 03

18
Iterative Scheme

• BP is memory intensive.
• Can consider few possible labels at a time
• After convergence we zoom in to heights close
  to the optimal

19
Evaluation

• Artificial deformed sphere
• Textured with random patern
• 20 images
• 40,000 sample points on sphere base surface

20
Evaluation

• Benchmark 2-view, disparity based Loopy Belief
  Propagation Sun et al ECCV02
• BP run on 10 pairs of nearby views
• Compare Disparity Maps given by
• 2-view BP
• Relief surfaces
• Ground truth

21
Evaluation
22
Results

• Sarcophagus

23
Results

• Sarcophagus

24
Results

• Sarcophagus

25
Results

• Building facade

26
Results

• Building facade

27
Results

• Stone carving

Relief surface with texture
Base surface
Relief surface
28
Summary

• MRF methods can be extended in the volumetric
  domain
• Advantages
• General surfaces can be represented
• Optimisation is tractable (MRF solvers)
• Smoothness is viewpoint independent

29
Future work

• Photoconsistency beyond Lambertian surface
  models. (Optimise both height and surface normal
  fields)
• Change in topology
• In cases where Cmn(hm,hn) hm-hn or
  hm-hn2 we can compute messages in O(L) time
  instead of O(L2) (Felzenszwalb Huttenlocher
  CVPR 04).