Comparison of Energy Minimization Algorithms for highly connected graphs

1
Comparison of Energy Minimization Algorithms for
highly connected graphs

• Vladimir Kolmogorov Carsten Rother
  University College London Microsoft
  Research
  Cambridge

2
Motivation
4-connected MRF (stereo, no occlusions)
Highly connected MRF (stereo
with occlusions)
Graph cuts
Szeliski et al., ECCV06

• TRW outperforms graph cuts and BP!
• Can compute global minimum
  Meltzer,Yanover,Weiss05

3
Motivation (cont.)

• Complex MRFs (highly connected)
• Important vision problem
• Whats the best energy minimization algorithm?
• How close we are to the global minimum?
• Algorithms drawbacks become more apparent

4
Outline

• I Defining the Problem Highly connected MRF
  (stereo with occlusions)
• II MRF Optimization Methods Efficient
  Message Passing
• III Experiments BP, TRW and GC
• IV Conclusions discussion

5
Stereo without occlusions simple MRF model
Right view
Labelling, left view
Left view
Energy
coherence cost (e.g. Potts)
matching cost Intensity(q)-Intensity(p)2
6
Stereo with occlusions complex MRF model
Kolmogorov,Zabih ECCV02
Right view
Labelling, left view
Left view
Labelling, right view
Energy
7
Stereo with occlusions complex MRF model
Kolmogorov,Zabih ECCV02
Right view
Labelling, left view
Left view
Labelling, right view
Energy
coherence
matching costs visibility constraints
8
Stereo with occlusions complex MRF model
Kolmogorov,Zabih ECCV02
Right view
Labelling, left view
Left view
Labelling, right view
Energy
p
k
q
9
Stereo with occlusions complex MRF model
Kolmogorov,Zabih ECCV02
Right view
Labelling, left view
Left view
Labelling, right view
Energy
p
k
p
q
q
10
Stereo with occlusions complex MRF model
Kolmogorov,Zabih ECCV02
Right view
Labelling, left view
Left view
Labelling, right view
Energy
p
k
p
q
q
11
Stereo with occlusions complex MRF model
Kolmogorov,Zabih ECCV02
Right view
Labelling, left view
Left view
Labelling, right view
Energy
p
k
p
q
q
12
Stereo with occlusions complex MRF model
Kolmogorov,Zabih ECCV02
Right view
Labelling, left view
Left view
Labelling, right view
Energy
p

• Mpq 0
• encorages matches
• discourages occlusions

q
13
MRF Optimization Methods

• Graph cuts (GC)
• Alpha expansion (as used in Kolmogorov, Zabih
  ECCV02)
• Max-product loopy belief propagation (BP)
• Max-product tree-reweighted message passing (TRW)

I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
14
Tree-reweighted message passing (TRW)Wainwright
et al. 02

• Main idea
• Split the graph into trees
• Select probability distribution over trees
• Iteratively apply two operations
• Run BP on each tree
• Average beliefs between different trees
• Sequential TRW (TRW-S) Kolmogorov05
• Specific choice of trees the order of
  operations
• Convergence guarantees

I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
15
Message passing sequential schedule

• Select ordering of nodes
• Pass messages according
  to this ordering
• Forward pass
• Backward pass
• Needs half the memory Kolmogorov05
• Generalization of technique in FelzenszwalbHutte
  nlocher04

1
2
3
4
5
6
7
8
9
I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
16
Message passing sequential schedule

• Select ordering of nodes
• Pass messages according
  to this ordering
• Forward pass
• Backward pass
• Needs half the memory Kolmogorov05
• Generalization of technique in FelzenszwalbHutte
  nlocher04

1
2
3
4
5
6
7
8
9
I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
17
Tree-reweighted message passing (TRW)Wainwright
et al. 02
Message update equation
weighting coefficient

• BP gpq 1
• TRW gpq?(0,1 depends on the choice of trees and
  probabilities
• This paper Can also depend on particular
  label gpq(xq)
• Important in stereo with occlusions

I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
18
Message passing for stereo with occlusions

• Stereo without occlusions O(N) edges
• One iteration O(NK) memory running time
• Using distance transforms FelzenszwalbHuttenloch
  er04
• Stereo with occlusions O(NK) edges
• Naive implemenation O(NK2) memory, O(NK3)
  running time
• Distance transforms O(NK2) memory, O(NK2)
  running time
• This paper O(NK) memory
  running time

N number of pixels, K number of disparities
I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
19
Experiments

• 6 Benchmark ground truth datasets from
  Scharstein, Szeliski 02 03
• Error Statistic

Cones
Sawtooth
Teddy
Venus
Map
Tsukuba
number of misclassified pixels (/-1) in
non-occluded areas
number of misclassified pixels (/-1) near
discontinuities
I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
20
Experiments - Tsukuba
right view
Ground truth
Left view
I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
21
Experiments - Teddy
Right view
Ground truth
Left view
I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
22
Experiments

• Graph Cut outperforms TRW and BP
• - both low error and low energy

I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
23
Experiments energy vs. accuracy
BP
GT
BP
BP
TRW
GT
GT
TRW
TRW
GC
GC
GC
Map
Sawtooth
Venus
24
Experiments - speed

• TRW did not converge (even after 50,000
  iterations)
• BP gets into a loop after 50-200 iterations
• TRW performs worse with larger number of labels
  K

25
How difficult is the problem?
(Emin - Ebound)/Ebound
Make energy of 4-connected MRF and our MRF
similar(add to our energy num. pixel C)
I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
26
Conclusions

• Graph Cuts outperforms TRW and BP for highly
  connected MRFs
• In contrast to 4-connected MRF
• Modelling occlusion for stereo is important
• New test bed for MRF optimization methods
• Highly connected graph, yet efficient message
  passing
• Exposes algorithms drawbacks

I Stereo with occlusions II Algorithms
III Experiments IV Conclusions
27
Discussion future work

• Problems with TRW slow convergence with higher
  connectivity
• Different choice of trees?
• Different weighting?
• Schedule of passing messages?
• TRW algorithm of Wainwright et al. instead of
  TRW-S?
• More freedom in choosing trees
• Analyze other applications with highly connected
  MRFs
• Digital Tapestry Rother et al. 05
• Non-submodular energies?

28
(No Transcript)
29
Motivation (cont.)

• Complex MRFs (highly connected)
• Important vision problem
• Whats the best energy minimization algorithm?
• How close we are to the global minimum?
• How to apply message passing techniques?
• Algorithms drawbacks become more apparent

30
Experiments TRW Settings
31
Motivation
4-connected MRF (stereo, no occlusions)
Highly connected MRF (stereo
with occlusions)
Kolmogorov, AISTATS 05

• TRW outperforms graph cuts and BP!
• Can compute global minimum
  Meltzer,Yanover,Weiss05

32
Importance of modelling occlusions
Ground Truth
With occlusionsKolmogorov,Zabih02
Without occlusionBoykov et al.01
Error statistics
I Stereo with occlusions II Algorithms
III Experiments IV Conclusions