Advances in Minimizing Submodular Energy Functions with Graph Cuts

1
Advances in Minimizing Submodular Energy
Functions with Graph Cuts

• Daniel Munoz
• misc reading group
• May 28, 2008

2
Covered Topics

• Review of energy minimization with graph cuts for
  MRF inference
• Approximating Potts model over high-order
  cliques, multi-label variables
• kohli-cvpr-07 P. Kohli, M.P. Kumar, P. Torr. P3
  Beyond Solving Energies with Higher Order
  Cliques. CVPR 2007.
• kohli-tr-08 P. Kohli, L. Ladicky, P. Torr.
  Graph Cuts for Minimizing Robust Higher Order
  Potentials. Technical report, Oxford Brookes
  University, 2008.
• kohli-cvpr-08 P. Kohli, L. Ladicky, P. Torr.
  Robust Higher Order Potentials for Enforcing
  Label Consistency. CVPR 2008.
• Exactly minimizing some functions of high-order
  cliques, multi-label variables
• ramalingam-cvpr-08 S. Ramalingam, P. Kohli, K.
  Alahari, P. Torr. Exact Inference in Multi-label
  CRFs with Higher Order Cliques. CVPR 2008.

3
References

• kolmogorov-pami-04 V. Kolmogorov, R. Zabih.
  What Energy Functions can be Minimized via Graph
  Cuts? PAMI 2004.
• veksler-phdthesis-99 O. Veksler. Efficient
  Graph-based Energy Minimization Methods in
  Computer Vision. PhD Thesis, Cornell University,
  1999.
• gupta-icml-07 R. Gupta, A. A. Diwan, S.
  Sarawagi. Efficient Inference with
  Cardinality-based Clique Potentials. ICML 2007.
• lan-eccv-06 X. Lan, S. Roth, D. Huttenlocher,
  M. J. Black. Efficient Belief Propagation with
  Learned Higher-Order Markov Random Fields. ECCV
  2006.
• boykov-pami-01 Y. Boykov, O. Veksler, and R.
  Zabih. Fast approximate energy minimization via
  graph cuts. PAMI 2001.
• hoeim-ijcv-07 D. Hoiem, A. Efros, M. Hebert.
  Recovering surface layout from an image. IJCV
  2007.
• rother-cvpr-05 C. Rother, S. Kumar, V.
  Kolmogorov, A. Blake. Digital Tapestry. CVPR
  2005.

4
References (ctd)

• veksler-cvpr-07 O. Veklser. Graph Cut Based
  Optimization for MRFs with Truncated Convex
  Priors. CVPR 2007.
• hammer-mp-84 P.L. Hammer, P. Hansen, B.
  Simeone. Roof Duality, Complementation, and
  Persistency in Quadratic 0-1 Optimization.
  Mathematical Programming, 1984
• kolmogorov-pami-07 V. Kolmogorov, C. Rother.
  Minimizing Non-submodular Functions with Graph
  Cuts A Review. PAMI 2007
• rosenberg-ccero-75 I.G. Rosenberg. Reducation
  of Bivalent Maximization to the Quadratic Case.
  Cahiers du Centre dEtudes de Recherche
  Operationnelle 1975.
• gallo-mp-89 G. Gallo, B. Simeone. On the
  supermodular knapsack problem. Mathematical
  Programming 1989
• orlin-ipco-07 J. Orlin. A faster strongly
  polynomial time algorithm for submodular function
  minimization. Integer Programming and
  Combinatorial Optimization 2007
• schlesinger-emmcvpr-07 D. Schlesinger. Exact
  solution of permuted submodular min-sum problems.
  EMMCVPR 2007.

5
Overview

• Problem formalization
• Graph-cut Minimization Review
• kohli-cvpr-07
• kohli-tr-08, kohli-cvpr-08
• ramalingam-cvpr-08

6
Problem formulization

• Random variables X X1 , , XN
• Xi L l1 , , lK
• A specific labeling configuration is denoted as x
  x1 , , xN
• Define a random field with vertices X and
  neighborhood system such that a clique c
  contains variables Xc such that
• The posterior is defined as a Gibbs distribution
  
• D data, Z normalizing constant (partition
  function), C set of all cliques
• Potential function of clique c,
  xc xi i c
• Standard Markov Random Field model
• Gibbs energy is the negative log
• Maximum a posteriori (MAP) assignment

7
Problem formulization

• Total energy
• Pairwise model
• Higher-order model

1st order function
High-order function
2nd order function
8
Binary Confusion

• Binary function
• 2nd order function?
• 2-label variables (L 2)?
• Boolean 2-label variable (not necessarily 0/1)
• Multi-label L gt 2
• Pairwise 2nd order function

9
Overview

• Problem formalizations
• Graph-cut Minimization Review
• kohli-cvpr-07
• kohli-tr-08, kohli-cvpr-08
• ramalingam-cvpr-08

10
Graph-cut Minimization Review

• Energy minimizations of boolean variables with
  graph cuts
• Graph is constructed with capacities such that
  the min-cut value is the minimum energy of the
  function (plus a constant)
• Sink and source terminals in a directed graph
  represent the boolean values that the variable
  can be assigned
• Nodes assigned respective values of the terminals
  they are connected to after the cut
• Example
• Philip H.S. Torr, Graph Cuts and their Use in
  Computer Vision, Invited tutorial at
  International Computer Vision Summer School 2007,
  Detection, Recognition and Segmentation in
  Context, July 2007

11
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2)
Source (0)
a1
a2
Sink (1)
12
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2) 2a1
Source (0)
2
a1
a2
Sink (1)
13
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2) 2a1 5a1
Source (0)
2
a1
a2
5
Sink (1)
14
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2) 2a1 5a1 9a2 4a2
Source (0)
2
9
a1
a2
5
4
Sink (1)
15
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2
Source (0)
2
9
a1
a2
2
5
4
Sink (1)
16
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (0)
2
9
1
a1
a2
2
5
4
Sink (1)
17
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (0)
2
9
a1 1 a2 1
1
a1
a2
2
Cost of st-cut 11
5
4
EMRF(1,1) 11
Sink (1)
18
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (0)
2
9
a1 1 a2 0
1
a1
a2
2
Cost of st-cut 8
5
4
EMRF(1,0) 8
Sink (1)
19
Energy Minimization using Graph cuts
What really happens? Building the graph
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
CONSTANT TERM K
Source (0)
9 a
2
a1 1 a2 0
1
a1
a2
2
Cost of st-cut 8 a
5
4 a
EMRF(1,0) 8 a
Sink (1)
20
Energy Minimization

• When can we minimize a function with graph cuts?
• When can we minimize a function, in general?
• Submodular vs. Non-submodular functions

21
Submodular functions

• In general, a submodular function can be
  minimized in polynomial time
• O(n5T n6) where T time to evaluate the
  function orlin-ipco-07
• Definitions of submodular functions of boolean
  variable set
• A function of 1 variable is always submodular
• A 2nd order function Eij(x1,x2) is submodular if
  and only if Eij(0,0) Eij(1,1) Eij(0,1)
  Eij(1,0)
• A mth order function E(x) is submodular if and
  only if all its projections on 2 variables are
  submodular
• Example Eijk(0, x2, x3) is a projection of
  Eijk(x1, x2, x3)
• The sum of two submodular functions is submodular
• Submodularity functions of multi-label variables
  will be defined later
• Example 2nd order, boolean Potts model
• Non-submodular for multi-label case and
  NP-complete to minimize

22
Non-submodular functions

• Non-submodular, 2nd order, boolean functions
• Minimizing is NP-hard kolmogorov-pami-04
• Approximate with Quadratic Pseudo-Boolean
  Optimization (QPBO) hammer-mp-84,
  kolmogorov-pami-07
• Approximating non-submodular, high-order,
  multi-label functions
• Encode high-order, multi-label function to
  high-order, boolean function ramalingam-cvpr-08
• Reduce high-order, boolean function to 2nd-order,
  boolean function
• Can always be done in poly-time
  rosenberg-ccero-75
• Apply QPBO

23
Move-making algorithms boykov-pami-01

• Able to efficiently approximate some functions
  that are NP-hard to minimize
• a-expansion algorithm transform into boolean
  variable problem
• Start with arbitrary labeling.
• For a L
• Consider energy if current label remains the same
  (0) or label takes value a (1).
• Minimize (min-cut), change labels appropriately
• Repeat unless energy did not decrease
• Converges to local minimum
• At best, factor of 2 approximation for any size
  clique gupta-icml-07
• What functions can be approximated with
  a-expansion?
• Whenever Eij(xi, xj) Eij(a,a) Eij(xi, a)
  Eij(a, xj) for all a in L
• Example Potts model kohli-cvpr-08
• Historical note
• Originally function had to be metric
• Metric functions are submodular
• aß-swap
• See boykov-pami-01
• If can apply a-expansion, can apply aß-swap

24
Summary of Efficiently Minimizing Energy with
Graph Cuts
25
Functions of label differences

• V(p,q) is 2nd order potential of the difference
  in the labels of pixels p and q
• These functions penalize big difference in label
  values between neighboring data
• Image restoration want to maintain similar
  intensities with neighbors

Robust or discontinuity-preserving
interactions(minimize is NP-complete)
Convex interactions(minimize is P)
boykov-pami-01(approximate)
veksler-phdthesis-99(exact)
(everywhere smooth)
(piecewise constant)
veksler-cvpr-07(approximate)
ishikawa-pami-03(exact)
(convex)
(piecewise smooth,truncated convex)
26
Overview

• Problem formalizations
• Graph-cut Minimization Review
• kohli-cvpr-07
• kohli-tr-08, kohli-cvpr-08
• ramalingam-cvpr-08

27
kohli-cvpr-07
Some text taken from Kohlis CVPR 07 talk

• P. Kohli, M.P. Kumar, P. Torr. P3 Beyond
  Solving Energies with Higher Order Cliques. CVPR
  2007.
• Motivation
• Pairwise model only accounts for local
  interaction among variables
• Field of Experts shows superiority of high-order
  interaction
• lan-eccv-06
• 2x2 clique potentials for Image Denoising
• 16 minutes per iteration
• Is there a general, high-order potential function
  that can be minimized efficiently?

Pairwise MRF
Noisy Image
Higher order MRF
28
kohli-cvpr-07

• Reminder Minimizing 2nd order, multi-label Potts
  model is NP-complete
• We cannot minimize Potts, but there exist
  efficient approximation algorithms with graph
  cuts
• a-expansion
• Approach extend notion of 2nd order Potts model
  to the high-order case
• P2 Potts model
• Pn Potts model

29
kohli-cvpr-07

• Pn Potts model
• When can we use a-expansion?
• If all projections of any a-expansion move
  energy on two variables are submodular
  Eij(0,0) Eij(1,1) Eij(0,1) Eij(1,0)
• How to show all projections of Pn Potts are
  submodular
• Redefine an equivalent version of Pn
  Pottswhere
• Encode 0 current label (xi), 1 a
• Project to general 2nd order function, then check
  to satisfy the submodularity
• Not difficult to show constraint satisfied
• Consider RHS when xi , xj ? a

30
kohli-cvpr-07

• Graph construction

Source
Ms
v1
v2
vn
Mt
a
Sink
31
kohli-cvpr-07

• Graph construction
• 2 auxiliary variables
• Merge graph construction with ones given from
  kolmogorov-pami-04 by Additivity Theorem

Source
Ms
v1
v2
vn
Mt
a
Sink
32
kohli-cvpr-07

• Graph construction
• 2 auxiliary variables
• Merge graph construction with ones given from
  kolmogorov-pami-04 by Additivity Theorem

Source
Ms
v1
v2
vn
Mt
a
Sink
33
kohli-cvpr-07

• Application texture-based video segmentation
• Does not use motion information
• Each pixel is a variable
• 2 different video sequences
• Learn on input keyframesand corresponding
  labeledsegmentations
• How many trainingkeyframes?

1 keyframe
Labeled segmentation
34
kohli-cvpr-07

• 1st order potential
• RGB distributions
• 2nd order potential
• g(i,j) difference between RGB values at pixel i
  and j
• 8-Neighborhood
• High-order potential
• 4x4 textons
• Overlapping patches (C N)
• G(c,s) minimum L1 difference between patch
  (clique) c and texton dictionary for label s
• Not given how parameters determined

35
kohli-cvpr-07

• Results
• Input Keyframes
• 2nd order interaction, alpha-expansion 2.5 sec
• 16th order interaction, alpha-expansion 3.0 sec

36
kohli-cvpr-07

• Results
• Input Keyframes
• 2nd order interaction, alpha-expansion 3.7sec
• 16th order interaction, alpha-expansion 4.4 sec

37
Overview

• Problem formalizations
• Graph-cut Minimization Review
• kohli-cvpr-07
• kohli-tr-08, kohli-cvpr-08
• ramalingam-cvpr-08

38
kohli-tr-08, kohli-cvpr-08

• kohli-tr-08 P. Kohli, L. Ladicky, P. Torr.
  Graph Cuts for Minimizing Robust Higher Order
  Potentials. Technical report, Oxford Brookes
  University, 2008.
• Theory
• kohli-cvpr-08 P. Kohli, L. Ladicky, P. Torr.
  Robust Higher Order Potentials for Enforcing
  Label Consistency. CVPR 2008.
• Application
• Motivation
• Assign cost proportional to amount of variables
  disagreeing
• 
  number of variables not taking
  dominant label

39
kohli-tr-08, kohli-cvpr-08

• Robust Pn Potts model
• where
• Q truncation parameter. 2Q lt c
• Q 1 equivalent to non-robust Pn Potts model
• Able to combine multiple versions to achieve
  arbitrary truncation

40
kohli-tr-08, kohli-cvpr-08

• Can be approximated with a-expansion
• Graph construction
• Add 2 auxiliary variables
• d dominant label
• (stuff)
• wi importance of labeling xi with dominant
  label
• Ex Assign low weight to pixels on segmentation
  boundary

(weighted)
41
kohli-tr-08, kohli-cvpr-08

• Application object segmentation and recognition
  with unsupervised segmentations
• Robust Pn potts model using multiple segmentation
  results as cliques
• Unary (TextonBoost, Color, Location)
• Pairwise (Color)
• High order (quality-sensitive consistency
  potential)
• Pn Potts
• Robust Pn Potts

42
kohli-tr-08, kohli-cvpr-08

• Clustering
• Mean-shift over coordinates and LUV
• 3 different segmentations passes
• Experiments
• MSRC-23 dataset 591 images
• Sowerby-7 dataset 104 images
• No reported quantitative results
• 50 of respective dataset for training
• CRF parameters 10 parameters to find
• Cross-validation on subset of training images
• 1) Unary only
• 2) Unary fixed, pairwise only
• 3) Unary fixed, high-order only
• 4) In the last step the ratio between pairwise
  and higher order potentials

Pair
Pn
R. Pn
Hand
43
kohli-tr-08, kohli-cvpr-08

• Evaluation
• Only for MSRC-23
• Manually fine labeled 27 images
• Examine accuracy on pixels lying on boundary of
  image, over different widths

44
Overview

• Problem formalizations
• Graph-cut Minimization Review
• kohli-cvpr-07
• kohli-tr-08, kohli-cvpr-08
• ramalingam-cvpr-08

45
ramalingam-cvpr-08

• S. Ramalingam, P. Kohli, K. Alahari, P. Torr.
  Exact Inference in Multi-label CRFs with Higher
  Order Cliques. CVPR 2008.
• Idea
• Transform high order, multi-label submodular
  functions into 2nd order, boolean submodular
  functions
• We are able to exactly solve 2nd order, boolean,
  submodular functions with graph cuts
  kolmogorov-pami-04
• Approach
• Represent a multi-variable variable with multiple
  boolean variables
• Construct a graph such that the only solutions
  encode the possible states of the variable
• Contributions
• Principle framework for transforming submodular,
  high order, multi-label functions to submodular,
  2nd order, boolean functions
• There exists no transformation for order 4 or
  more, unless P NP
• Application significant improvements on Hoiems
  work

46
ramalingam-cvpr-08

• Reminder graph construction for submodular,
  boolean, 2nd order functions given in
  kolmogorov-pami-04
• Posiforms
• Posiform of f(x1,x2,x3,x4) 2 4x2x4 7x1x2x3
  is equal to g(x1,x2,x3,x4)
  -2 4(1-x4) 4(1-x2)x4 7x1x2x3
• Boolean variables
• All variable coefficients are non-negative
• Theorem The recognition of submodularity in
  degree 4 posiforms is co-NP-complete gallo-mp-89

47
ramalingam-cvpr-08

• Multi-label submodularity
• Assume L is an ordered label set
• Notion of above/below is present between any 2
  labels
• Eij(a,b) Eij(a1, b1) Eij(a1,b) Eij(a,
  b1) for all a,b in L
• Potts model is not submodular in multi-label case
• If it exists, can always find ordering of L such
  that the function is submodular
• schlesinger-emmcvpr-07

48
ramalingam-cvpr-08

• 1) Encode multi-label variable with boolean
  variables
• Consider 4-label variable y1, encode with 3
  boolean variables x1, x2, x3

Unused states (infeasible)
49
ramalingam-cvpr-08

• 2) Encoding functions
• Define a functions, for all a in L
• If y1 a, define fy1a(x1, x2, x3) 1,
  otherwise fy1a(x1, x2, x3) 0
• Assume linear form
• Case y1 1
• System of linear equations to solve
• fy11 x1
• Repeat for all possible values of y1
• Final encodings

50
ramalingam-cvpr-08

• Replace all occurrences of multi-label variable
  with boolean variables
• Example (1 variable function always submodular)
• Original energy
• E(y1) 10 y1 y1 1,2,3,4
• Transformed Energy
• E(x1, x2, x3) 10x1 20(x2 - x1) 30(x3-x2)
  40(1-x3)
• Optimal answer x3 1, x2 1, x1 1 ? y1 1
• Does this always work?
• Yes for 1st, 2nd, 3rd order
• Suppose given 4th order, boolean function and
  there exists a way to transform it into a 2nd
  order, boolean function. We can then easily
  check for submodularity ?
• Contradiction Recognizing submodularity of
  degree 4 posiforms is co-NP-complete
  gallo-mp-89 ?

51
ramalingam-cvpr-08

• Application improving single-view reconstruction
  hoiem-ijcv-07
• Labels support (ground plane), vertical
  (surfaces rising from the ground), sky
• Approach Conditional Random Fields
• Nodes superpixel segments
• Unary potential boosted decision tree
  classifiers hoiem-ijcv-07
• Pairwise potential statistics of neighboring
  superpixl pairs in training set
• 3rd order potential
• Natural ordering of superpixels in vertical
  direction
• Directly use distribution (negative
  log-likelihood) of labelings of vertically
  aligned superpixel triplets
• Distribution not guaranteed to besubmodular
• Truncate non-submodular terms(add constants such
  that the function is submodular)
• rother-cvpr-05

52
ramalingam-cvpr-08

• Results

Ground truth
Hoiem
Ramalingam
53
ramalingam-cvpr-08

• Results
• Misclassifications of individual pixels in the
  image

54
The end

• Questions?