Interactive Graph Cuts for Segmentation in N-D Images

1
Interactive Graph Cuts for Segmentation in N-D
Images
Yuri Boykov, Marie-Piere Jolly
Mohit Gupta 02/15/2006 Advanced Perception
2
Image Segmentation
Images Label Me Database
3
Focus of this work Foreground vs. Background

• The framework presented can be easily extended
  to multi-label segmentation though
• Multi-label segmentation solved by iteratively
  solving many 2-label
• sub-problems (Boykov, Veksler, Zabih)

Images Yin Li et al (Lazy Snapping)
4
Focus of this work Foreground vs. Background

• The framework presented can be easily extended
  to multi-label segmentation though
• Multi-label segmentation solved by iteratively
  solving many 2-label
• sub-problems (Boykov, Veksler, Zabih)

Supervised Method
Images Yin Li et al (Lazy Snapping)
5
Taxonomy Previous Work

• Fully Automatic Methods
• EigenVectors Based approaches (Normalized Cuts,
  Perona and Freeman Algorithm etc.)
• Supervised Methods
• Boundary Based approaches use of an evolving
  curve (snakes, intelligent scissor, image
  snapping)
• Region Based approaches hints about foreground
  vs. background (intelligent paint, GrabCut)

6
Taxonomy Previous Work
A Gradient based vector-field

• Fully Automatic Methods
• EigenVectors Based approaches (Normalized Cuts,
  Perona and Freeman Algorithm etc.)
• Supervised Methods
• Boundary Based approaches use of an evolving
  curve (snakes, intelligent scissor, image
  snapping)
• Region Based approaches hints about foreground
  vs. background (intelligent paint, GrabCut)

7
Taxonomy Previous Work
Edge Weights depend on the boundary-ness
High probability of boundary ? Low weight

• Fully Automatic Methods
• EigenVectors Based approaches (Normalized Cuts,
  Perona and Freeman Algorithm etc.)
• Supervised Methods
• Boundary Based approaches use of an evolving
  curve (snakes, intelligent scissor, image
  snapping)
• Region Based approaches hints about foreground
  vs. background (intelligent paint, GrabCut)

8
Taxonomy Previous Work
Boundary Shortest Path between graph vertices

• Fully Automatic Methods
• EigenVectors Based approaches (Normalized Cuts,
  Perona and Freeman Algorithm etc.)
• Supervised Methods
• Boundary Based approaches use of an evolving
  curve (snakes, intelligent scissor, image
  snapping)
• Region Based approaches hints about foreground
  vs. background (intelligent paint, GrabCut)

9
Taxonomy Previous Work

• Fully Automatic Methods
• EigenVectors Based approaches (Normalized Cuts,
  Perona and Freeman Algorithm etc.)
• Supervised Methods
• Boundary Based approaches use of an evolving
  curve (snakes, intelligent scissor, image
  snapping)
• Region Based approaches hints about foreground
  vs. background (intelligent paint, GrabCut)

10
Case for a Supervised, Region Based Method

• Fully Automatic
• Never perfect Hard to get
• crisp boundaries for inherently
• ambiguous, low contrast images
• Boundary Based Supervised
• Complex Object Users nightmare

"Whenever something can be done in two ways,
someone will be confused. Whenever something is a
matter of taste, discussions can drag on
forever." -- Bjarne Stroustrup
11
Interactive Graph Cuts Overview

• Supervised Region based segmentation technique
• What user does?
• Provide clues as to the
  desired segmentation
• Foreground and
  Background seed
  pixels
• Energy of the segmentation
• Lower the energy, better the segmentation

12
Problem Formulation

• Input
• Set of pixels P,
• Neighborhood system
  N p,q
• Hard Constraints clues
  for segmentation
• Objective function Soft Constraint (Energy)
• Output
• Assignment vector A (A1, , Ap , , AP),
  where Ai e 0,1
• A defines a segmentation of the image

13
Objective FunctionEnergy of the Segmentation
TO DO Minimize Energy E(A) while satisfying the
hard constraints

• Regional Term R(A) ? Cost for assigning labels
  to individual pixels
• Boundary Term B(A) ? Cost for making the
  boundary pass between
• two given
  pixels

14
Graph Cuts and Image Segmentation

• A node for every pixel
• Two terminal nodes
• n-links and t-links
• Edge weights

15
Min-Cut Energy Minimization

• Graph Cut
• Segmentation
• A (A1, , Ap , , AP)

16
Min-Cut Energy Minimization

• Graph Cut
• Segmentation
• A (A1, , Ap , , AP)
• C E(A)
• Min C min E(A)

17
Min-Cut Energy Minimization

• Graph Cut
• Segmentation
• A (A1, , Ap , , AP)
• C E(A)
• Min C min E(A)

Min-cut Optimal Segmentation!
18
Implementation

• User Marked Seeds provide
• foreground and background intensity
  distributions
• in addition to hard constraints
• Color, Texture, intensity, location as features

19
Results

• Low values of l
• Boundary term dictates
• Region Shrinking

• High values of l
• Region term dictates
• Neighborhood info ignored

20
We dont want toy results

• Start with a few obj and bckg seeds
• Refinement in trouble places

Images Yin Li et al (Lazy Snapping)
21
Interactivity
New obj/bckg pixel added/deleted
Only two links change
Real time re-computation Possible ? Interactive!
22
More Results
Images Yin Li et al (Lazy
Snapping), Boykov and Jolly
23
Performance

• Efficient polynomial time algorithms available
  for min-cut
• Segmentation Problem ? Sparse graphs
• Computations in the blink of an eye
• Efficiency measured in terms of amount of human
  effort
• Bell Example takes about a minute

24
Markov Random Fields (MRFs)

• S discrete set of sites S 1, , m
• Ld discrete set of labels, eg. 1, L.
• Neighborhood system N
• A labeling assigns a label to every site,
• f f1, fm. fi is the label of site i.
• Neighborhood ? conditional independence
• F is an MRF on S w.r.t. N iff
• P(f) gt 0
• P(fi fS-i) P(fi fNi)

25
Lazy Snapping (SIGGRAPH2004)
Yin Li, Jian Sun, Chi-Keung Tang, Heung-Yeung
Shum(MSRA)

• Refine the coarse results returned by Graph-Cut
  Based methods (Boykov and Jolly)
• Extremely Fast
• Commercial System ? User Friendly

26
Modus Operandi How does it work?

• Supervised Region Based Method
• Three Step Process
• Pre-segmentation ( super pixels )
• Object Marking Step ( a la Boykov and Jolly)
• Local Refinement Step

27
Pre-segmentation

• Exploit Spatial Coherency ? Replace Pixels by
  Super-Pixels
• Aggressive unsupervised segmentation ?
  preserves color coherency within regions

28
Pre-segmentation
29
Object Marking Step
Similar to Boykov and Jolly
Different Penalty Functions
More similar a node to obj / bckg, lesser the
corresponding energy term
More similar the nodes, larger the boundary
energy term
30
Boundary Editing

• Object Boundary from previous step represented as
  polygon
• Polygon can be edited for local refinement
• Local refinement ? New constraints
• Another optimization for better fit

31
Boundary Editing

• Dij term penalizes a node pair far away from the
  polygon ? brings boundary closer to the polygon

b 1 ? boundary snaps to polygon
Important The optimized boundary snaps to the
object boundary even though the polygon vertices
may not be on it.
32
citius altius fortius

• Faster, easier, accurater

33
The pretty SIGGRAPH video
34
Multi-way graph cuts(Boykov, Veksler, Zabih)
Images Label Me Database
35
Multi-way graph cuts
Slides Yuri Boykov
36
Multi-way graph cuts

• BAD NEWS
• NP-hard problem (3 or more labels)
• two labels can be solved in polynomial time via
  s-t cuts
• GOOD NEWS
• a-expansion approximation algorithm
• guaranteed approximation quality (2-approx)

Slides Yuri Boykov
37
Multi-way graph cuts

• NP-hard problem (3 or more labels)
• two labels can be solved via s-t cuts
• a-expansion approximation algorithm
• guaranteed approximation quality (2-approx)
• a-expansion move

Slides Yuri Boykov
38
a-expansion move
Basic idea
break multi-way cut computation into a sequence
of binary s-t cuts
Slides Yuri Boykov
39
a-expansion move
Basic idea
break multi-way cut computation into a sequence
of binary s-t cuts
Iteratively, each label competes with other
labels for space in the image
40
a-expansion algorithm

• Start with any initial solution
• For each label a in any (e.g. random) order
• Compute optimal a-expansion move (s-t graph cuts)
• Decline the move if there is no energy decrease
• Stop when no expansion move would decrease energy

Slides Yuri Boykov
41
a-expansion algorithm

• Start with any initial solution
• For each label a in any (e.g. random) order
• Compute optimal a-expansion move (s-t graph cuts)
• Decline the move if there is no energy decrease
• Stop when no expansion move would decrease energy

A 2-approx algorithm for an NP-complete
problem! Converges in a few iterations!
42
Video Segmentation
Video Object Cut and Paste (SIGGRAPH2004)
Yin Li, Jian Sun, Heung-Yeung Shum(MSRA)
43
Climbing the dimension ladder2D ? 3D

• Video Segmentation 3D graph on volume
  of frames
• Edges for temporal coherence in addition to
  spatial coherence
• obj bckg seeds on a few key-frames

Image Yin Li et al
44
Whats new? Temporal Coherence
Our old friend
45
Local Refinement Problem

• obj / bckg color models built globally from
  key frames
• Might get confused between the two !

46
Local Refinement Solution

• Mark windows around problem areas in key frames
• Windows propagate through middle frames using a
  feature tracking algorithm
• Apply 2D pixel level graph cut segmentation
• Important Seeds generated automatically

47
Video Segmentation Results
Images Boykov and Jolly
48
Another (obviously pretty) SIGGRAPH video
49
Conclusion

• Merits
• Globally Optimum Segmentation, when cost function
  is clearly defined (normalized cuts only give an
  approximate solution)
• Very Fast Interactive Rates
• Natural Extension to N-Dimensional images
• Easily Editable More user friendly
• Possible Improvement
• Automatic Seed Selection From Coarse (LabelMe)
  kind of segmentation