Graph-based Segmentation

1
Graph-based Segmentation
02/25/10

• Computer Vision
• CS 543 / ECE 549
• University of Illinois
• Derek Hoiem

2
Last class

• Gestalt cues and principles of organization
• Mean-shift segmentation
• Good general-purpose segmentation method
• Generally useful clustering, tracking technique
• Watershed segmentation
• Good for hierarchical segmentation
• Use in combination with boundary prediction

3
Todays class

• Treating the image as a graph
• Normalized cuts segmentation
• MRFs Graph cuts segmentation
• Recap
• Go over HW2 instructions

4
Images as graphs
i
wij
c
j

• Fully-connected graph
• node for every pixel
• link between every pair of pixels, p,q
• similarity wij for each link

Source Seitz
5
Similarity matrix
Increasing sigma
6
Segmentation by Graph Cuts
A
B
C

• Break Graph into Segments
• Delete links that cross between segments
• Easiest to break links that have low cost (low
  similarity)
• similar pixels should be in the same segments
• dissimilar pixels should be in different segments

Source Seitz
7
Cuts in a graph
B
A

• Link Cut
• set of links whose removal makes a graph
  disconnected
• cost of a cut

• One idea Find minimum cut
• gives you a segmentation
• fast algorithms exist for doing this

Source Seitz
8
But min cut is not always the best cut...
9
Cuts in a graph
B
A

• Normalized Cut
• a cut penalizes large segments
• fix by normalizing for size of segments
• volume(A) sum of costs of all edges that touch A

Source Seitz
10
Recursive normalized cuts

• Given an image or image sequence, set up a
  weighted graph G(V, E)
• Vertex for each pixel
• Edge weight for nearby pairs of pixels
• Solve for eigenvectors with the smallest
  eigenvalues (D - W)y ?Dy
• Use the eigenvector with the second smallest
  eigenvalue to bipartition the graph
• Note this is an approximation
• 4. Recursively repartition the segmented parts
  if necessary

http//www.cs.berkeley.edu/malik/papers/SM-ncut.
pdf
Details
11
Normalized cuts results
12
Normalized cuts Pro and con

• Pros
• Generic framework, can be used with many
  different features and affinity formulations
• Provides regular segments
• Cons
• Need to chose number of segments
• High storage requirement and time complexity
• Bias towards partitioning into equal segments
• Usage
• Use for oversegmentation when you want regular
  segments

13
Graph cuts segmentation
14
Markov Random Fields
Node yi pixel label
Edge constrained pairs
Cost to assign a label to each pixel
Cost to assign a pair of labels to connected
pixels
15
Markov Random Fields
Unary potential

• Example label smoothing grid

0 -logP(yi 0 data) 1 -logP(yi 1 data)
Pairwise Potential
0 1 0 0 K 1 K 0
16
Solving MRFs with graph cuts
Source (Label 0)
Cost to assign to 0
Cost to split nodes
Cost to assign to 1
Sink (Label 1)
17
Solving MRFs with graph cuts
Source (Label 0)
Cost to assign to 0
Cost to split nodes
Cost to assign to 1
Sink (Label 1)
18
Grab cuts and graph cuts
Magic Wand (198?)
Intelligent ScissorsMortensen and Barrett (1995)
GrabCut
User Input
Result
Regions
Regions Boundary
Boundary
Source Rother
19
Colour Model
R
R
Iterated graph cut
Foreground Background
Foreground
G
Background
G
Background

• Gaussian Mixture Model (typically 5-8 components)

Source Rother
20
Graph cuts Boykov and Jolly (2001)
Image
Source Rother
21
Graph cuts segmentation

• Define graph
• usually 4-connected or 8-connected
• Define unary potentials
• Color histogram or mixture of Gaussians for
  background and foreground
• Define pairwise potentials
• Apply graph cuts
• Return to 2, using current labels to compute
  foreground, background models

22
Moderately straightforward examples
GrabCut completes automatically
23
Difficult Examples

• Camouflage
• Low Contrast

Fine structure
Harder Case
Initial Rectangle
InitialResult
24
Using graph cuts for recognition
TextonBoost (Shotton et al. 2009 IJCV)
25
Using graph cuts for recognition
Unary Potentials
Alpha Expansion Graph Cuts
TextonBoost (Shotton et al. 2009 IJCV)
26
Limits of graph cuts

• Associative edge potentials penalize different
  labels
• If not associative, can sometimes clip potentials
• Approximate for multilabel
• Alpha-expansion or alpha-beta swaps

Must satisfy
27
Graph cuts Pros and Cons

• Pros
• Very fast inference
• Can incorporate recognition or high-level priors
• Applies to a wide range of problems (stereo,
  image labeling, recognition)
• Cons
• Not always applicable (associative only)
• Need unary terms (not used for generic
  segmentation)
• Use whenever applicable

28
Further reading and resources

• Normalized cuts and image segmentation (Shi and
  Malik)
• http//www.cs.berkeley.edu/malik/papers/SM-ncut.p
  df
• N-cut implementation
• http//www.seas.upenn.edu/timothee/software/ncut/
  ncut.html
• Graph cuts
• http//www.cs.cornell.edu/rdz/graphcuts.html
• Classic paper What Energy Functions can be
  Minimized via Graph Cuts? (Kolmogorov and Zabih,
  ECCV '02/PAMI '04)

29
Recap of Grouping and Fitting
30
Line detection and Hough transform

• Canny edge detector smooth ?
  derivative ? thin ? threshold ? link
• Generalized Hough transform points vote for
  shape parameters
• Straight line detector canny
  gradient orientations ? orientation binning ?
  linking ? check for straightness

31
Robust fitting and registration

• Key algorithm
• RANSAC

32
Clustering

• Key algorithm
• Kmeans

33
EM and Mixture of Gaussians

• Tutorials http//www.cs.duke.edu/courses/spring04
  /cps196.1/.../EM/tomasiEM.pdfhttp//www-clmc.usc.e
  du/adsouza/notes/mix_gauss.pdf

34
Segmentation

• Mean-shift segmentation
• Flexible clustering method, good segmentation
• Watershed segmentation
• Hierarchical segmentation from soft boundaries
• Normalized cuts
• Produces regular regions
• Slow but good for oversegmentation
• MRFs with Graph Cut
• Incorporates foreground/background/object model
  and prefers to cut at image boundaries
• Good for interactive segmentation or recognition

35
Next section Recognition

• How to recognize
• Specific object instances
• Faces
• Scenes
• Object categories
• Materials