Markov Random Fields for Computer Vision Applications

1
Markov Random Fields for Computer Vision
Applications

• Carsten Rother
• Collaborators
• Vladimir Kolmogorov, Andrew Blake, Antonio
  Criminisi, John Winn, Jamie Shotton, Sanjiv Kumar
  

2
Markov Random Field Models for Computer Vision

• MRF model optimization
• (inference)
• Graph Cut (GC)
• Belief Propagtion (BP)
• Tree-Reweighted Message Parsing (TRW)
• Iterated Conditional Modes (ICM)
• Dynamic Programming (DP)
• Random Walker

• Applications
• 2D/3D Image segmentation
• Video segmentation/tracking
• Object Recognition
• Alpha matting
• Stereo matching
• Texture modelling
• Texture Synthesis
• Photo Collages
• Panoramic Stitching

• MRF model - learning
• Pseudo-Likelihood approximation
• Training in Pieces
• Fixed Point method

3
Overview

• Image Segmentation What is a Markov Random
  Field (MRF)?
• Stereo Matching Multiple Labels and High
  Connectivity
• Further Applications and Demos

4
Overview

• Image Segmentation What is a Markov Random
  Field (MRF)?
• Stereo Matching Multiple Labels and High
  Connectivity
• Further Applications and Demos

5
Interactive Image Segmentation
Fast Accurate ?
6
Interactive Image Segmentation
Magic Wand (198?)
Intelligent ScissorsMortensen and Barrett (1995)
GrabCut
User Input
Result
Regions
Regions Boundary
Boundary
part of Microsoft Expression
7
Interactive 3D Segmentation
(courtesy Yuri Boykov)
8
Framework

• Input Image
• Output Segmentation
• Energy
• Optimization

9
Data Model
Foregroundcolour
Backgroundcolour
Global optimum of the energy

• D is the log probability of a Gaussian Mixture
  Model

10
Coherence Model
An object is a coherent set of pixels
25
Error () over training set
How do we choose ?
25
11
Probabilistic View

• Gibbs Distribution

• Maximum a posteriori estimator (MAP)

same as

• Bayes Rule

constant
Definition of an MRF
12
Graph Cuts
Boykov and Jolly ICCV 01
Image
Min Cut Problem is equal to MaxFlow Problem
Fulkerson 56
13
Graph Cuts
(courtesy Yuri Boykov)
Example of Pushing Max Flow through the Graph
14
GrabCut Problem
?
?
User Initialisation
Unknown Colour model
Graph cuts to infer the segmentation
K-means for learning colour distributions
15
Iterated Graph Cuts
Guaranteed toconverge
1
3
4
2
Energy after each Iteration
Result
16
Colour Model
R
R
Iterated graph cut
Foreground Background
Foreground
G
Background
G
Background

• Gaussian Mixture Model (typically 5-8 components)

17
Examples
GrabCut completes automatically
18
Overview

• Image Segmentation What is a Markov Random
  Field (MRF)?
• Stereo Matching Multiple Labels and High
  Connectivity
• Further Applications and Demos

19
Stereo ApplicationCriminisi et. al. ICCV 03
Depth Map
Novel View
Right View
Left View
20
Stereo Matching
Rectified Epipolar geometry
(courtesy Andrea Fusiello )
21
Matching with Coherency
Right input image
Left input image

• A label gives a match

output
22
Alpha Expansion Move AlgorithmBoykov et. al.
PAMI 03
ExpansionMove
Loop over all Labels
Binary Graph Cut Keep old label (0) or take new
label alpha (1)
23
Stereo with OcclusionKolmogorov et. al. ECCV 04
1

• Highly connected MRF
• Label both images simultaneously
• Each pixel is connected to k pixels in the other
  image (k number of disparity levels)

3
24
Comparison of Optimization MethodsSzeliski et.
al. ECCV 06, Kolmogorov et. al ECCV 06
Goal Collect benchmark datasets from different
MRF applications to compare optimization
methods
Ground Truth
Right image
Left image
Belief Propagation
Tree-ReweightedMessage Parsing
Graph Cut
ICM
25
Comparison of Optimization Methods
Highly connected MRF
4-connected MRF
Put a graphic from your AIStat paper
GC
seconds
seconds
26
Overview

• Image Segmentation What is a Markov Random
  Field (MRF)?
• Stereo Matching Multiple Labels and High
  Connectivity
• Further Applications and Demos

27
Stereo SegmentationKolmogorov et. al. CVPR 05
MRF on ternary variables Foreground,
Background, Occlusion
Right View
Left View
28
Recognition SegmentationShotton et. al. ECCV
06
 
21 classes
29
Recognition SegmentationShotton et. al. ECCV
06
 
MRF Coherency Model as in GrabCut
Data Likelihood learned shape texture
Input Image
30
Alpha MattingRother et. al. Siggraph 04
Input Image
Binary MaskGrabCut
S continuous between 0 and 1
31
Panoramic Stitching
Agarwala et. al. Siggraph 04
Input Image Set
32
Panoramic Stitching
Agarwala et. al. Siggraph 04
Labelling
Final Result (including Poisson Blending)
33
Auto Collage
Rother et. al. Siggraph 06, CVPR 05
34
Auto Collage Rother et. al. Siggraph 06, CVPR
05
35
Texture Video SynthesisKwatra et. al.
Siggraph 03
Input
Input
Output
Output
36
Conclusions

• MRF models are very important for Computer
  Vision Applications
• What is the best MRF model, depending on
  application?
• Which optimization method is the best?
• How to learning the model?