The Layout Consistent Random Field for detecting and segmenting occluded objects

1
The Layout Consistent Random Fieldfor detecting
and segmenting occluded objects
John Winn Jamie Shotton
CVPR, June 2006
2
LayoutCRF contributions

• Detection and segmentation
• Handles occlusion and deformation
• Multiple objects simultaneously
• Multiple classes

3
Roadmap

• Related work
• Layout consistency
• Layout Consistent Random Field
• Results

4
Related work constellation models
X
Fergus et al. CVPR 2003
Leibe et al. ECCV 2004
Crandall et al. ECCV 2006
Kumar et al. CVPR 2005
5
Related work constellation models
X
X
X
X
Fergus et al. CVPR 2003
Leibe et al. ECCV 2004
Crandall et al. ECCV 2006
Kumar et al. CVPR 2005
6
Related work windowed detectors
Classifier
Car
Localised features
Sliding window
Viola and Jones CVPR 2001 Shotton et al.
ICCV 2005
7
Related work windowed detectors
Classifier
Car? Wall?
Localised features
Sliding window
Viola and Jones CVPR 2001 Shotton et al.
ICCV 2005
8
Related work multiclass segmentation
TextonBoost Shotton et al. ECCV 2006
building
tree
car
road
Doesnt exploit layout of parts cant identify
object instances
Tu et al. CVPR 2003 He et al. CVPR 2004
9
Roadmap

• Related work
• Layout consistency
• Layout Consistent Random Field
• Results

10
Dense part labelling

• Automatic per-pixel labelling based on a grid of
  parts

11
Dense part labelling
Background label
12
Patch-based part detector
13
Decision trees
Extremely efficient at both training and test
time. e.g. takes 2ms to apply to 160x120 image
using difference of pixel intensities.
Improved performance with multiple decision trees
(random forest).
Performs as well as boosting with shared
features, but can process much more data in the
same time.
14
Patch-based part detector
15
Layout consistency
16
Layout consistency
17
Layout consistency
18
Layout consistency
Neighboring pixels
(p,q)
(p,q)
(p1,q)
(p1,q-1)
Allows for deformation/rotation
(p1,q1)
19
Layout consistency
Neighboring pixels
(p,q)
?
(p,q1)
(p,q)
(p1,q1)
(p-1,q1)
Layoutconsistent
20
Occlusions
Not layout consistent occlusion (or invalid)
21
Effect of layout consistency
Input image
Part detector output
With layout consistency
Layout consistent regions
22
Roadmap

• Related work
• Layout consistency
• Layout Consistent Random Field
• Results

23
Layout Consistent Random Field
Part labels h
24
Layout Consistent Random Field
Part labels h
25
Layout Consistent Random Field
Parameters ? ßbg , ßoe , ßco , ßiif , e0 , ?
(set by hand)
26
Inference of MAP labelling
Graph cuts with customised alpha-expansion move
Part labels h
Boykov and Jolly, ICCV 2001
27
Inference of MAP labelling
Graph cuts with customised alpha-expansion move
Proposed labelling
Part labels h
Boykov and Jolly, ICCV 2001
28
Inference of MAP labelling
Graph cuts with customised alpha-expansion move
Expansion move not accepted
Proposed labelling
Part labels h
Boykov and Jolly, ICCV 2001
29
Inference of MAP labelling
Graph cuts with customised alpha-expansion move
Proposed labelling
Part labels h
Boykov and Jolly, ICCV 2001
30
Example inference
31
Decision tree re-learning

• Part-labels are inferred (constrained by known
  mask) and decision forest re-trained

32
Limitation of layout consistency

• Allows arbitrary stretching/scaling

33
Global layout
Part labels h
Global layout constraint is (weak) star-shaped
constellation model Constrains part
locationsrelative to centroid
Instance T1
Allows competition between different object
instances
34
Example with global consistency
Input image
Layout consistent regions
Instance labelling
35
Roadmap

• Related work
• Layout consistency
• Layout Consistent Random Field
• Results

36
UIUC car database
Segmentation accuracy 96.5 pixels correct
(assessed on 20 randomly selected, hand-labelled
images)
37
UIUC car database
Segmentation accuracy 96.5 pixels correct
(assessed on 20 randomly selected, hand-labelled
images)
38
UIUC car database detection
Results refer to detection of unoccluded cars
only.
39
Detecting heavily occluded faces

• Caltech face database with artificial occlusions
• AR face database with real occlusions

40
Stability of part labelling
Part color key
41
Multi-class detection

• Can extend to multiple classes with different
  numbers of part labels for each class
• Example building has multiple parts, other
  classes have one

42
Summary future directions

• Summary
• LayoutCRF achieves multi-class detection and
  segmentation of occluded, deformable objects

• Future directions
• Extend to multiple viewpoints and multiple scales
• Share parts between classes
• Incorporate object context (car above road)
• Incorporate geometric cues

43
Thank you
jwinn_at_microsoft.com http//johnwinn.org/
44
(No Transcript)