LOCUS Learning Object Classes with Unsupervised Segmentation

1
LOCUS(Learning Object Classes with Unsupervised
Segmentation) A variational approach to learning
model-based segmentation.
John Winn Microsoft Research Cambridge with
Nebojsa Jojic, MSR Redmond
7th July 2006
2
Overview

• Learning object models
• The LOCUS model
• Experiments results
• Extensions to LOCUS

3
Goal

• Long Term Goal
• Recognise 10,000 object classes.

4
Learning from buckets of images
Learningalgorithm
Horsemodel

• Object Segmentation
• Object Recognition
• Object Detection

5
Object segmentation
LOCUS
Horsemodel
6
Related work
7
Constellation models

• Weakly supervised
• Probabilistic framework

• Sparse
• No segmentation

Object class recognition by unsupervised
scale-invariant learning. R. Fergus, P. Perona,
and A. Zisserman. CVPR 2003 A Bayesian approach
to unsupervised One-Shot learning of Object
categories. L. Fei-Fei, R. Fergus, and P.
Perona. ICCV 2003
8
Fragment-based

• Dense model
• Supervised

• Non-probabilistic
• No global shape model

Learning to segment. E. Borenstein and S. Ullman.
ECCV 2004 Combining top-down and bottom-up
segmentation. E. Borenstein, E. Sharon, and S.
Ullman. CVPR 2004
9
Codebook-based

• Probabilistic
• Dense model

• Supervised
• Ad-hoc inference

Combined object categorization and segmentation
with an implicit shape model. B. Leibe, A.
Leonardis, and B. Schiele. ECCV 04
10
OBJ CUT

• Probabilistic
• Dense model

• Supervised
• Requires video

11
LOCUS overview

• Weakly supervised learning Buckets of images -
  no annotation required.
• Probabilistic generative modelof both object and
  background.
• Dense modelAll pixels modelled, not just at
  interest points.
• Combines global and local cuesModels global
  shape and local appearance edges.
• Iterative inference processSimultaneous
  localisation, segmentation, pose estimation.

12
The LOCUS model
13
LOCUS model
Shared between images
Class shape p
Class edge sprite µo,so
Deformation field D
Position size T
Different for each image
Mask m
Edge image e
Object appearance ?1
Background appearance ?0
Image
14
LOCUS model appearance
Mask m
Image z
15
LOCUS model mask
background
object
8-neighbour Markov Random Field (as used in
GrabCut)
16
LOCUS model shape/position

17
Iterative inference
Class shape p
Iteration 1
T4
T2
T3


18
Iterative inference
Class shape p
Iteration 2
T4
T2
T3


19
Iterative inference
Class shape p
Iteration 3
T4
T2
T3


20
Iterative inference
Class shape p
Iteration 5
T4
T2
T3


21
Iterative inference
Class shape p
Iteration 8
T4
T2
T3


22
Iterative inference
Class shape p
Iteration 12
T4
T2
T3


23
Non-rigid objects
Class shape p
Translation and scale is not enough.
24
LOCUS model pose
Class shape p
25
LOCUS model pose
Class shape p

26
LOCUS model edge
Original images

27
LOCUS model overview
Shared between images
Class shape p
Class edge sprite µo,so
Deformation field D
Position size T
Different for each image
Mask m
Edge image e
Object appearance ?1
Background appearance ?0
Image
28
Inference

• Aim to infer all latent variables,
• For each image background appearance ?0, object
  appearance ?1, deformation D, transformation T,
  mask m,
• Class variables shape p, edge sprite µo, so.
• Bayesian inference is carried out using
  variational message passing with a fully
  factorised variational distribution.
• Optimisation of grid-structured variational free
  energy terms (relating to the deformation field D
  and the mask m) achieved using graph cuts.

29
Experiments results
30
Experiments

• LOCUS applied to 8 sets of 20 images each
  containing objects of the same class.

• Horses
• Faces
• Cars (rear)
• Cars (side)

• Motorbikes
• Aeroplanes
• Cows
• Trees

For each class, we ran separate experiments for
color and texture appearance models.
31
Results horses
32
Results horses
33
Results cars
34
Results cars
35
Results remaining classes
36
Segmentation accuracy
To evaluate segmentation quantitively, we used
hand segmentations for horses and cars (side).
37
Object registration
Transformation deformation field registers
object outlines (and some internal edges).
38
Object registration
39
Extensions to LOCUS
40
Recognition segmentation

• Object recognition using only global shape

Overall 88 accuracy.
41
Probabilistic Index Maps
2 indices
9 indices
Each image has a palette of appearance models
palette invariance.
42
Probabilistic Index Maps
43
Learning objects from video
Object shape
Object edge sprite
44
Locumotion

• Add flow and track constraints to achieve motion
  segmentation

Tracking/flow estimation by Larry Zitnick
45
Conclusions

• LOCUS gives unsupervised segmentations of
  accuracy equivalent to state-of-the-art
  supervised methods.
• General-purpose model allows
• Object localisation
• Pose estimation
• Object segmentation
• Motion segmentation/object tracking
• Object recognition/detection (in combination with
  discriminative model)

46
Questions ?