Semantic Shift for Unsupervised Categorization and Localization

1
Semantic Shift for Unsupervised Categorization
and Localization

• David Liu and Tsuhan Chen
• Electrical and Computer Engineering Department
• Carnegie Mellon University

2
Parts-based Artworks

• As long as the parts are present, the exact
  position does not matter too much

Picasso, 1943
Caltech face dataset
Dali, 1936
3
Overview

• Probabilistic Latent Semantic Analysis (PLSA)
• Incorporating spatial information
• Prior work
• Proposed method Semantic-Shift
• Results

4
Goal Unsupervised Image Categorization
Caltech face background dataset
5
Goal Unsupervised Image Categorization
UIUC car dataset
6
Documents and Images Are Analogous
Documents
Images
Topic(Sport, Health, )
Topic (Face, Car, )
Word
Visual word
7
Visual Words
Difference of Gaussian interest point detector
SIFT descriptor (Lowe 99)
Vector quantization
8
Bag of Words Representation
count
words
9
A Categorization Model

• Text categorization (Nigam et al. 98)
• Image categorization (Csurka et al. 04)

count
words
words
w
w
zface
zcar
w
w
Topic Appearance
w
w
10
The Effect of Background Clutter
11
Probabilistic Latent Semantic Analysis (PLSA)

• Hofmann 01, Monay and Gatica-Perez 04, Sivic et
  al. 05, Quelhas et al. 05
• Models complex scenes

z
w
d
z
w
Document Characteristic
Topic Appearance
doc 1
z
w

• Inference
• Categorization
• Segmentation

• Learning Maximum likelihood estimation using EM
  algorithm

z
w
d
z
w
doc 2
z
w
12
PLSA Difficulties
13
Incorporating Spatial Information Prior Work
Liu Chen, 06

• A number of S 10 fixed spatial distributions

position
s
x
d
z
w
Nd
s1 to s9
s10
D
appearance
14
Proposed Method Semantic-Shift
x
image topic word appearance word
position
d
z
w
Nd
D
Topic Appearance
Document Characteristic
Location Semantics
15
Location Semantics
x
d
z
w
Nd
D

• Assume single foreground object
• Location semantics
• Foreground
• Background Complement of foreground
  distribution

16
Semantic-Shift Algorithm for Learning
Document Characteristic
Topic Appearance
bag of word-position pairs
17
Semantic-Shift Algorithm for Inference

• Inference
• Categorization
• Segmentation

18
Semantic-Shift Demo
Location semantics
posterior
Topic appearance
19
PLSA
Semantic-Shift
20
Categorization Performance on Face Dataset
AUC
PLSA
Semantic-Shift
21
Categorization Performance on Car Dataset
AUC
PLSA
Semantic-Shift
22
Summary

• Unsupervised categorization and segmentation
• Several ways of including spatial information
• Proposed method Semantic Shift
• Future extensions
• Multiple topics
• Multiple objects

23
(No Transcript)
24
Incorporating Spatial Information Prior Work (I)

• 2D Discriminative Random Fields (Kumar Hebert
  03)

• Extension to video (Liu Chen 06)

25
Proposed Method Semantic-Shift

• Appearance analyze across images
• Location analyze within image
• Appearance and location estimations are coupled

26

• Initialize P(zd,w) with PLSA
• Identify foreground topic
• Estimate location and scale
• Semantic-Shift

27
Semantic-Shift
PLSA
PLSA
Semantic Shift
Semantic Shift
LocationSemanticsare shifting
28
x
d
z
w
x
w
z
D
29
Datasets

• UIUC car dataset
• Training 550 cropped car images 382 non-car
  images
• Testing 170 car images 118 non-car images
• Caltech face background dataset
• 450 frontal face images of 27 or so unique people
  
• 550 images of assorted scenes
• Half for training, half for testing

30
Categorization Performance on Face Dataset