Agenda

1
Agenda

• Introduction
• Bag-of-words model
• Visual words with spatial location
• Part-based models
• Discriminative methods
• Segmentation and recognition
• Recognition-based image retrieval
• Datasets Conclusions

2
(No Transcript)
3
Analogy to documents
Of all the sensory impressions proceeding to the
brain, the visual experiences are the dominant
ones. Our perception of the world around us is
based essentially on the messages that reach the
brain from our eyes. For a long time it was
thought that the retinal image was transmitted
point by point to visual centers in the brain
the cerebral cortex was a movie screen, so to
speak, upon which the image in the eye was
projected. Through the discoveries of Hubel and
Wiesel we now know that behind the origin of the
visual perception in the brain there is a
considerably more complicated course of events.
By following the visual impulses along their path
to the various cell layers of the optical cortex,
Hubel and Wiesel have been able to demonstrate
that the message about the image falling on the
retina undergoes a step-wise analysis in a system
of nerve cells stored in columns. In this system
each cell has its specific function and is
responsible for a specific detail in the pattern
of the retinal image.
4
A clarification definition of BoW

• Independent features
• Histogram representation of image
• Discrete appearance representation

5
Representation
2.
1.
3.
6
1.Feature detection and representation
7
1.Feature detection and representation

• Regular grid
• Vogel Schiele, 2003
• Fei-Fei Perona, 2005

8
1.Feature detection and representation

• Regular grid
• Vogel Schiele, 2003
• Fei-Fei Perona, 2005
• Interest point detector
• Csurka, et al. 2004
• Fei-Fei Perona, 2005
• Sivic, et al. 2005

9
1.Feature detection and representation
Compute SIFT descriptor Lowe99
Normalize patch
Detect patches Mikojaczyk and Schmid 02 Mata,
Chum, Urban Pajdla, 02 Sivic Zisserman,
03
Slide credit Josef Sivic
10
1.Feature detection and representation
11
2. Codewords dictionary formation
12
2. Codewords dictionary formation
Vector quantization
Slide credit Josef Sivic
13
2. Codewords dictionary formation
Fei-Fei et al. 2005
14
Image patch examples of codewords
Sivic et al. 2005
15
3. Image representation
frequency
codewords
16
Representation
2.
1.
3.
category models (and/or) classifiers
17
Learning and Recognition
category models (and/or) classifiers
18
Learning and Recognition

• Generative method
• - topic models
• Discriminative method
• - SVM

category models (and/or) classifiers
19
Probabilistic Latent Semantic Analysis (pLSA)

• Background Hoffman, 2001 Blei, Ng Jordan,
  2004 ? Latent Dirichlet Allocation
• Object categorization
• Sivic et al. 2005 Sudderth et al. 2005
• Natural scene categorization Fei-Fei et al. 2005
• In this case, use it for unsupervised
  learningfrom image collections

20
Probabilistic Latent Semantic Analysis
dj the jth image in an image collection z
latent theme or topic of the patch N number of
patches per image wi visual word of patch
Sivic et al. ICCV 2005
21
Feature detection and representation
Image collection
d
w
P(widj)
22
The pLSA model
Slide credit Josef Sivic
23
Learning the pLSA parameters
Observed counts of word i in document j
Maximize likelihood of data using EM
M number of codewords N number of images
Slide credit Josef Sivic
24
Recognition using pLSA
Slide credit Josef Sivic
25
Demo

• Course website

26
task face detection no labeling
27
Demo learnt parameters

• Learning the model do_plsa(config_file_1)
• Evaluate and visualize the model
  do_plsa_evaluation(config_file_1)

Codeword distributions per theme (topic)
Theme distributions per image
28
Demo recognition examples
29
Learning and Recognition

• Generative method
• - topic models
• Discriminative method
• - SVM

category models (and/or) classifiers
30
Discriminative methods based on bag of words
representation

• Grauman Darrell, 2005, 2006
• SVM w/ Pyramid Match kernels
• Others
• Csurka, Bray, Dance Fan, 2004
• Serre Poggio, 2005

31
Summary Pyramid match kernel
optimal partial matching between sets of features

• Pyramid is in feature space, spatial information
  not used
• Efficient to compute linear in
  features/image
• Satisfies Mercer Condition, so can be used as a
  kernel in an SVM

Grauman Darrell, 2005, Slide credit Kristen
Grauman
32
Pyramid Match (Grauman Darrell 2005)
Histogram intersection
Slide credit Kristen Grauman
33
Pyramid Match (Grauman Darrell 2005)
Histogram intersection
Slide credit Kristen Grauman
34
Pyramid match kernel

• Weights inversely proportional to bin size
• Normalize kernel values to avoid favoring large
  sets

Slide credit Kristen Grauman
35
Example pyramid match
Level 0
Slide credit Kristen Grauman
36
Example pyramid match
Level 1
Slide credit Kristen Grauman
37
Example pyramid match
Level 2
Slide credit Kristen Grauman
38
Example pyramid match
pyramid match
optimal match
Slide credit Kristen Grauman
39
Object recognition results

• ETH-80 database 8 object classes
• (Eichhorn and Chapelle 2004)
• Features
• Harris detector
• PCA-SIFT descriptor, d10

Kernel Complexity Recognition rate
Match Wallraven et al. 84
Bhattacharyya affinity Kondor Jebara 85
Pyramid match 84
d descriptor dim. m features L
levels in pyramid
Slide credit Kristen Grauman