Object detection using local and global features

1
Object detection using local and global features

• Daniel Eaton, Kevin Murphy, UBC
• Based on joint work with
• Antonio Torralba, Bill Freeman, MIT
• 13 July 2005

2
Outline

• Current approaches to object detection
• Need for global context
• Local features
• Global features
• Combining local and global features
• Multi-class detection

3
Standard approachsliding window classifier
Examples ViolaJones, Schneiderman Kanade,
Poggio et al.
Slide from Leon Sigal
4
Handling different sizesapply classifier to
image pyramid
Source Leon Sigal
5
Other approaches

• Pictorial structures build classifier for each
  part of an object, and combine using graphical
  model (Felzenswab Huttenlocher)
• Constellation models compute sparse set of
  features, find subset with low energy (good
  appearance and shape) (Fergus, Perona Zisserman)

6
Standard approaches to object localization are
too myopic
Local pieces of image are ambiguous
7
A car on the street?
8
An ashtray on the table?
9
Global scene context affects interpretation of
local patches
Pictures from Torralba, IJCV 2003
10
Class-presence can be inferred from global image
features
11
Proposed model
G
Global features (whole image)
O
X
Object presence (0absent, 1present)
Location/scale of object
L
Local features (patches)

• We will train classifiers to combine local and
  global features to solve 2 tasks
• Object presence detection
• Object localization

Simplification of model in Murphy, Torralba,
Freeman, NIPS 2003
(In previous model, X was a bank of N binary
variables here X is single N-ary variable)
12
Dataset

• Derived from MIT-CSAIL Scenes Objects (Torralba
  Murphy)
• 1,217 images
• Has some annotation errors (unlabelled data,
  mislabelled data)

13
Outline

• Current approaches to object detection
• Need for global context
• Local features
• Global features
• Combining local and global features
• Multi-class detection

14
Local features for localization
G
Global features (gist of image)
O
X
Object presence (0absent, 1present)
Location/scale of object
L
Local features (patches)
P(XiL)bi
15
Features filtered fragments
P
Feature representation
What gets stored
?

f
g
Computing the feature on a novel image


)
(
?
?
?
?f
?g
?P
16
Feature vectors

• Feature dictionary composed of N features (filter
  and patch source/size/location chosen at random)

1
2
N
-Apply all features to an image
Positive vector
1
Negative vectors
2
3

N
-Training Compute features, sample positive
negative vectors, train classifier with gentle
boost (weak learners decision stumps) -Test
Compute features, apply classifier at each pixel
17
Classifier boosted decision trees

• Best off-the-shelf classifier in the world
  Leo Breiman, 1998
• 1 node tree stump (weak learner)
• Used for feature selection
• Fast to train, fast to apply
• Gentle boost for boosting algorithm (real-valued
  confidences)

18
Performance of localizationusing local features
(on UIUC)

• Baseline performance indicator
• PASCAL-VOC PR criterion

1-
19
Local features for presence
G
Global features (gist of image)
O
X
Object presence (0absent, 1present)
Location/scale of object
L
Local features (patches)
P(O1L) ?(wT 1 maxi Li)
If O0, then X is undefined.
20
Outline

• Current approaches to object detection
• Need for global context
• Local features
• Global features
• Combining local and global features
• Multi-class detection

21
The gist of the image

• Captures spatially localized texture/structure
  info.
• Average energy of filter response at multiple
  scales, orientations and locations
• Dimensionality reduction via PCA

Oliva Torralba, 2001
22
Global features for localization
G
Global features (gist of image)
Object presence (0absent, 1present)
X
Location/scale of object
O
L
Local features (patches)
P(XG) modeled using mixture density network
(c.f., mixture of experts)
23
Global features for localization
24
Performance of localization using global features

• Scenes are arranged in horizontal layers

25
Global features for presence
G
Global features (gist of image)
O
Object presence (0absent, 1present)
X
Location/scale of object
L
Local features (patches)
P(O1G) modeled using naïve Bayes
classifier(i.e., diagonal covariance Gaussian)
26
Outline

• Current approaches to object detection
• Need for global context
• Local features
• Global features
• Combining local and global features
• Multi-class detection

27
Localization using local and global features
G
Global features (gist of image)
Object presence (0absent, 1present)
O
X
Location/scale of object
L
Local features (patches)
We use a product of experts model
P(XiG,L) (1/Z) P(XiG)? P(XiL)
28
Performance of localization using local and
global features
- Quantitative results soon
29
Presence detection using local and global features
G
Global features (gist of image)
Object presence (0absent, 1present)
O
Location/scale of object
X
L
Local features (patches)
P(O1G,L) ?(wT 1 P(O1G) P(O1L))
30
Performance of presence detection using local
global features (1/3)
31
Performance of presence detection using local
global features (2/3)
32
Performance of presence detection using local
global features (3/3)
33
Outline

• Current approaches to object detection
• Need for global context
• Local features
• Global features
• Combining local and global features
• Multi-class detection, Future (Immediate) Work

34
Many objects co-occur
35
Multi-class detection
G
O
O
X
X
L
L
Gist feature can be shared across classes
36
We assume that correlations have hidden common
cause scene-type
Office
Street
37
Adding categorical scene-type
S
O
X
X
O
L
L
G
Scene classification using global image features
and object presence. We can classify scene and
then run object detectors, or vice versa.
Gist not good at discriminating between kitchen
vs. bathroom, but hopefully we can resolve
ambiguity by detecting local objects (table chair
vs. toilet)
38
Future work 1pictorial structures for scenes
X
X
L
L
G

• - Model spatial correlation between objects
  rather than parts (weaker constraints).
• - Learn optimal tree-structured model.
• (Eliminate O nodes by having special state X0
  to indicate absence.)
• How handle multiple instances? (Want to avoid
  data association problem.)

39
Future work 2combining feature sharing and gist
X2
X1
L1
L2
G
L12

• G is shared across classes and across space.
• - Feature sharing can discover shared local
  features(good for classifying many objects)
• - What about features that are partially shared
  across classesand across space (good for
  localizing many objects)?
• - Complex features can combine across space or
  across channels

40
Future work 3dynamic re-ordering of features
X2
X1
f1
f2
f12
f1
f2
f12

• When we have many possible features we could use
  (shared to different degrees across classes and
  space), what order should we evaluate them in?
• When should we send spatial messages?
• Policy should depend on current belief state
  (i.e., be opportunistic).

41
What does the gist capture?
Identical-gist (pairwise)
42
Feature filters

• Composed of

Dictionary of filters. Filter 1 is a delta
function, 27 are 3x3 Gaussian derivatives, 8 is
a 3x3 Laplacian, 9 is a 5x5 corner detector,
1013 are long edge detectors (of size 3x5, 3x7,
5x3 and 7x3).
-Patch size ranges from 7x7-21x21