Voting objects, their shapes and more A partial review of recent work by Bastian Leibe et al'

1
Voting objects, their shapes and more---- A
partial review of recent work by Bastian Leibe
et al.

• Si, Zhangzhang

2
Contents

• 1. Background Bag-of-words technique
• 2. Implicit shape model
• 3. Bottom-up proposal Top-Down verification
• 4. Dealing with overlapping hypotheses
• 5. Tracking in dynamic scene
• 6. Beyond independent voting

3
Bag of Words

• An image represented as a set of patterns
  (detected by some interest point detector, and
  represented by SIFT descriptors).
• Inference is done by independent voting by
  patterns observed in an image.

Declare a car in the image if number of votes for
car exceeds a certain threshold.
4
Bag of Words

• Different types (clusters) of patterns have
  different votes (weights).
• Obtain pattern types by vector quantization
  (clustering).
• Obtain weights and threshold by support vector
  machines.

K-means clustering
The space of patterns (SIFT descriptors).
Interpretation An instance from cluster-1 has w1
votes for the object category of interest.
5
Implicit Shape Model

• Use patches themselves (scaled down to 2525
  pixels) as patterns.
• A white-box way of voting
• A patch activates several codebook entries
  (clusters).
• Each patch has one vote to cast. (fair?) And
• The hypothesis that gets the most votes wins.

6
Implicit Shape Model

• Problem un-informative patterns cast noisy votes
  which overwhelm informative votes.
• Coping with voting noise
• Agglomerate clustering to ensure patches within a
  cluster are very similar. ? Clutter patches form
  smaller, purer clusters.
• Filter out noisy patches using segmentation.

7
Implicit Shape Model

• Vote not only for object categories, but also
• 1. object center, scale
• 2. pixel-wise figure/ground labels (segmentation)
  for each object hypothesis

8
Implicit Shape Model Model details
Voting object categories and their
locations/scales
Voting for segmentation
In training data there are labeled object masks.
9
Hypotheses as maxima in voting space
Object hypotheses are found as maxima in the 3D
voting space using Mean Shift Mode Estimation
with a scale-adaptive balloon density estimator
and a uniform cubical kernel K.
10
Patch distance, Cluster distance
11
Summary Voting as Bottom-up Proposals

• Voting in Implicit Shape Model a
  straight-forward estimation of .
  X is data (patches in the above context), Y is
  label (category, position, segmentation).
• Space of X is large vector quantization.
• Restrictions
• Voting is independent.
• No hierarchy is present.

12
Top-down verification

• ISM patches are too local, no global
  consistency.
• Given a silhouette, we want to match it to image
  in order to verify the hypothesis.

Matching result
hypothesis
DT of canny edge detection
13
Dealing with overlapping hypotheses
What if two or more hypotheses overlap?
Search for combination of hypotheses that best
explain the image (MDL).
The savings of a particular hypothesis h (it is a
figure/ground segmentation) is
Sdata number of pixels explained by h Smodel a
constant
(inconfidence of the hypothesis)
14
Dealing with overlapping hypotheses
Can be formulated as a Quadratic Boolean
Optimization problem
where the binary vector m indicates which
hypotheses are activated.
given that h_i occludes h_j.
15
Tracking in Dynamic Scene

• It is a system integrating Structure-from-Motion,
  object detection, tracking components.

Trajectory hypotheses
Car/human detectors
Video
Tracking in event cones
2D hypotheses of cars and pedestrians
SfM
3D hypotheses
verify
improve
Scene Geometry
16
2D Hypotheses using ISM detectors

• - Car detectors for 5 views human detector for
  single view.
• Harris-Lap, Hessian-Lap, DoG interest points.
• Local shape context descriptors.
• Result detection segmentation.

Detection
Segmentation
17
Ground plane constraints
Estimate ground plane using joining point of car
wheel and road strip (returned by 2D hypotheses.
Without ground-plane constraint
Without ground-plane constraint
18
3D hypothesis
Likelihood of 2D hypothesis
2D hypothesis
Two 2D hypotheses are consistent if they
correspond to the same 3D object. ? p(hH) is
approximated by clustering 3D detections.
Distance prior (uniform range) Size prior
(Gaussian).
Messy?
Pixels that h occupies
19
Adding temporal information
Now, hypotheses are space-time trajectories.
Dynamic predictive model
20
Trajectory hypotheses
21
Beyond independent voting
Fast mining of feature combinations.
Data to mine from occurrence indicators at
different locations within a neighborhood window.
Concatenated into a vector of NQ.
N words Q tiles.
Supervision each window is labeled object or
background.
Return the feature combinations x if
where
Dangerous!
22
Take-home messages

• Combine different cues (3D geometry 2D
  appearance, static image motion, bottom-up
  proposal top-down verification, shape color)
• Codebook entries should be compact to reduce
  noisy votes (proposals).

23
References
Robust Object Detection with Interleaved
Categorization and SegmentationB. Leibe, A.
Leonardis, B. Schiele.to appear in International
Journal of Computer Vision
Efficient Mining of Frequent and Distinctive
Feature ConfigurationsT. Quack, V. Ferrari, B.
Leibe, L. Van Gool.in International Conference
on Computer Vision (ICCV'07)
Dynamic 3D Scene Analysis from a Moving Vehicle
B. Leibe, N. Cornelis, K. Cornelis, and L. Van
Gool. in IEEE Conference on Computer Vision and
Pattern Recognition (CVPR'07)
Segmentation Based Multi-Cue Integration for
Object DetectionB. Leibe, K. Mikolajczyk, B.
Schiele. in British Machine Vision Conference
(BMVC'06)
Pedestrian Detection in Crowded ScenesB. Leibe,
E. Seemann, and B. Schiele. in IEEE Conference
on Computer Vision and Pattern Recognition
(CVPR'05),
Interleaved Object Categorization and
SegmentationB. Leibe and B. Schiele. in British
Machine Vision Conference (BMVC'03),