Title: Voting objects, their shapes and more A partial review of recent work by Bastian Leibe et al'
1Voting objects, their shapes and more---- A
partial review of recent work by Bastian Leibe
et al.
2Contents
- 1. Background Bag-of-words technique
- 2. Implicit shape model
- 3. Bottom-up proposal Top-Down verification
- 4. Dealing with overlapping hypotheses
- 5. Tracking in dynamic scene
- 6. Beyond independent voting
3Bag of Words
- An image represented as a set of patterns
(detected by some interest point detector, and
represented by SIFT descriptors). - Inference is done by independent voting by
patterns observed in an image.
Declare a car in the image if number of votes for
car exceeds a certain threshold.
4Bag of Words
- Different types (clusters) of patterns have
different votes (weights). - Obtain pattern types by vector quantization
(clustering). - Obtain weights and threshold by support vector
machines.
K-means clustering
The space of patterns (SIFT descriptors).
Interpretation An instance from cluster-1 has w1
votes for the object category of interest.
5Implicit Shape Model
- Use patches themselves (scaled down to 2525
pixels) as patterns. - A white-box way of voting
- A patch activates several codebook entries
(clusters). - Each patch has one vote to cast. (fair?) And
- The hypothesis that gets the most votes wins.
6Implicit Shape Model
- Problem un-informative patterns cast noisy votes
which overwhelm informative votes. - Coping with voting noise
- Agglomerate clustering to ensure patches within a
cluster are very similar. ? Clutter patches form
smaller, purer clusters. - Filter out noisy patches using segmentation.
7Implicit Shape Model
- Vote not only for object categories, but also
- 1. object center, scale
- 2. pixel-wise figure/ground labels (segmentation)
for each object hypothesis
8Implicit Shape Model Model details
Voting object categories and their
locations/scales
Voting for segmentation
In training data there are labeled object masks.
9Hypotheses as maxima in voting space
Object hypotheses are found as maxima in the 3D
voting space using Mean Shift Mode Estimation
with a scale-adaptive balloon density estimator
and a uniform cubical kernel K.
10Patch distance, Cluster distance
11Summary Voting as Bottom-up Proposals
- Voting in Implicit Shape Model a
straight-forward estimation of .
X is data (patches in the above context), Y is
label (category, position, segmentation). - Space of X is large vector quantization.
- Restrictions
- Voting is independent.
- No hierarchy is present.
12Top-down verification
- ISM patches are too local, no global
consistency. - Given a silhouette, we want to match it to image
in order to verify the hypothesis.
Matching result
hypothesis
DT of canny edge detection
13Dealing with overlapping hypotheses
What if two or more hypotheses overlap?
Search for combination of hypotheses that best
explain the image (MDL).
The savings of a particular hypothesis h (it is a
figure/ground segmentation) is
Sdata number of pixels explained by h Smodel a
constant
(inconfidence of the hypothesis)
14Dealing with overlapping hypotheses
Can be formulated as a Quadratic Boolean
Optimization problem
where the binary vector m indicates which
hypotheses are activated.
given that h_i occludes h_j.
15Tracking in Dynamic Scene
- It is a system integrating Structure-from-Motion,
object detection, tracking components.
Trajectory hypotheses
Car/human detectors
Video
Tracking in event cones
2D hypotheses of cars and pedestrians
SfM
3D hypotheses
verify
improve
Scene Geometry
162D Hypotheses using ISM detectors
- - Car detectors for 5 views human detector for
single view. - Harris-Lap, Hessian-Lap, DoG interest points.
- Local shape context descriptors.
- Result detection segmentation.
Detection
Segmentation
17Ground plane constraints
Estimate ground plane using joining point of car
wheel and road strip (returned by 2D hypotheses.
Without ground-plane constraint
Without ground-plane constraint
183D hypothesis
Likelihood of 2D hypothesis
2D hypothesis
Two 2D hypotheses are consistent if they
correspond to the same 3D object. ? p(hH) is
approximated by clustering 3D detections.
Distance prior (uniform range) Size prior
(Gaussian).
Messy?
Pixels that h occupies
19Adding temporal information
Now, hypotheses are space-time trajectories.
Dynamic predictive model
20Trajectory hypotheses
21Beyond independent voting
Fast mining of feature combinations.
Data to mine from occurrence indicators at
different locations within a neighborhood window.
Concatenated into a vector of NQ.
N words Q tiles.
Supervision each window is labeled object or
background.
Return the feature combinations x if
where
Dangerous!
22Take-home messages
- Combine different cues (3D geometry 2D
appearance, static image motion, bottom-up
proposal top-down verification, shape color) - Codebook entries should be compact to reduce
noisy votes (proposals).
23References
Robust Object Detection with Interleaved
Categorization and SegmentationB. Leibe, A.
Leonardis, B. Schiele.to appear in International
Journal of Computer Vision
Efficient Mining of Frequent and Distinctive
Feature ConfigurationsT. Quack, V. Ferrari, B.
Leibe, L. Van Gool.in International Conference
on Computer Vision (ICCV'07)
Dynamic 3D Scene Analysis from a Moving Vehicle
B. Leibe, N. Cornelis, K. Cornelis, and L. Van
Gool. in IEEE Conference on Computer Vision and
Pattern Recognition (CVPR'07)
Segmentation Based Multi-Cue Integration for
Object DetectionB. Leibe, K. Mikolajczyk, B.
Schiele. in British Machine Vision Conference
(BMVC'06)
Pedestrian Detection in Crowded ScenesB. Leibe,
E. Seemann, and B. Schiele. in IEEE Conference
on Computer Vision and Pattern Recognition
(CVPR'05),
Interleaved Object Categorization and
SegmentationB. Leibe and B. Schiele. in British
Machine Vision Conference (BMVC'03),