SemiLocal Affine Parts for Object Recognition

1
Semi-Local Affine Partsfor Object Recognition

• Svetlana Lazebnik
• Jean PonceUniversity of Illinois at
  Urbana-Champaign
• Cordelia Schmid
• INRIA Rhône-Alpes

2
Overview

• Goal
• Learning models for recognition of 3D object
  classes
• Challenges
• Geometric invariance
• Robustness to clutter, occlusion
• Weakly supervised learning
• Proposed approach
• An object representation using semi-local affine
  parts

3
Low-Level Features Local Affine Regions

• Detector Laplacian (Gårding Lindeberg, 1996)
• Descriptors spin images and RIFT (Lazebnik et
  al., 2004)

4
Learning Parts

• Ideal approach simultaneous correspondence
  search across entire training set

5
Two-Image Matching

• Goal to find collections of local affine regions
  that can be mapped onto each other using a single
  affine transformation
• Implementation greedy search based on geometric
  and photometric consistency constraints
• Returns multiple correspondence hypotheses
• Automatically determines number of regions in
  correspondence
• Works on unsegmented, cluttered images (weakly
  supervised learning)

A
6
Matching Details

• Initialization
• Identify triples of neighboring regions (i, j, k)
  in first image
• Find all triples (i', j', k') in the second image
  such that i' (resp. j', k') is a potential match
  of i (resp. j, k), and j', k' are neighbors of i'

j
j'
i
i'
k'
k
7
Matching (cont.)

• Beginning with each seed triple, iterate
• Estimate the affine transformation between
  centers of corresponding regions in current group
  of matches

A
8
Matching (cont.)

• Beginning with each seed triple, iterate
• Estimate the affine transformation between
  centers of corresponding regions in current group
  of matches
• Determine geometric consistency of current group
  of matches

• Geometric consistency criteria
• Distance between ellipse centers (residual)
• Difference of major and minor axis lengths
• Difference of ellipse orientations

9
Matching (cont.)

• Beginning with each seed triple, iterate
• Estimate the affine transformation between
  centers of corresponding regions in current group
  of matches
• Determine geometric consistency of current group
  of matches
• Search for additional matches in the neighborhood
  of the current group

10
Matching 3D Objects
11
Matching 3D Objects
closeup
closeup
12
Matching Faces
spurious match ???
13
Application Finding Local Symmetries
14
Application Finding Repeated Patterns
15
Learning Object Models

• Match multiple pairs of training images to
  produce a set of candidate parts
• Evaluate repeatability of candidate parts on
  validation set
• Region repeatability
• Part repeatability
• Retain a fixed number of parts having the best
  repeatability score

of hypotheses in which region was
detected total of hypotheses
total of detected regions in all
hypotheses total of hypotheses part size
16
Recognition

• For each test image and each class c, compute the
  relative repeatability score
• Evaluation multi-class classification and binary
  detection

Si number of regions in hypothesis of pc,i Si
size of pc,i
17
Recognition Experiment Butterflies
Admiral Swallowtail Machaon
Monarch 1 Monarch 2 Peacock
Zebra

• 26 training images per class
• 8 initial pairs
• 10 validation images
• 437 test images
• 619 images total

18
Butterfly Parts
19
Classification Results
(top 10 parts retained following validation)
Total part size (smallest/largest)
20
Classification Rate vs. Number of Parts
21
Detection Results (ROC Curves)
Circles reference relative repeatability rates.
Red square ROC equal error rate (in parentheses)
22
Successful Detection Examples
Training images
Test images (blue occluded regions)
All regions found in the test images
23
Unsuccessful Detection Examples
Training images
Test images (blue occluded regions)
All regions found in the test images
24
Future Work

• Improve the local features
• Alternative affine region detectors Kadir et al.
  (2004), Matas et al. (2002), Mikolajczyk Schmid
  (2002), Tuytelaars Van Gool (2004), etc.
• Heterogeneous features (regions, curve segments,
  etc.)
• Model inter-part relations
• More extensive evaluation

25
Improving the Local Features

• Edge-based ellipse detector (extension of Jurie
  Schmid, 2004)

• Compare with the Laplacian detector

26
Relations Proposed Approach

• Treat semi-local affine parts as black boxes
• Define binary neighborhood relations between
  parts
• Learn a generative model or a classifier over the
  resulting feature representation

p3
p1
p4
Related work Agarwal Roth (2002)
27
Evaluation Birds
Egret
Puffin
Snowy Owl
Mandarin Duck
Wood Duck
28
Candidate Parts
Wood Duck
Puffin
29
Summary

• Finding a part vocabulary
• Weakly supervised learning
• Direct correspondence search (vs. EM)
• Learning from small training sets
• No explicit background modeling
• Additional application
• Finding symmetries, repetitions within a single
  image
• Future work
• Need better detectors/descriptors
• Need relations