Representing, Learning, and Recognizing NonRigid

1
Representing, Learning, and Recognizing Non-Rigid
Textures and Texture Categories Svetlana
Lazebnik Cordelia Schmid Jean Ponce Beckman
Institute Gravir Laboratory Beckman
Institute UIUC, USA INRIA, France UIUC, USA
Supported in part by the UIUC Campus Research
Board, the UIUC/CNRS Collaborative Research
Agreement, and the National Science Foundation
under grant IRI-990709.
2
LeCun03

• Affine-invariant patches.

• 3D objects are never planar in the large,
• but they are always planar in the small.

• Representation Local invariants and
• their spatial layout.

3
(Lindeberg Garding97)
(Mikolcajczyk Schmid02)
4

• Spatial selection
• Shape selection

• Affine adaption

Schaffalitzky Zisserman (2001) Tuytelaars
Van Gool (2003)
5
Affine adaptation/Rectification process
Image 2
Image 1
(0,1)
(1,0)
(0,0)
Lindeberg Garding (1997)
Rectified patch
Mikolcajczyk Schmid (2002)
6
Intensity-Domain Spin Images
Range spin images Johnson Hebert (1998)
7
System architecture (Lazebnik, Schmid, Ponce,
CVPR03)

• Signature S ( m1 , w1 ) , , ( mk , wk
  )

• Earth Movers Distance D( S , S ) ?i,j
  fij d( mi , mj) / ?i,j fij

Signatures and EMD for image retrieval Rubner,
Tomasi, Guibas (1998)
8
Texture retrieval/classification experiments
10 texture classes, with 20 samples per class.
Schmid (2001) Varma Zisserman (2002)
Average recognition rate
Average recognition rate
NN classification
9
More retrieval/classification experiments
Brodatz database
Average recognition rate
Average recognition rate
111 images divided into 9 windows 111 classes
with 9 samples per class

• Picard et al. (1993, 1996)
• Xu et al. (2000)

10
Texture Classes
NOTE we do NOT use color information.
T1 (brick)
T2 (carpet)
T3 (chair)
T4 (floor 1)
T5 (floor 2)
T6 (marble)
T7 (wood)
Multi-texture Samples
11
A Two-Layer Architecture (Lazebnik, Schmid,
Ponce, ICCV03)

• Modeling
• Use EM to learn a mixture-of-Gaussians model of
  each texture class.
• Compute co-occurrence statistics of sub-class
  labels over affinely adapted neighborhoods.

• Recognition
• Use the generative model to obtain initial class
• membership probabilities.
• Use relaxation (Rosenfeld et al., 1976) to refine
  these probabilities.

Malik, Belongie, Leung, Shi (2001) Schmid
(2001) Kumar Hebert (2003)
12
Neighborhood Statistics

• Estimate
• probability p(c,c),
• correlation r(c,c).

13
Relaxation (Rosenfeld et al., 1976)
Iterate, for all regions i
where
and wij0 is region j is not in the neighborhood
of i, with ?j wij1.
14
Classification rates for single-texture images
10 training images per class, 10 test images per
class.
15
Weakly-Supervised Modeling
Idea Replace L mixture models with M components
by a single mixture model with L x M
components.

• Annotate each image with the set C of labels
• associated with classes occurring in it.
• Run EM
• E step update class membership probabilities
• p (clm x, C ) / p ( x clm ) p ( clm C
  ).
• M step update model parameters.

Nigam, McCallum, Thrun Mitchell (2000)
16
ROC Curves
Single-texture training images only
T1 (brick)
T2 (carpet)
T3 (chair)
T4 (floor 1)
T5 (floor 2)
T6 (marble)
T7 (wood)
Single- and multi-texture training images
T1 (brick)
T2 (carpet)
T3 (chair)
T4 (floor 1)
T5 (floor 2)
T6 (marble)
T7 (wood)
10 single-texture images per class, 13
two-texture training images, 45
multi-texture test images.
17
Effect of relaxation on labeling
Original image
Top before relaxation, bottom after relaxation
18
Successful Segmentation Examples
19
Unsuccessful Segmentation Examples
20
Animal Dataset

• No manual segmentation.

• 10 training images for each animal background,
  20 test images per class.

Bradshaw, Scholkopf, Platt (2001) Schmid
(2001) Kumar Hebert (2003)
21
(No Transcript)
22
Oh well..
23
(No Transcript)
24

• 3D Objects without distinctive texture
• Category-level recognition
• of 3D objects
• Please join us in trying to solve the
• 3D object recognition problem..