Learning Decompositional Shape Models from Examples

1
Learning Decompositional Shape Models from
Examples

• Alex Levinshtein
• Cristian Sminchisescu
• Sven Dickinson
• University of Toronto

2
Hierarchical Models
Manually built hierarchical model proposed by
Marr And Nishihara (Representation and
recognition of the spatial organization of three
dimensional shapes, Proc. of Royal Soc. of
London, 1978)
3
Our goal
Automatically construct a generic hierarchical
shape model from exemplars

• Challenges
• Cannot assume similar appearance among different
  exemplars
• Generic features are highly ambiguous
• Generic features may not be in one-to-one
  correspondence

4
Automatically constructed Hierarchical Models
Input
Question What is it?
Output
5
Stages of the system
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
6
Blob Graph Construction
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
7
Blob Graph Construction
On the Representation and Matching of Qualitative
Shape at Multiple Scales A. Shokoufandeh, S.
Dickinson, C. Jonsson, L. Bretzner, and T.
Lindeberg,ECCV 2002

• Edges are invariant to articulation
• Choose the largest connected component.

8
Feature matching
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
9
Feature matching
One-to-one matching. Rely on shape and context,
not appearance!
Many-to-many matching
?
10
Feature embedding and EMD
Spectral embedding
11
Returning to our set of inputs

• Many-to-many matching of every pair of exemplars.

12
Part Extraction
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
13
(No Transcript)
14
Results of the part extraction stage
15
What is next?
16
Extracting attachment relations
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
17
Extracting attachment relations
Right arm is typically connected to torso in
exemplar images !
18
Extracting decomposition relations
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
19
Extracting decomposition relations
20
Model construction stage summary
Model Construction

• Clustering blobs into parts based on one-to-one
  matching results.
• Recovering relations between parts based on
  individual matching and attachment results.

21
Assemble Final Model
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
22
(No Transcript)
23
(No Transcript)
24
Conclusions

• General framework for constructing a generic
  decompositional model from different exemplars
  with dissimilar appearance.
• Recovering decompositional relations requires
  solving the difficult many-to-many graph matching
  problem.
• Preliminary results indicate good model recovery
  from noisy features.

25
Future work

• Construct models for objects other than humans.
• Provide scale invariance during matching.
• Automatically learn perceptual grouping relations
  from labeled examples.
• Develop indexing and matching framework for
  decompositional models.