Shape%20Descriptors%20I

1
Shape Descriptors I

• Thomas Funkhouser
• CS597D, Fall 2003Princeton University

2
3D Representations

• What properties are required for analysis and
  retrieval?

Analysis
Retrieval
Editing
Display
Property
Intuitive specification Yes No No No Guaranteed
continuity Yes No No No Guaranteed
validity Yes No No No Efficient boolean
operations Yes No No No Efficient
rendering Yes Yes No No Accurate Yes Yes ? ? Conci
se ? ? ? Yes Structure Yes Yes Yes Yes
3
Shape Analysis Problems
1)

• Examples
• Feature detection
• Segmentation
• Labeling
• Registration
• Matching
• Retrieval
• Recognition
• Classification
• Clustering

2)
3)
Query
4)
Ranked Matches
How can we find 3D models best matching a query?
4
Shape

• Definition from Merriam-Websters Dictionary
• a the visible makeup characteristic of a
  particular item or kind of item b spatial
  form or contour

5
Shape

• Shape is independent of similarity transformation
  (rotation, scale, translation, mirror)

6
Shape Similarity

• Need a shape distance function d(A,B) that
• matches our intuitive notion of shape similarity
• can be computed robustly and efficiently
• Perhaps, shape distance function should be a
  metric
• Non-negative d(A,B) ? 0 for all A and B
• Identity d(A,B) 0 if and only if AB
• Symmetry d(A,B) d(B,A) for all A and B
• Triangle inequality d(A,B) d(B,C) ? d(A,C)

7
Example Distance Functions

• Lp norm
• Hausdorff distance
• Others (Fréchet, etc.)

8
Shape Matching

• Compute shape distance function for pair of 3D
  models
• Can matching two objects
• Can find most similar object among a small set

Are these the same chair?
9
Shape Retrieval

• Find 3D models with shape most similar to query
• Searching large database must take less than O(n)

Is this blue chair in the database?
10
Shape Retrieval

• Build searchable shape index

Geometric Query
Shape Analysis
Shape Descriptor
Similar Objects
ShapeRetrieval
Database of 3D Models
Shape Index
Shape Analysis
11
Shape Retrieval

• Find 3D models with shape similar to query

3D Query
Best Matches
3D Database
12
Challenge

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating

3D Query
ShapeDescriptor
BestMatches
3D Database
13
Challenge

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating

3D Query
ShapeDescriptor
BestMatches
3D Database
14
Challenge

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating

3D Query
ShapeDescriptor
BestMatches
3D Database
15
Challenge

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating

3D Query
ShapeDescriptor
BestMatches
3D Database
16
Challenge

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating

3D Query
ShapeDescriptor
BestMatches
3D Database
17
Challenge

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating
• Invariant to transformations
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies

Different Transformations(translation, scale,
rotation, mirror)
18
Challenge
Image courtesy ofRamamoorthi et al.

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating
• Invariant to transformations
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies

Scanned Surface
19
Challenge
Images courtesy of Viewpoint Stanford

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating
• Invariant to transformations
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies

Different Genus
Different Tessellations
20
Challenge
Images courtesy of Utah De Espona

• Need shape descriptor that is
• Concise to store
• Quick to compute
• Efficient to match
• Discriminating
• Invariant to transformations
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies

No Bottom!
Q?_at_A!
21
Taxonomy of Shape Descriptors

• Structural representations
• Skeletons
• Part-based methods
• Feature-based methods
• Statistical representations
• Voxels, moments, wavelets,
• Attributes, histograms, ...
• Point descriptors

22
Taxonomy of Shape Descriptors
Images courtesy of Amenta Osada

• Structural representations
• Skeletons
• Part-based methods
• Feature-based methods
• Statistical representations
• Voxels, moments, wavelets,
• Attributes, histograms, ...
• Point descriptors

23
Taxonomy of Shape Descriptors
Image courtesy of De Espona

• Structural representations
• Skeletons
• Part-based methods
• Feature-based methods
• Statistical representations
• Voxels, moments, wavelets,
• Attributes, histograms, ...
• Point descriptors

?
24
Taxonomy of Shape Descriptors

• Structural representations
• Skeletons
• Part-based methods
• Feature-based methods
• Statistical representations
• Voxels, moments, wavelets,
• Attributes, histograms, ...
• Point descriptors

?
25
Statistical Shape Descriptors

• Alignment-dependent
• Voxels
• Wavelets
• Moments
• Extended Gaussian Image
• Spherical Extent Function
• Spherical Attribute Image

• Alignment-independent
• Shape histograms
• Harmonic descriptor
• Shape distributions

26
Feature Vectors
Image courtesy ofMao Chen

• Map shape onto point in multi-dimensional space
• Similarity measure is distance in feature space

Tables
Feature 1
Desks
File cabinets
Feature 2
27
Feature Vectors
Image courtesy ofMao Chen

• Cluster, classify, recognize, and retrieve
  similarfeature vectors using standard methods

What feature vectors?
Tables
Feature 1
Desks
File cabinets
Feature 2
28
Voxels

• Use voxel values as feature vector (shape
  descriptor)
• Feature space has N3 dimensions (one dimension
  for each voxel)
• d(A,B) A-BN
• Example

d
,
A-B
A
B
N
29
Voxels
Image courtesy ofMisha Kazhdan

• Can store distance transform (DT) in voxels
• A-DT(B)1 represents sum of distances from
  every point on surface of A to closest point on
  surface of B

Surface
Distance Transform
30
Voxels
Image courtesy ofMisha Kazhdan

• Can store distance transform (DT) in voxels
• A-DT(B)1 represents sum of distances from
  every point on surface of A to closest point on
  surface of B

Surface
Distance Transform
31
Voxels
Image courtesy ofDaniel Keim, SIGMOD 1999

• Can build hierarchical search structure
• e.g., interior nodes store MIV and MSV

32
Voxel Retrieval Experiment

• Test database is Viewpoint household
  collection1,890 models, 85 classes

153 dining chairs
25 livingroom chairs
16 beds
12 dining tables
8 chests
28 bottles
39 vases
36 end tables
33
Evaluation Metric

• Precision-recall curves
• Precision retrieved_in_class / total_retrieved
• Recall retrieved_in_class / total_in_class

34
Evaluation Metric

• Precision-recall curves
• Precision 0 / 0
• Recall 0 / 5

1
2
3
4
5
6
Query
7
9
8
Ranked Matches
35
Evaluation Metric

• Precision-recall curves
• Precision 1 / 1
• Recall 1 / 5

1
2
3
4
5
6
Query
7
9
8
Ranked Matches
36
Evaluation Metric

• Precision-recall curves
• Precision 2 / 3
• Recall 2 / 5

1
2
3
4
5
6
Query
7
9
8
Ranked Matches
37
Evaluation Metric

• Precision-recall curves
• Precision 3 / 5
• Recall 3 / 5

1
2
3
4
5
6
Query
7
9
8
Ranked Matches
38
Evaluation Metric

• Precision-recall curves
• Precision 4 / 7
• Recall 4 / 5

1
2
3
4
5
6
Query
7
9
8
Ranked Matches
39
Evaluation Metric

• Precision-recall curves
• Precision 5 / 9
• Recall 5 / 5

1
2
3
4
5
6
Query
7
9
8
Ranked Matches
40
Voxel Retrieval Experiment

• Test database is Viewpoint household
  collection1,890 models, 85 classes

153 dining chairs
25 livingroom chairs
16 beds
12 dining tables
8 chests
28 bottles
39 vases
36 end tables
41
Voxel Retrieval Results
Voxels Random
42
Voxels

• Properties
• Discriminating
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies
• Quick to compute
• Efficient to match?
• Concise to store
• Invariant to transforms

43
Wavelets
Image courtesy ofJacobs, Finkelstein, Salesin

• Define shape with wavelet coefficients

16,000 coefficients
400 coefficients
100 coefficients
20 coefficients
44
Wavelets
Jacobs, Finkelstein, SalesinSIGGRAPH 95

• Descriptor 1
• Given an NxNxN grid, generate an NxNxN array of
  the wavelet coefficients for the standard Haar
  basis functions

45
Wavelets
Jacobs, Finkelstein, SalesinSIGGRAPH 95

• Descriptor 1
• Given an NxNxN grid, generate an NxNxN array of
  the wavelet coefficients for the standard Haar
  basis functions
• Descriptor 2
• Truncate Find the m largest coefficients and set
  all others equal to zero
• Quantize Set the non-zero coefficients to 1 or
  1 depending on their sign

46
Jackie Chan Example

• Original Image (256x256)

47
Truncated And Quantized to 5000
48
Truncated And Quantized to 1000
49
Truncated And Quantized to 500
50
Truncated 100
51
Truncated 50
52
Truncated 10
53
Torus Example
54
Torus Truncated to 5000
55
Torus Truncated to 1000
56
Torus Truncated to 500
57
Torus Truncated to 100
58
Torus Truncated to 50
59
Wavelets
Jacobs, Finkelstein, SalesinSIGGRAPH 95

• Distance Function 1
• The query metric is defined bywhere
  Ai,j,k and Bi,j,k are the truncated and
  quantized coefficients and wi,j,k are weights,
  fine tuned to the database.

60
Wavelets
Jacobs, Finkelstein, SalesinSIGGRAPH 95

• Distance Function 2
• The query metric can be approximated byto
  enable efficient indexing and search.

61
Wavelets
Jacobs, Finkelstein, SalesinSIGGRAPH 95

• Properties
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies
• Quick to compute
• Efficient to match
• Concise to store
• Discriminating?
• Invariant to transforms

62
Moments

• Define shape by moments of inertia

63
Moments Retrieval Experiment

• Test database is Viewpoint household
  collection1,890 models, 85 classes

153 dining chairs
25 livingroom chairs
16 beds
12 dining tables
8 chests
28 bottles
39 vases
36 end tables
64
Moments Retrieval Results
Voxels Moments Elad et al. Random
65
Moments Retrieval Results
Voxels Moments Elad et al. Random
66
Moments

• Properties
• Insensitive to topology
• Robust to degeneracies
• Quick to compute
• Efficient to match
• Concise to store
• Insensitive to noise
• Invariant to transforms
• Discriminating

67
Extended Gaussian Image

• Define shape with histogram of normal directions
• Invertible for convex objects
• Spherical function

3D Model
EGI
68
EGI Retrieval Experiment

• Test database is Viewpoint household
  collection1,890 models, 85 classes

153 dining chairs
25 livingroom chairs
16 beds
12 dining tables
8 chests
28 bottles
39 vases
36 end tables
69
EGI Retrieval Results
Voxels Moments Elad et al. EGI Horn 84 Random
70
Extended Gaussian Images

• Properties
• Insensitive to topology
• Quick to compute
• Efficient to match
• Concise to store
• Insensitve to noise
• Robust to degeneracies
• Invariant to transforms
• Discriminating

71
Other Rotation-Dependent Descriptors
Spherical Extent Functions(Vranic Saupe, 2000)
Shape Histograms (sectors) (Ankherst, 1999)
72
Shape Descriptors II

• Thomas Funkhouser
• CS597D, Fall 2003Princeton University

73
Taxonomy of Shape Descriptors

• Structural representations
• Skeletons
• Part-based methods
• Feature-based methods
• Statistical representations
• Voxels, moments, wavelets,
• Attributes, histograms, ...
• Point descriptors

74
Statistical Shape Descriptors

• Alignment-dependent
• Voxels
• Wavelets
• Moments
• Extended Gaussian Image
• Spherical Extent Function
• Spherical Attribute Image

• Alignment-independent
• Shape histograms
• Harmonic descriptor
• Shape distributions

75
Statistical Shape Descriptors

• Alignment-dependent
• Voxels
• Wavelets
• Moments
• Extended Gaussian Image
• Spherical Extent Function
• Spherical Attribute Image

• Alignment-independent
• Shape histograms
• Harmonic descriptor
• Shape distributions

76
Alignment

• Translation (Center of Mass)
• Scale (Radial Deviation)

77
Alignment

• Rotation (PCA)
• Principal axes are eigenvectors associated with
  largest eigenvalues of 2nd order moments
  covariance matrix

PCA Computation
Principal Axis Alignment
78
Alignment

• Rotation (PCA)
• Principal axes are eigenvectors associated with
  largest eigenvalues of 2nd order moments
  covariance matrix

Not very robust!
79
Alignment

• Mirror
• PCA does not give directions for principal axes

Need heuristics to determine positive axes!
80
Alignment-Independent Descriptors

• Observation it is difficult to normalize for
  differences in rotation and
  mirroring

Three mugs aligned automatically with PCA
Motivation build a shape descriptor that is
invariant to rotations and
mirrors and as discriminating
as possible
81
Shape Histograms
Image courtesy of Ankerst et al, 1999

• Shape descriptor stores histogram of how much
  surface resides at different radii from center of
  mass

Radius
Shape Histograms (shells) (Ankherst, 1999)
82
Shape Histograms
Image courtesy of Misha Kazhdan

• Shape descriptor stores histogram of how much
  surface resides at different radii from center of
  mass

0.7
0.3
0.1
3D Model
SphericalDecomposition
ShapeDescriptor
83
Shape Histogram Experiment

• Test database is Viewpoint household
  collection1,890 models, 85 classes

153 dining chairs
25 livingroom chairs
16 beds
12 dining tables
8 chests
28 bottles
39 vases
36 end tables
84
Shape Histogram Retrieval Results

• Precision-recall curves (mean for all queries)

1
Shape Histogram Ankerst et al. EGI
Horn Moments Elad et al. Random
0.8
0.6
Precision
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
Recall
85
Shape Histograms

• Properties
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies
• Quick to compute
• Efficient to match
• Concise to store
• Invariant to rotations
• Discriminating?

86
Harmonic Shape Descriptor

• Key idea
• Decompose each sphere into irreducible set of
  rotation independent components
• Store how much of the model resides in each
  component

3D Model
HarmonicDecompositions
ShapeDescriptor
87
Step 1 Normalization

• Normalize for translation and scale

3D Model
88
Step 2 Voxelization

• Rasterize polygon surfaces into 3D voxel grid

3D Voxel Grid
89
Step 3 Spherical Decomposition

• Intersect with concentric spheres

Spherical Functions
90
Step 4 Frequency Decomposition

• Represent each spherical function as a sum of
  harmonic frequencies (orders)

Spherical Functions
91
Step 4 Frequency Decomposition

• Represent each spherical function as a sum of
  harmonic frequencies (orders)

SphericalFunction
Spherical Functions
92
Step 4 Frequency Decomposition

• Represent each spherical function as a sum of
  harmonic frequencies (orders)

SphericalFunction
Harmonic Decomposition
93
Step 4 Frequency Decomposition

• Represent each spherical function as a sum of
  harmonic frequencies (orders)

SphericalFunction





Constant
1st Order
2nd Order
94
Step 4 Frequency Decomposition

• Represent each spherical function as a sum of
  harmonic frequencies (orders)

Amplitudes are invariant to rotation





SphericalFunction





Frequency Decomposition
95
Step 5 Amplitude Computation

• Store how much (L2-norm) of the shape resides
  in each harmonic frequency of each sphere

Harmonic Shape Descriptor
96
Matching Harmonic Descriptors

• Define similarity as L2-distance between
  descriptors
• Enables nearest neighbor indexing and fast search
• Provides lower bound for L2-distance between
  models

-
-
-

Sim
,
-
97
Harmonic Shape Descriptor

• Properties
• Concise to store?
• Quick to compute?
• Insensitive to noise?
• Insensitive to topology?
• Robust to degeneracies?
• Invariant to transforms?
• Efficient to match?
• Discriminating?

98
Harmonic Shape Descriptor

• Properties
• Concise to store
• Quick to compute?
• Insensitive to noise?
• Insensitive to topology?
• Robust to degeneracies?
• Invariant to transforms?
• Efficient to match?
• Discriminating?

Polygons
Voxels
Spherical Decomposition
1.6 seconds (on average)
Frequency Decomposition
Harmonic Shape Descriptor
99
Harmonic Shape Descriptor

• Properties
• Concise to store
• Quick to compute?
• Insensitive to noise?
• Insensitive to topology?
• Robust to degeneracies?
• Invariant to transforms?
• Efficient to match?
• Discriminating?

Polygons
Voxels
Spherical Decomposition
1.6 seconds (on average)
Frequency Decomposition
Harmonic Shape Descriptor
100
Harmonic Shape Descriptor

• Properties
• Concise to store
• Quick to compute
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies
• Invariant to transforms?
• Efficient to match?
• Discriminating?

Rasterize polygon surfaces (no solid
reconstruction)
101
Harmonic Shape Descriptor

• Properties
• Concise to store
• Quick to compute
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies
• Invariant to transforms
• Efficient to match?
• Discriminating?

102
Harmonic Shape Descriptor

• Properties
• Concise to store
• Quick to compute
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies
• Invariant to transforms
• Efficient to match?
• Discriminating?

103
Harmonic Shape Descriptor

• Properties
• Concise to store
• Quick to compute
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies
• Invariant to transforms
• Efficient to match?
• Discriminating?

104
Harmonic Matching Results

• Test database is Viewpoint household
  collection1,890 models, 85 classes

153 dining chairs
25 livingroom chairs
16 beds
12 dining tables
8 chests
28 bottles
39 vases
36 end tables
105
Harmonic Retrieval Results

• Precision-recall curves (mean for all queries)

1
Harmonic Shape Descriptor Shape Histogram
Ankerst et al. EGI Horn Moments Elad et
al. Random
0.8
0.6
Precision
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
Recall
106
Statistical Shape Descriptors

• Alignment-dependent
• Voxels
• Wavelets
• Moments
• Extended Gaussian Image
• Spherical Extent Function
• Spherical Attribute Image

• Alignment-independent
• Shape histograms
• Harmonic descriptor
• Shape distributions

107
Shape Distributions

• Motivation general approach to finding a
  common parameterization for matching

3D Surface
Audio
2D Contour
3D Volume
108
Shape Distributions

• Key idea map 3D surfaces to common
  parameterization by randomly sampling shape
  function

Probability
Randomlysampleshape function
Distance
SimilarityMeasure
Probability
Distance
3D Models
D2 Shape Distributions
109
Which Shape Function?

• Implementation simple shape functions based on
  angles, distances, areas, and volumes

?
?
D1 (distance) Ankerst 99
D2 (distance)
A3 (angle)
D3 (area)
D4(volume)
110
D2 Shape Distribution

• Properties
• Concise to store?
• Quick to compute?
• Invariant to transforms?
• Efficient to match?
• Insensitive to noise?
• Insensitive to topology?
• Robust to degeneracies?
• Discriminating?

111
D2 Shape Distribution

• Properties
• Concise to store?
• Quick to compute?
• Invariant to transforms?
• Efficient to match?
• Insensitive to noise?
• Insensitive to topology?
• Robust to degeneracies?
• Discriminating?

Skateboard
Probability
Distance
512 bytes (64 values) 0.5 seconds (106 samples)
112
D2 Shape Distribution

• Properties
• Concise to store
• Quick to compute
• Invariant to transforms?
• Efficient to match?
• Insensitive to noise?
• Insensitive to topology?
• Robust to degeneracies?
• Discriminating?

113
D2 Shape Distribution

• Properties
• Concise to store
• Quick to compute
• Invariant to transforms
• Efficient to match?
• Insensitive to noise?
• Insensitive to topology?
• Robust to degeneracies?
• Discriminating?

Porsche
Skateboard
Probability
Distance
114
D2 Shape Distribution

• Properties
• Concise to store
• Quick to compute
• Invariant to transforms
• Efficient to match
• Insensitive to noise?
• Insensitive to topology?
• Robust to degeneracies?
• Discriminating?

115
D2 Shape Distribution

• Properties
• Concise to store
• Quick to compute
• Invariant to transforms
• Efficient to match
• Insensitive to noise
• Insensitive to topology
• Robust to degeneracies
• Discriminating?

116
D2 Shape Distribution Results

• Question
• How discriminating areD2 shape distributions?
• Test database
• 133 polygonal models
• 25 classes

117
D2 Shape Distribution Results

• D2 distributions are different across classes

D2 shape distributions for 15 classes of objects
118
D2 Shape Distribution Results
D2 distributions for 5 tanks (gray) and 6 cars
(black)
119
D2 Shape Distribution Results

• Similarity Matrix
• Darkness representssimilarity
• Blocks
• Tanks, cars
• Airplanes
• Humans
• Helicopters

120
D2 Retrieval Experiment

• Test database is Viewpoint household
  collection1,890 models, 85 classes

153 dining chairs
25 livingroom chairs
16 beds
12 dining tables
8 chests
28 bottles
39 vases
36 end tables
121
D2 Retrieval Results

• Precision-recall curves (mean for all queries)

1
Harmonic Shape Descriptor D2 Shape Distribution
Osada et al. Shape Histogram Ankerst et
al. EGI Horn Moments Elad et al. Random
0.8
0.6
Precision
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
Recall
122
Shape Distributions

• Next steps
• Better shape functions
• Better comparsion methods
• Analysis apps

123
D2 Shape Distribution Results

• Recognizing gross shapes with D2 distributions

D2 shape distributions for 15 classes of objects
124
D2 Shape Distribution Results

• Recognizing gross shapes with D2 distributions

D2 shape distributions for 15 classes of objects
125
D2 Shape Distribution Results

• Recognizing gross shapes with D2 distributions

D2 shape distributions for 15 classes of objects
126
D2 Shape Distribution Results

• Recognizing gross shapes with D2 distributions

D2 shape distributions for 15 classes of objects
127
D2 Shape Distribution Results

• Recognizing gross shapes with D2 distributions

D2 shape distributions for 15 classes of objects
128
Taxonomy of Shape Descriptors

• Structural representations
• Skeletons
• Part-based methods
• Feature-based methods
• Statistical representations
• Voxels, moments, wavelets,
• Attributes, histograms, ...
• Point descriptors

129
Taxonomy of Shape Descriptors

• Structural representations
• Skeletons
• Part-based methods
• Feature-based methods
• Statistical representations
• Voxels, moments, wavelets,
• Attributes, histograms, ...
• Point descriptors

Next Time!