Shape-Representation

1
Shape-Representation
and
Shape Similarity
Part 1 Shapes
Dr. Rolf Lakaemper
2
May I introduce myself

• Rolf Lakaemper
• PhD (Doctorate Degree) 2000
• Hamburg University, Germany
• Currently Assist. Professor at Department
• of Computer and Information Sciences,
• Temple University, Philadelphia, USA
• Main Research Area Computer Vision

3
Motivation
WHY SHAPE ?
4
Motivation
These objects are recognized by
5
Motivation
These objects are recognized by
Texture Color Context Shape
X X
X X
X
X
X
X X
6
Why Shape ?
Several applications in computer vision use shape
processing Object recognition Image
retrieval Processing of pictorial information
Video compression (eg. MPEG-7) This
presentation focuses on object recognition and
image retrieval.
7
Motivation
Typical Application Multimedia Image Database
Query by Shape / Texture / (Color / Keyword)
8
Blobworld
Example 1 Blobworld http//elib.cs.berkeley.edu/
photos/blobworld/start.html
BLOB Binary Large Object, an indistinct
shapeless (really ?) form
9
Blobworld
Blobworld Query by Shape / Texture / Location /
Color
Selected Blob Query by Color and Texture of
Blob Result Blobs with similar Color and
Texture Satisfying ?
10
Blobworld
Blobworld Query by Shape / Texture / Location /
Color
Selected Blob Query by Shape of
Blob Result are these shapes similar
? Satisfying ?
11
Blobworld
Result SHAPE recognition seems to be necessary
but not easy !
12
ISS Database
Example 2 ISS-Database http//knight.cis.temple.
edu/shape
13
The Interface (JAVA Applet)
14
The Sketchpad Query by Shape
15
The First Guess Different Shape - Classes
16
Selected shape defines query by shape class
17
Result
18
ISS Database
ISS Query by Shape / Texture
Sketch of Shape Query by Shape
only Result Satisfying ?
19
ISS Database
SHAPE recognition seems to be possible and
leads to satisfying results !
20
ISS Database
The ISS-Database will be topic of part IV of
this tutorial so stay alert !
21
Overview

• Overview Part 1
• Why shape ?
• What is shape ?
• Shape similarity
• Metrices
• Classes of similarity measures
• Feature Based Coding
• Examples for global similarity

22
Why Shape ?
Why Shape ?

• Shape is probably the most important property
  that is perceived about objects. It allows to
  predict more facts about an object than other
  features, e.g. color (Palmer 1999)
• Thus, recognizing shape is crucial for object
  recognition. In some applications it may be the
  only feature present, e.g. logo recognition

23
Why Shape ?

• Shape is not only perceived by visual means
• tactical sensors can also provide shape
  information that are processed in a similar way.
• robots range sensor provide shape information,
  too.

24
Shape

• Typical problems
• How to describe shape ?
• What is the matching transformation?
• No one-to-one correspondence
• Occlusion
• Noise

25
Shape

• Partial match only part of query appears in part
  of database shape

26
What is Shape ?

• What is Shape ?
• Plato, "Meno", 380 BC
• "figure is the only existing thing that is found
  always following color
• "figure is limit of solid"

27
What is Shape ?
lets start with some properties easier to
agree on Shape describes a spatial
region Shape is a (the ?) specific part of
spatial cognition Typically addresses 2D
space why ?
28
What is Shape ?
3D gt 2D projection
29
What is Shape ?
the original 3D (?) object
30
What is Shape ?
Moving on from the naive understanding, some
questions arise Is there a maximum size for a
shape to be a shape? Can a shape have holes?
Does shape always describe a connected region?
How to deal with/represent partial shapes
(occlusion / partial match) ?
31
What is Shape ?
Shape or Not ? Continuous transformation
from shape to no shape Is there a point when it
stops being a shape?
32
What is Shape ?
Shape or Not ? Continuous transformation
from shape to two shapes Is there a point when
it stops being a single shape?
33
What is Shape ?
But theres no doubt that a single, connected
region is a shape. Right ?
34
What is Shape ?
A single, connected region. But a shape
? A question of scale !
35
What is Shape ?

• Theres no easy, single definition of shape
• In difference to geometry, arbitrary shape is not
  covered by an axiomatic system
• Different applications in object recognition
  focus on different shape related features
• Special shapes can be handled
• Typically, applications in object recognition
  employ a similarity measure to determine a
  plausibility that two shapes correspond to each
  other

36
Similarity
So the new question is What is Shape
Similarity ? or How to Define a Similarity
Measure
37
Similarity
Again its not so simple (sorry). Theres
nothing like THE similarity measure
38
Similarity
which similarity measure, depends on which
required properties, depends on which
particular matching problem, depends on which
application
39
Similarity
which application
Simple Recognition (yes / no)
... robustness
Common Rating (best of ...)
Analytical Rating (best of, but...)
... invariance to basic transformations
40
Similarity
which problem computation problem d(A,B)
decision problem d(A,B) lte ? decision
problem is there g d(g(A),B) lte ?
optimization problem find g min d(g(A),B)
41
Similarity
which properties We concentrate here on the
computational problem d(A,B)
42
Similarity Measure

• Requirements to a similarity measure
• Should not incorporate context knowledge (no AI),
  thus computes generic shape similarity

43
Similarity Measure

• Requirements to a similarity measure
• Must be able to deal with noise
• Must be invariant with respect to basic
  transformations

Next Strategy
Scaling (or resolution)
Rotation
Rigid / non-rigid deformation
44
Similarity Measure

• Requirements to a similarity measure
• Must be able to deal with noise
• Must be invariant with respect to basic
  transformations
• Must be in accord with human perception

45
Similarity Measure
Some other aspects worth consideration
Similarity of structure Similarity of
area Can all these aspects be expressed by
a single number?
46
Similarity Measure

• Desired Properties of a Similarity Function C
• (Basri et al. 1998)
• C should be a metric
• C should be continous
• C should be invariant (to)

47
Properties
Metric Properties S set of patterns Metric d
S S R satisfying 1. Self-identity " xÎS,
d(x,x)0 2. Positivity " x ¹yÎS, d(x,y)gt0 3.
Symmetry " x, yÎS, d(x,y) d(y,x) 4. Triangle
inequality " x, y, zÎS, d(x,z)d(x,y)d(y,z)
Semi-metric 1, 2, 3 Pseudo-metric 1, 3, 4 S
with fixed metric d is called metric space
48
Properties

• Self-identity " xÎS, d(x,x)0
• Positivity " x ¹yÎS, d(x,y)gt0
• surely makes sense

49
Properties

50
Properties

51
Properties

• In general
• a similarity measure in accordance with human
  perception is NOT a metric. This leads to deep
  problems in further processing, e.g. clustering,
  since most of these algorithms need metric spaces
  !

52
Properties
Continuity usually useful, although
sometimes not in accordance with principles of
Gestalt properties, e.g. symmetry, collinearity.

53
Properties

54
Properties

55
Properties

• Some more properties
• One major difference should cause a greater
  dissimilarity than some minor ones.
• S must not diverge for curves that are not
  smooth (e.g. polygons).
• However, these demands are contradictory (proof
  is left as an exercise)

56
Similarity Measures

• Classes of Similarity Measures
• Similarity Measure depends on
• Shape Representation
• Boundary
• Area (discrete point set)
• Structural (e.g. Skeleton)
• Comparison Model
• feature vector
• direct

57
Similarity Measures
direct feature based
Boundary Spring model, Cum. Angular Function, Chaincode, Arc Decomposition (ASR-Algorithm) Central Dist. Fourier Distance histogram
Area (point set) Hausdorff Moments Zernike Moments
Structure Skeleton ---

58
Feature Based Coding
Feature Based Coding This category defines all
approaches that determine a feature-vector for a
given shape. Two operations need to be defined
a mapping of shape into the feature space and a
similarity of feature vectors.

Representation
Feature Extraction
Vector Comparison
59
Vector Comparison
Vector Comparison

,
60
Vector Comparison

61
Vector Comparison

62
Vector Comparison

63
Vector Comparison

• More Vector Distances
• Quadratic Form Distance
• Earth Movers Distance
• Proportional Transportation Distance

64
Vector Comparison

• Histogram Comparison
• Vector Comparison
• Histogram Intersection

65
Vector Comparison

66
Feature Based Coding
Again Two operations need to be defined a
mapping of shape into the feature space and a
similarity of feature vectors. We hence have TWO
TIMES an information reduction of the basic
representation, which by itself is already a
mapping of the reality.

Representation
Feature Extraction
Vector Comparison
67
Vector Comparison

• Example 1
• Elementary Descriptors
• Shape A,B given as
• Area (continous) or
• Point Sets (discrete)
• (Elementary Descriptors are 1dimensional feature
  vectors)

68
Vector Comparison

69
Vector Comparison
Boundary Box (area) These shapes are equal

70
Vector Comparison
Boundary Box (area) these shapes differ

71
Vector Comparison

• Example 2
• (Discrete) Moments
• Shape A,B given as
• Area (continous) or
• Point Sets (discrete)

72
Moments
Discrete Point Sets

73
Moments

74
Moments

75
Moments

76
Discrete Moments
Exercise Please compute all 7 moments for the
following shapes, compare the vectors using
different comparison techniques ?

77
Discrete Moments
Result each shape is transformed to a
7-dimensional vector. To compare the shapes,
compare the vectors (how ?).

78
Vector Comparison
Example 3 Central Distance Fourier Shape A,B
given as Contour (Boundary)

79
Vector Comparison
Contour is given as list of euclidean
coordinates 0,0 1,0 2,0 2,1 2,2 3,3 4,3
5,2

0
1
2
3
4
7
...
5
6
...
80
Central Distance Fourier
(MATLAB DEMO)

81
3D Distance Histogram
Example 4 3D Distance Histogram Shape A,B given
as 3D point set

82
3D Distance Histogram
(MATLAB DEMO)

83
Vector Comparison

• All Feature Vector approaches have similar
  properties
• Provide a compact representation
• this is especially interesting for database
  indexing !
• Works for any shape
• Requires complete shapes (global comparison)
• Sensible to noise (except Zernike moments which
  are computationally demanding)
• Map dissimilar shapes to similar feature
  vectors (!)
• They can be used as a prefilter for database
  applications !
• Make the choice of a similarity function
  difficult

84
Direct Comparison

End of Feature Based Coding ! Next Direct
Comparison
85
Vector Comparison
Direct Comparison
Example 1 Hausdorff Distance Shape A,B given as
point sets Aa1,a2, Bb1,b2,


86
Vector Comparison
Feature Based Coding


87
Vector Comparison
Hausdorff Distance


88
Vector Comparison
Boundary Representation
Hausdorff Unstable with respect to noise (This
is easy to fix ! How ?) Problem Invariance
! Nevertheless Hausdorff is the motor behind
many applications in specific fields (e.g.
character recognition)


89
Vector Comparison
Boundary Representation
Example 2 Chaincode Comparison Shape A,B given as
chaincode


90
Vector Comparison
Boundary Representation


91
Vector Comparison
Boundary Representation
A binary image can be converted into a chain
code representing the boundary. The boundary is
traversed and a string representing the curvature
is constructed.


1
2
3
0
4
C
5
6
7
5,6,6,3,3,4,3,2,3,4,5,3,
92
Vector Comparison
Boundary Representation
For curvature classes, a similarity can be
defined


93
Vector Comparison
Boundary Representation
To extend this measure to strings, two steps are
carried out. 1. Extend the measure to character
against string, for example by summing up
individual similarity measures. 2. Employing a
matching to compute a correspondence of
sub-strings. Hereby, the matching constitutes
from 1-to-1, 1-to-many, and many-to-1-matchings.
It is computed as string matching by means of
dynamic programming.


94
Vector Comparison
Boundary Representation
Digital curves suffer from effects caused by
digitalization, e.g. rotation


95
Vector Comparison
Boundary Representation
Compare chaincodes by string matching As
string-matching is not able to model a matching
of digital curves adequately, more sophisticated
matching algorithms are employed in real
applications using chain codes Weighted
Levensthein Distance Defines an edit distance for
transforming one string into another. Costs are
defined for altering, deleting, or inserting a
character. Extended Distance Formal translation
system with costs assigned to individual
production rules.


96
Vector Comparison
Structural Representation
Example 3 Skeletons Shape A,B primarily given as
area or boundary, structure is derived from
representation


97
Vector Comparison
Structural Representation
Structural approaches capture the structure of a
shape, typically by rep-resenting shape as a
graph. Typical example skeletons


98
Vector Comparison
Structural Representation
The computation can be described as a medial axis
transform, a kind of discrete generalized voronoi.


99
Vector Comparison
Structural Representation
The graph is constructed mirroring the adjacency
of the skeletons parts. Edges are labeled
according to the qualitative classes. Matching
two shapes requires matching two usually
different graphs against each other.


100
Vector Comparison
Structural Representation
Problems of skeletons - Pruning


101
Vector Comparison
Structural Representation

• Robustness
• (MATLAB Demo)

102
Vector Comparison
Shape similarity
All similarity measures shown can not deal with
occlusions or partial matching (except skeletons
?) ! They are useful (and used) for specific
applications, but are not sufficient to deal with
arbitrary shapes Solution Part based
similarity !