Symmetrybased Segmentation and Recognition

1
Symmetry-based Segmentation and Recognition
Thomas B. Sebastian
Brown University
2
Summary

• Three main contributions of my dissertation
• Effective segmentation method for carpal bones
  in CT images
• Implements skeletal coupling between seeds
• Solves convergence problem of deformable models
• Generic curve matching algorithm with several
  applications
• Robust shape recognition algorithm for matching
  shock graphs
• Gives excellent recognition rates for indexing
  large shape databases

3
Contents

• Segmentation using skeletally coupled deformable
  model
• Introduction
• Previous approaches
• Skeletal coupling
• Results
• Curve matching and its applications
• Shape recognition using shock graphs
• Indexing into large shape databases

4
Medical application

• Study 3D kinematics of carpal bones
• Identify wrist injuries where radiographs are
  normal
• Quantify shape of bone to characterize disease
  progression, e.g., in Kienbock disease
• Compute curvature maps

5
Challenges

• Weak and diffused edges

• Gaps in bone boundary

6
Challenges (cont.)

• Texture in spongy bone

• Narrow inter-bone gaps

7
Contents

• Segmentation using skeletally coupled deformable
  model
• Introduction
• Previous approaches
• Skeletal coupling
• Results
• Curve matching and its application
• Shape recognition using shock graphs
• Indexing into large shape databases

8
Deformable models
Initialize multiple seeds, which grow under
image-dependent forces

• Drawbacks
• Fail to converge near weak edges
• Do not capture narrow inter-bone gaps

9
Seeded region growing Biscoff PAMI

• Initializes seeds and grows them by annexing an
  adjacent pixel
• Only the "closest" pixel is added at each
  iteration
• Implements global competition among all seeds

• Drawbacks
• Leaks into small gaps as there are no geometric
  constraints

10
Region Competition Zhu/Yuille, PAMI

• Initializes seeds and grows them using a
  combination of statistical and smoothing forces

• Implements local competition between seeds when
  they contact each other
• Allows for exchange of pixels between regions and
  recovery from errors

11
Region Competition (cont.)

• Drawbacks
• Merges some adjacent bones
• Fails to capture low-contrast bones

12
Contents

• Segmentation using skeletally coupled deformable
  model
• Introduction
• Previous approaches
• Skeletal coupling
• Results
• Curve matching and its application
• Shape recognition using shock graphs
• Indexing into large shape databases

13
Skeletal coupled deformable model (SCDM)

• SCDM combines advantages of previous techniques
• Subpixel nature of deformable models
• Global competition of seeded region growing
• Local competition of region competition

14
Skeletal coupling Overview
15
Growth of regions

• In isolation, growth of regions depends on a
  local statistical force

16
Local competition by skeletal coupling

• The inter-region skeleton is used to couple
  growing regions
• The regions compete for pixels in the middle

• When seeds come in contact, l1, same as region
  competition

17
Long-range skeletal competition

• The inter-region skeleton is viewed as predicted
  boundary and the growth of regions is modulated
  by its suitability

18
Total skeletally-coupled forces

• The total statistical force is a combination of
  local and long-range forces

19
Contents

• Segmentation using skeletally coupled deformable
  model
• Introduction
• Previous approaches
• Skeletal coupling
• Results
• Curve matching and its application
• Shape recognition using shock graphs
• Indexing into large shape databases

20
Convergence of SCDM

• SCDM solves the convergence problem of
  traditional deformable models

21
SCDM segmentation results
Results are clinically meaningful
22
3D model of carpal bones
3D model is created by stacking up 2D contours
3D model is useful in studying carpal kinematics,
creating computational atlases, etc.
23
Carpal bone segmentation Comparison
SRG/RegComp
SCDM

• Rating by hand surgeons (RIH)
• SCDM 83
• SRG 14
• RegComp 3

24
Contents

• Segmentation using skeletally coupled deformable
  model
• Introduction
• Previous approaches
• Skeletal coupling
• Results
• Curve matching and its applications
• Shape recognition using shock graphs
• Indexing into large shape databases

25
Curve matching Sebastian et al, PAMI

• Goal is to find optimal alignment (pairing of
  points) and distance (deformation cost)

• Cost is measured by length and curvature
  differences of infinitesimal segments and summing
  them

26
Curve matching Sebastian et al, PAMI

• Alignment is a pairing of points of C, C
• Treat curves C and C as the x and y-axes
• Alignment is then a curve in 2D space, called
  alignment curve
• Dynamic programming to find optimal alignment
  curve

27
Prototype formation

• Average curve can be computed by averaging
  corresponding sub-segments

28
Shape morphing

• Weighted averages can be used to generate morph
  sequence

29
Handwritten character recognition

• 327 characters (34 categories)
• 98.5 recognition rate

30
Gesture recognition
Intuitive matches
31
Gesture recognition (cont.)
Outperforms other methods
Precision
Recall
32
Contents

• Segmentation using skeletally coupled deformable
  model
• Introduction
• Previous approaches
• Skeletal coupling
• Results
• Curve matching and its applications
• Shape recognition using shock graphs
• Indexing into large shape databases

33
Shock graph representation of shapes

• Shocks (or medial axis or skeleton) are locus of
  centers of maximal circles that are bitangent to
  shape boundary

Shape boundary
Shocks
34
Distance between shapes

• Shape space is collection of all shapes
• Shape is a point
• Shape deformation sequence is a path

• Cost of optimal deformation sequence is distance
  from A to B

• There are infinitely many deformation paths
• Shape space has to be discretized

35
Changes in shock graph topology

• Shock graph topology is unaltered for most shape
  deformations

• At transition shapes small changes lead to abrupt
  changes in the shock graph topology

36
Partitioning of shape space

• Shape cell Collection of shapes with same shock
  graph topology

• Transition shapes form the boundary between shape
  cells

Shape cell 2
Shape cell 1
37
Discretization of deformation paths

• Shape deformation bundle Collection of
  deformation paths passing through the same set of
  shape cells

• Represented by set of transition shapes it passes
  through

Dynamic programming is used to find optimal edit
sequence
38
Matching results Sebastian et al, ICCV 01
Edit-distance algorithm gives intuitive results
Same colors indicate matching edges
gray-colored edges are pruned
39
Robustness to transformations
Boundary noise
In optimal edit sequence noisy branches are
pruned
Articulation
Edit-distance is robust in presence of part-based
changes
40
Robustness to transformations (cont.)
Viewpoint variation

• Deform edit handles smooth changes
• Splice and contract edits handle abrupt changes

41
Robustness to transformations (cont.)
Partial occlusion
Edit-distance is robust to partial occlusion
42
Indexing into shape databases
Edit-distance algorithm allows 100 shape
recognition between different shape categories
Results duplicated in two databases 99 shapes
and 216 shapes
43
Indexing Results
It also allows nearly 100 recognition between
shapes in the same shape category
44
Contents

• Segmentation using skeletally coupled deformable
  model
• Introduction
• Previous approaches
• Skeletal coupling
• Results
• Curve matching and its applications
• Shape recognition using shock graphs
• Indexing into large shape databases

45
Indexing into large databases ECCV 02

• Indexing results are excellent using a large
  database (1032 shapes, 200 exemplars)
• Correct category is selected in top 1, 2 and 5
  matches with 78, 91, and 99 success rate
  respectively

46
Number of exemplars

• Using fewer exemplars suggests a hierarchical
  representation
• 2-3 primary exemplars rule out 75 of categories
• Additional 2-3 auxiliary exemplars rule out
  another 50 of categories

47
Acknowledgments

• Collaborators/Advisors
• Prof. Benjamin Kimia, Brown University
• Prof. Philip Klein, Brown University
• Dr. Joseph Crisco, RI Hospital

48
Selected References

• SCDM segmentation
• Segmentation of carpal bones from CT images using
  SCDM MedIA02
• Segmentation of carpal bones using SCDM
  MICCAI98
• Curve matching
• On aligning curves PAMI02
• Alignment-based recognition of shape outlines
  IWVF01
• Constructing curve atlases MMBIA00
• Shock-graph matching
• Recognition of shapes by editing their shock
  graphs ICCV01
• Shape matching using edit distance An
  implementation SODA01
• Indexing into databases
• Shock-based indexing into large databases
  ECCV02
• Metric-based shape retrieval in large databases
  ICPR02
• Curves vs. skeletons for object recognition
  ICIP01