Automatic Registration for Articulated Shapes

1
Automatic Registration for Articulated Shapes

• July 3, 2008
• Eurographics Symposium on Geometry Processing
• Will Chang and Matthias Zwicker
• Department of Computer Science and Engineering
• University of California, San Diego

2
Our Goal

• Align partial articulated shapes
• Significant movement and missing data
• Do not require markers or a template
• Articulated motion assumption
• Movement of the object composed of rigid parts
• Reasonable for many objects (humans, animals,
  etc.)

3
Previous Work

• Require small changes in pose / temporal
  coherence
• ICP-based approaches (Pekelny Gotsman, EG 2008)
• Modeling a space-time evolving surface (Mitra et
  al., SGP 2007)
• Tracking time-varying points (Wand et al., SGP
  2007)
• Require user-placed feature markers
• Cross-parameterization (Kraevoy and Sheffer,
  Siggraph 2004)
• Registration of body scans (Allen et al.,
  Siggraph 2003)
• 3D scan completion (Pauly et al., SGP 2005)
• Require knowledge of the entire shape (a
  template)
• Correlated Correspondence (Anguelov et al., NIPS
  2004)
• Our approach removes these requirements

4
Problem Formulation
4
5
Problem Formulation

• Align a pair of articulated shapes
• Consider the problem as determining how each
  point moved
• Goal solve for the transformation at each point
• that aligns the shapes (continuous field of
  transformations)

Transformations ?
Source Shape
Target Shape
6
Problem Formulation

• Formulate a preference for the correct
  transformations
• Data cost measures alignment of the shape
• Smoothness cost preserves overall structure of
  the shape
• Optimize transformations over all points in shape
• Continuous optimization problem
• Usually a good initialization is needed to solve
  this well

argmin
Data Cost
Smoothness Cost
Continuous field of transformations
7
Articulated Motion Assumption

• Articulated motion ? small set of transformations
• Predetermine a set of transformations describing
  the motion
• Assign the transformations to the points

8
Our Approach

• Optimize an assignment from a finite set of
  transformations
• A discrete labelling problem ? Graph Cuts for
  optimization

argmin
Data Cost
Smoothness Cost
Continuous field of transformations
Assignment from a set of transformations
Transformations from finite set
Source Shape
Target Shape
9
Algorithm Description
Motion Sampling
Global Motion Optimization
Motion Sampling
10
Motion Sampling

• Idea from Partial Symmetry Detection (Mitra et
  al. 06)
• Similar to Hough Transform
• Intuition
• Rigid parts move according to a single
  transformation
• Movement of all points on a single rigid part are
  the same
• Sample the motion by computing transformations
  between many pairs of points between the shapes
• Computed transformations will naturally form
  clusters corresponding to the movement of rigid
  parts
• Many false positives ? need matching pruning

11
Motion Sampling Illustration

• Find transformations that move parts of the
  source to parts of the target

Source Shape
Target Shape
12
Motion Sampling Illustration

• Find transformations that move parts of the
  source to parts of the target

Source Shape
Target Shape
13
Motion Sampling Illustration

• Find transformations that move parts of the
  source to parts of the target

14
Motion Sampling Illustration

• Find transformations that move parts of the
  source to parts of the target

Rotations
Translations
Source Shape
Target Shape
Transformation Space
15
Motion Sampling Illustration

• Find transformations that move parts of the
  source to parts of the target

s1?t1
s1?t2
Rotations
s1
t1
Translations
t2
s2
s2?t1
s2?t2
Source Shape
Target Shape
Transformation Space
16
Motion Sampling Pipeline

• Pre-compute local frames
• Feature-based matching
• Reduces the number of point pairs
• Compute and store rigid transformations
• Cluster rigid transformations
• Prune insignificant transformations (optional)

17
Geometric Features

• Identifies geometrically similar local
  neighborhoods
• Spin Images (Johnson and Herbert 1998)
• Histogram of points on the shape
• Bins concentric circular rings stacked along
  normal

18
Clustering

• Variant of mean-shift clustering (Tuzel et al.
  05)
• How to measure distance in transformation space?
• in Lie group (SE3) rotation
  translation
• Measure intrinsic distance between identity and
• Approximated distance metric
• Use corresponding elements Lie
  algebra (se3)
• 6-dimensional vector (axis-angle rotation
  linear velocity)

18
19
Matching Pruning

• Optional step to eliminate false positives
• Incrementally grow a matching region between
  shapes
• Keep the best k transformations with the largest
  regions

20
Limitations of Motion Sampling

• Final Output finite set of rigid transformations
• If there are multiple similar parts
• Does not figure out the correct part
• Disambiguate in the optimization step

Candidate Transformations
Source with Selected Region
Visualized Transformations
21
Algorithm Description
Motion Sampling
Global Motion Optimization
22
Global Motion Optimization

• Optimize an assignment from a finite set of
  transformations
• A discrete labelling problem ? Graph Cuts for
  optimization

argmin
Data Cost
Smoothness Cost
Assignment from a set of transformations
Transformations from finite set
Source Shape
Target Shape
22
23
Data Term

• Move all points as close as possible to the
  target
• How to measure distance to target?
• Apply selected transformation for all
  
• Measure distance to closest point in target

24
Data Term

• Move all points as close as possible to the
  target
• Choose suitable cost based on missing data, noise
• Can precompute closest points

24
25
Smoothness Term

• Preserve edge length between neighboring points
• Disambiguates multiple possible mappings

Original Length
Transformed Length
26
Symmetric Cost Function

• Swapping source / target can give different
  results
• Optimize assignment in both meshes (forward
  backward)
• Enforce consistent assignment penalty when

27
Final Cost Function

• Data and smoothness terms apply to both shapes
• Additional symmetric consistency term
• Weights to control relative influence of each
  term

Data
Smoothness
argmin


Source
Source
Assignment from a set of transformations
Data
Smoothness


Target
Target
Symmetric Consistency
Source Target
28
Optimization Using Graph Cuts

• Graph Setup
• Nodes points in source and target meshes
• Edges neighboring points in source and target
  meshes
• edges connecting closest points under each
  transform

29
RESULTS
30
Horse Dataset Results
12 poses of galloping horse total of 66 pairs,
correct leg matched in 64 pairs
31
Synthetic Dataset Example
Source
Target
1.5
0
Distance (from Target) to the closest point (
bounding box diagonal)
Aligned Result
32
Synthetic Dataset Example
Result w/ 50 Simplified Meshes
32
33
Synthetic Dataset w/ Holes
Source
Target
5.3
0
Aligned Result
Distance (from Target) to the closest point (
bounding box diagonal)
34
Arm Dataset Results
12 poses of arm scans total of 66 pairs, arm
hand orientation matched in all pairs
34
35
Arm Dataset Example
Missing Data
Missing Data
Source
Noisy Target
36
Arm Dataset Example
Distance (from Target) to the closest point (
bounding box diagonal)
Aligned Result
Motion Segmentation
37
Hand Dataset Example
Missing Data
Source
Target
38
Hand Dataset Example
Distance (from Target) to the closest point (
bounding box diagonal)
Aligned Result
Motion Segmentation
39
Performance

• Graph cuts optimization is most time-consuming
  step
• Symmetric optimization doubles variable count
• Symmetric consistency term introduces many edges

40
Limitations

• Errors in registration
• Trade-off between data and smoothness costs
• Data weight too high ? May break smoothness
• Smoothness weight too high ? Prefer bad alignment

Source
Target
Registration
41
Limitations

• Errors in registration
• Motion Sampling too much missing data
• ? Less accurate transformations
• ? May miss important transformations
• Hard assignment of transformations
• ? It breaks up shapes

Source
Target
Registration
42
Conclusions

• Automatic method for registering articulated
  shapes
• No template, markers, or manual segmentation
  needed
• Explicitly sample a discrete set of motion
• Optimize the assignment of transformations
• Graph cut result gives intuitive segmentation
• Additional opportunities for research
• Improving optimization / setting parameters
• Improve speed (multi-resolution optimization)
• Improve accuracy (update transformations during
  optimization)
• Automatically build a simplified kinematic model

43
Thank you for listening!
43
44
Quantitative Verification

• Numerically measure registration quality using
  the Symmetric Hausdorff Distance
• Hausdorff distance for each vertex, find the
  distance to the closest point in the other shape.
  Finally, take the maximum over all vertices.
• Symmetric take the maximum over both source and
  target
• Each dataset has a different scale
• Expressed as a fraction of the length of the
  bounding box diagonal
• The bounding box is different for each example
• Then comparison is different for each example
• So we picked a single bounding box diagonal
  length value for all examples in each dataset
• Theres also missing data
• Dont measure distance if the point or the
  corresponding closest point is on the boundary of
  the shape

45
Registration Error Analysis
1.5
0
46
Registration Error Analysis
5.3
0
47
Registration Error Analysis
48
Registration Error Analysis
49
Algorithm Overview

• Registration Finding a transformation at each
  point
• Articulated motion assumption
• Discrete set of transformations entire object
  movement
• Algorithm
• Sample the motion directly
• Extract significant transformations between parts
  of the object
• Optimize assignment of transformations at each
  point
• Disambiguate symmetric parts by preserving edge
  lengths

50
Problem Formulation

• Formulate a preference for the correct assignment
• Continuous optimization problem (difficult!)

Data Term (measures alignment)
Smoothness Term (measures consistency)
?
Edge in the set of edges
Source Shape
Target Shape
51
Previous Work

• Require small changes in pose / temporal
  coherence
• ICP-based approach of (Pekelny Gotsman, EG
  2008)
• User labels rigid parts in the first frame
• Each rigid part registered in subsequent frames

Reconstructed Surface and Skeleton
Registered 3D Scans
(Images from Pekelny and Gotsman 2008)
51
52
Previous Work

• Require small changes in pose / temporal
  coherence
• Model a space-time surface (Mitra et al., SGP
  2007)
• Requires dense spatial and temporal sampling

Example A 2D time-varying surface
53
Previous Work

• Require user-placed feature markers
• Example-based 3D scan completion (Pauly et al.,
  SGP 2005)
• Fill holes by warping similar shapes in a database

54
Previous Work

• Require knowledge of the entire shape (a
  template)
• Correlated Correspondence (Anguelov et al., NIPS
  2004)
• Goal is to match the corresponding point in the
  template
• Cost function matches features and preserves
  geodesic distance

Partial Example
Registered result
Ground Truth
Template Model
55
Final Cost Function

• Optimize both directions simultaneously
• Weights to control relative
  influence of each term

56
References for Images and Citations

• http//www.nextengine.com (also gallery section)
• http//www.geomagic.com/en/solutions/nasa.php
• http//www.geomagic.com/en/solutions/kavo.php
• http//www.sonypictures.com
• http//en.wikipedia.org/wiki/Motion_capture
• http//www.siggraph.org/publications/newsletter/vo
  lume/201cchanging-the-way-computers-see-humans-and
  -the-way-humans-see-motion-capture201d
• http//www.xyzrgb.com/
• http//dentalrestoration101.com/
• http//www.rapidform.com/Contents/Product/category
  _id/51
• http//whatisthematrix.warnerbros.com/vfx/rl_cmp/v
  fx_image_11.html
• Articulated Object Reconstruction and Markerless
  Motion Capture from Depth Video. Pekelny, Y. and
  Gotsman, C. Computer Graphics Forum 27(2) (Proc.
  Eurographics), 2008.
• Dynamic Geometry Registration. Niloy J. Mitra,
  Simon Floery, Maks Ovsjanikov, Natasha Gelfand,
  Leonidas Guibas, Helmut Pottmann. Symposium on
  Geometry Processing 2007.
• Partial and Approximate Symmetry Detection for 3D
  Geometry. Niloy J. Mitra, Leonidas Guibas, Mark
  Pauly. ACM SIGGRAPH 2006.
• Example-Based 3D Scan Completion. Mark Pauly,
  Niloy Mitra, Joachim Giesen, Markus Gross,
  Leonidas J. Guibas. Symposium on Geometry
  Processing 2005.
• The Correlated Correspondence Algorithm for
  Unsupervised Registration of Nonrigid Surfaces.
  D. Anguelov, P. Srinivasan, D. Koller, S. Thrun,
  H. Pang and J. Davis. Proceedings of the Neural
  Information Processing Systems (NIPS) Conference,
  2004.
• Efficient multiple model recognition in cluttered
  3-D scenes. A. Johnson and M. Hebert. Proceedings
  of the IEEE Conference on Computer Vision and
  Pattern Recognition (CVPR '98), June, 1998, pp.
  671 - 677.
• Fast Approximate Energy Minimization via Graph
  Cuts. Yuri Boykov, Olga Veksler, Ramin Zabih.
  PAMI 2001.
• Interactive Skeleton-Driven Dynamic Deformations.
  Steve Capell, Seth Green, Brian Curless Tom
  Duchamp, Zoran Popovic. Proceedings of ACM
  SIGGRAPH 2002.
• Articulated body deformation from range scan
  data. ALLEN, B., CURLESS, B., and POPOVIC, Z. ACM
  Transactions on Graphics (ACM SIGGRAPH 2002), 21,
  3, 612-619.