Capturing and Animating Skin Deformation in Human Motion

1
Capturing and Animating Skin Deformation in
Human Motion
Sang Il Park Jessica K. Hodgins Carnegie Mellon
University
2
Introduction

• Conventional motion capture records only the
  skeleton motion

Conventional motion capture
Our method
3
Introduction

• Use a conventional optical motion capture system

40-60 markers
4
Introduction

• Use a conventional optical motion capture system

5
Introduction

• Use a conventional optical motion capture system

6
Introduction

• Use a conventional optical motion capture system

7
Introduction

• Use a conventional optical motion capture system

Data collection and cleaning
8
Introduction

• Use a conventional optical motion capture system

Data collection and cleaning
Skin Animation
9
Background

• One weight enveloping (skinning) Lewis et al.
  2000, Kavan and Zara 2005, Hyun et al. 2005, Guo
  and Wong 2005
• Simple and fast
• Hard to preserve volume
• Pose-space interpolation Sloan et al. 2001,
  Allen et al. 2002, Anguelov et al. 2005
• Example-based approach
• Deformation is a function of pose (no dynamic
  effects)
• Anatomically based modeling Scheepers et al.
  1997, Wilhelms and Gelder 1997, Nedel and
  Thalmann 2000, Teran et al. 2005, Larboulette et
  al. 2005, Zordan et al. 2004
• Anatomical accuracy
• Hard to simulate a full body

10
Background

• One weight enveloping (skinning) Lewis et al.
  2000, Kavan and Zara 2005, Hyun et al. 2005, Guo
  and Wong 2005
• Simple and fast
• Hard to preserve volume
• Pose-space interpolation Sloan et al. 2001,
  Allen et al. 2002, Anguelov et al. 2005
• Example-based approach
• Deformation is a function of pose (no dynamic
  effects)
• Anatomically based modeling Scheepers et al.
  1997, Wilhelms and Gelder 1997, Nedel and
  Thalmann 2000, Teran et al. 2005, Larboulette et
  al. 2005, Zordan et al. 2004
• Anatomical accuracy
• Hard to simulate a full body

11
Background

• One weight enveloping (skinning) Lewis et al.
  2000, Kavan and Zara 2005, Hyun et al. 2005, Guo
  and Wong 2005
• Simple and fast
• Hard to preserve volume
• Pose-space interpolation Sloan et al. 2001,
  Allen et al. 2002, Anguelov et al. 2005
• Example-based approach
• Deformation is a function of pose (no dynamic
  effects)
• Anatomically based modeling Scheepers et al.
  1997, Wilhelms and Gelder 1997, Nedel and
  Thalmann 2000, Teran et al. 2005, Larboulette et
  al. 2005, Zordan et al. 2004
• Anatomical accuracy
• Hard to simulate a full body

12
Background

• One weight enveloping (skinning) Lewis et al.
  2000, Kavan and Zara 2005, Hyun et al. 2005, Guo
  and Wong 2005
• Simple and fast
• Hard to preserve volume
• Pose-space interpolation Sloan et al. 2001,
  Allen et al. 2002, Anguelov et al. 2005
• Example-based approach
• Deformation is a function of pose (no dynamic
  effects)
• Anatomically based modeling Scheepers et al.
  1997, Wilhelms and Gelder 1997, Nedel and
  Thalmann 2000, Teran et al. 2005, Larboulette et
  al. 2005, Zordan et al. 2004
• Anatomical accuracy
• Hard to simulate a full body

13
Overview
Data collection cleaning
Merging disconnected trajectories
Skin animation
Hole filling
Capture session
Resulting animation
Surface modelin rest pose
14
Overview
Data collection cleaning
Merging disconnected trajectories
Skin animation
Hole filling
Capture session
Resulting animation
Surface modelin rest pose
15
Capture session

• 350 markers (diameter 3.0 mm)
• 12 near-infrared Vicon cameras

16
Data collection and cleaning
17
Data collection and cleaning
3D trajectory
3D trajectory
3D trajectory
3D trajectory
18
Data collection and cleaning
3D trajectory
19
Data collection and cleaning

• Occlusions happen
• in a large region
• for a long time

20
Data collection and cleaning
21
Data collection and cleaning
22
Data collection and cleaning
23
Data collection and cleaning
Lipman et al. 2005
24
Data collection and cleaning
Merging disconnected trajectories
Hole filling
25
Data collection and cleaning
Merging disconnected trajectories
Hole filling
26
Reference pose
27
Merging

• Estimate position of missing marker
• Search the closest partial trajectory

28
Merging

• Estimate position of missing marker
• Search the closest partial trajectory

29
Merging

• Estimate position of missing marker
• Search the closest partial trajectory

Configuration at time t
Absolute orientation problem Horn 1987
30
Data collection and cleaning
Merging disconnected trajectories
Hole filling
31
Data collection and cleaning
Merging disconnected trajectories
Hole filling
PCA
32
Hole filling

• Estimate position of missing markers

Configuration at time t
Computing coefficients of basis of the PCA model
33
Overview
Data collection cleaning
Skin animation
Merging disconnected trajectories
Skin animation
Hole filling
Capture session
Resulting animation
Surface modelin rest pose
34
Skin animation

• Interpolating in the large non-linear deformation
  space

35
Approach

• Separating the displacement vector components
  into rigid deformation and its local deformation

Rigid segmentation
Non-linear deformation field
Resolving residuals
Markers
1

Radial basis function
36
Approach

• Separating the displacement vector components
  into rigid deformation and its local deformation

Rigid segmentation
Non-linear deformation field
Resolving residuals
Markers
1

Radial basis function
37
Approach

• Separating the displacement vector components
  into rigid deformation and its local deformation

Rigid segmentation
Non-linear deformation field
Resolving residuals
Markers
1

Radial basis function
38
Approach

• Separating the displacement vector components
  into rigid deformation and its local deformation

Rigid segmentation
Non-linear deformation field
Resolving residuals
Markers
1

Radial basis function
39
Rigid segmentation
Manual segmentation (once for subject)
40
Deformation field

• Quadratic transformation (3X9 matrix)

Linear
Pure quadratic
Mixed quadratic
Mueller et al. 2005
41
Deformation field
Linear
Pure quadratic
Mixed quadratic
42
Resolving residuals

• Small residuals left? Radial basis interpolation

43
Experimental results
44
Experimental results
45
Experimental results

• Accuracy of hole filling (Leave-one-out cross
  validation)

Region Abdomen Elbow Thigh Knee
Avg. Error 0.017 0.022 0.020 0.023
Max. Error 0.052 0.062 0.045 0.051
Error is normalized by the average distance to
the neighbors
46
Experimental results
47
Experimental results
48
Experimental results
49
Summary Future work

• Accurate capture of skin deformations for a
  variety of body types
• Subject specific
• Reuse of deformations for a particular subject or
  for a different subject
• Large number of markers
• Supplement markers with an anatomical model or
  with a statistical model

50
Thank you!

• Questions?