Vision based Animation The Inverse of an Inverse Problem

1
Vision based AnimationThe Inverse of an Inverse
Problem
Henning Biermann Chris Bregler Aaron
Hertzmann Lorie Loeb Kathy Pullen Danny Yang
2
The Standard Vision Pipeline
Recognition
Modeling
Capture
Image / Video
3
Vision for Graphics / Animation
Image / Video
Synthesis
Modeling
Capture
Image / Video
4
Vision for Graphics / Animation
Expressive Animations
Modeling
Capture
Image / Video
5
The Standard Analogy
6
The Graphics Problem
7
The Vision Problem
8
Vision Based Animation
9
Vision for Graphics
Image / Video
Synthesis
Modeling
Capture
Image / Video
10
Vision for Graphics
Image / Video
Synthesis
Richer Modeling
Capture
Image / Video
11
Vision for Animation
Image / Video
Synthesis
Kinematics Dynamics
Richer Modeling
Capture
Image / Video
12
Models 2 Issues

• Explicit / Implicit tradeoff

Hand Coded
Learned
13
Models 2 Issues

• Explicit / Implicit tradeoff

Hand Coded
Learned
M
X
Y
Fit Black-Box
Analytically derived
14
Models 2 Issues

• Explicit / Implicit tradeoff
• Model / Data tradeoff

Analytical
Learned
Pure Model
Pure Data
15
Models 2 Issues

• Explicit / Implicit tradeoff
• Model / Data tradeoff

Analytical
Learned
Pure Model
Pure Data
16
Case 1 Models of Kinematics
- Approximate Kinematic Chains. - Fine-tune
Kinematic Chains. - Acquisition of Nonrigid
Models.
17
Approximate Kinematic Chains
- Enough for Animation ?
18
Case 1 Models of Kinematics
- Approximate Kinematic Chains. - Fine-tune
Kinematic Chains. - Acquisition of Nonrigid
Models.
19
Fine-Tune Kinematics
Anna Rabinowicz
20
Fine-Tune Kinematics
Kathy Pullen
21
Case 1 Models of Kinematics
- Approximate Kinematic Chains. - Fine-tune
Kinematic Chains. - Acquisition of Nonrigid
Models.
22
Acquisition of Nonrigid 3D Kinematics

With Gene Alexander, Henning Biermann, Aaron
Hertzmann, Lorenzo Torresani, Danny
Yang
23
Solution for Non-rigid Tracking / Reconstruction
- Prior Model
2D PCA-based (Cootes et al 95) 3D
BlanzVetter-99
24
3D Model Acquisition
- Multi-view input Pighin et al 98
25
Solution for Rigid 3D Acquisition
Structure from Motion - Tomasi-Kanade-92
3D Pose 3D rigid Object
Factorization
26
Acquisition without prior model ?

• No Model available ?
• Model too generic/specific ?
• Stock-Footage only in 2D ?

27
Solution based on Factorization

• We want 3 things
• 3D non-rigid shape model
• for each frame
• 3D Pose
• non-rigid configuration (deformation)

• gt Tomasi-Kanade-92

Rank 3
W P S
28
Solution based on Factorization

• We want 3 things
• 3D non-rigid shape model
• for each frame
• 3D Pose
• non-rigid configuration (deformation)

• gt PCA-based representations

Rank K
W P non-rigid S
29
Factoring twice

30
Nonrigid 3D Kinematics from point tracks

31
Nonrigid 3D Kinematics from dense flow

32
Nonrigid 3D Kinematics from dense flow

33
Nonrigid 3D Kinematics from dense flow

34

Nonrigid 3D Kinematics from dense flow
Synthesis
Modeling
Motion Capture
35
Case 2 Models of Dynamics
Pure Model
Pure Data
36
Case 2 Models of Dynamics
Pure Model
Pure Data
Learning Controller
Mocap. Library
37
Case 2 Models of Dynamics
Pure Model
Pure Data
Parametric
Non-Parametric
38
Case 2 Models of Dynamics
Pure Model
Pure Data
Kernel-Based Densities
39
Case 2 Models of Dynamics
Kathy Pullen
40
Case 2 Models of Dynamics
Motion Texture
Sampling / Matching
41
Case 2 Models of Dynamics
Sampling / Matching
P(Li Lj)
42

Synthesizing Subtleties Variations
Synthesis
Modeling
Motion Capture
Kathy Pullen
43

Synthesizing Subtleties Variations
Synthesis
Modeling
Motion Capture
Kathy Pullen
44

Synthesizing Subtleties Variations
Synthesis
Modeling
Motion Capture
Kathy Pullen
45

Synthesizing Subtleties Variations
Synthesis
Modeling
Motion Capture
Kathy Pullen
46

Synthesizing Subtleties Variations
Synthesis
Modeling
Motion Capture
Kathy Pullen
47
Vision Based Animation
Animations
Kinematics Dynamics
Richer Modeling
48
Human Movement Research
Computer Science Vision / Graphics
Bio-Mechanics Motion Models/Analysis
Art Animation / Dance
Medicine/ Neurology Movement Disorders
49
Model
Data
50
Expressive Animation
Model
Data
51
Human Movement Research
Computer Science Vision / Graphics
Bio-Mechanics Motion Models/Analysis
Art Animation / Dance
Medicine/ Neurology Movement Disorders
52

Project Mariposa Animation Principles
Lorie Loeb
53

Project Mariposa Animation Principles
Lorie Loeb
Realism is out
54

Project Mariposa Animation Principles
Lorie Loeb
Realism is out Expressiveness is in
55
Expressive Animation
Model
Data
56
Controversy Motion-Capture based Animation
Model
Data
57
Controversy Motion-Capture based Animation
Model
Data
58
Controversy Motion-Capture based Animation
Model
Data
59
Controversy Motion-Capture based Animation
Model
Data
60
Synthesis by Analysis
Animations
61
Enough for Animation ?
Expressive Animations
Video/Mocap
62
Expressive Animation
Expressive Output
Cartoon Capture