Character Animation from 2D Pictures and 3D Motion Data

1
Character Animation from 2D Pictures and 3D
Motion Data

• Alexander Hornung, Ellen Dekkers, and Leif
  Kobbelt
• RWTH-Aachen University
• ACM Transactions on Graphics 2009

1
2
Outline

• Introduction
• Flow Chart
• Joint selection
• Camera and model pose determination
• Initial model-fitting
• Animation
• Results
• Conclusion

2
3
Introduction(1/2)

• Reconstruction of a textured 3D model, and to
  animate this model
• Hard to preserve the
• photorealism or style
• of the original input image.

3
4
Introduction(2/2)

• generate animations of photographed or painted 2D
  characters based on 3D motion data
• Correctly handle projective distortion effieft

5
Introduction(1/2)
Joint Selection
Camera and model pose determination
Initial Model Fitting (ARAP)
Animation
ASAP for motion sequence
5
6
Joint selection

• The user manually select joint positions in the
  input image.
• Automatic procedure
• Difficult to estimate
• occlusion
• Diverse dataset of
• example poses
• Non human characters

6
7
Camera and model pose determination(1/2)

• Compute camera projection model which describes
  the mapping from 3D joint to 2D image space
• Find an optimal 3D pose X0 in our motion sequence
• doesnt
  correspond to a geometrically plausible camera
  model

7
8
Camera and model pose determination(2/2)

• Constrain
• To the Projection matrix
• Use iterative LM to minimize the projection error

9
Initial Modeling Fitting(1/6)

• A set of generic shape templates T(2D
  nonmanifold triangle mesh )
• A template T consists of a triangulated set of
  vertices representing the shape boundary and
  skeleton joints
• Selected by the extrinsic camera data R and C in
  relation to the best-matching model pose Xo

9
10
Initial Modeling Fitting(2/6)
10
11
Initial Modeling Fitting(3/6)

• As Rigid As Possible (ASAP)
  Igarashi et al.2005
• T gt I gt F gt D

11
12
Initial Modeling Fitting(4/6)

• T gtILaplace-based deformation (Sorkine et al.
  2004)
• I gtFUsing only rotation and
• translation in least square sense
• G gt D Computing the average
• of corresponding vertex positions

13
Initial Modeling Fitting(5/6)

• The shape template for a given image is first
  fitted to the user-selected joint positions using
  ARAP
• T gt I gt F gt T
• Boundaries of D are snapped to the characters
  silhouette using a 2D-snakes approach Kass et
  al. 1987
• Image completion poisson-matting

14
Initial Modeling Fitting(6/6)

• T gt I gt F gt T0
• Transforming the characters image pose, into
  the best-matching 3D pose using the projected
  joint positions as vertex
  constrain

15
Animation(1/3)

• Perspective scaling and distortion effects are
  completely ignored.

15
16
Animation(2/3)

• ASAP(as similar as possible )
• T gt I gt F gt D
• Inflate the shape template T
• Using ARAP policy

16
17
ASAP

• One bones local coordinate

18
ASAP

• The 2D vertices v of a triangle are unprojected
  to 3D point V0(spanned by Ix and Iy)
• Adding a offset to V0

19
ASAP

• transform to local coordinate
• Transform to other frame of the motion data
• vPVt

20
Animation(3/3)

• Simply exchange the 3D motion
• Boundary snapping and texture generation do not
  have to be recomputed.
• Apply a different motion to a character is
  generally a matter of seconds

20
21
Results(1/3)
21
22
Results(2/3)
22
23
Results(3/3)

• Slight modification to Meerkat since it has
  significantly shorter legs.
• Total 15 to 30 minutes for the skeleton joint
  selection and the boundary snapping
• Generally do not take much longer than one hour
  to prepare (including image completion and
  matting results).

23
24
Conclusion(1/2)

• Present a complete, easy-to-use system for
  animating 2D images of arbitrary characters with
  3D motion.
• introduced an as-similar-as-possible shape
  deformation algorithm to deform these shapes
• One restriction of our technique is changing its
  moving direction, or where it turns its head

24
25
Conclusion(2/2)

• Future Work
• Automatic pose estimation in a relatively
• common pose, such as standing or walking.
• smooth transitions from the user-selected pose
  to the best-matching pose of the 3D motion
• integrate global effects such as shadows or
  reflections to improve the visual appearance of
  some scenes.
• Animate animals, or even plants.

25