GeometryDriven Photorealistic Facial Expression Synthesis

1
Geometry-Driven Photorealistic Facial Expression
Synthesis

• Qingshan Zhang, Zicheng Liu, Baining Guo, Demetri
  Terzopoulos, and Heung-Yeung Shum
• IEEE Transactions on Visualization and Computer
  Graphics
• VOL. 12, NO. 1, JANUARY/FEBRUARY 2006

2
Outline

• Introduction
• Related work
• Geometry-Driven Expression Synthesis
• Motion Propagation
• Results
• Conclusion

3
Introduction

• Expression mapping
• Shortcoming not produce expression details such
  as wrinkles caused by skin deformations.
• Geometry-driven facial expression synthesis
  system

Feature point position
Expression Synthesis
Example
4
Introduction

• Expression mapping
• Shortcoming not produce expression details such
  as wrinkles caused by skin deformations.
• Geometry-driven facial expression synthesis
  system
• Expression editing

5
Related Work

• Morph-based approach
• Pighin et al. Synthesizing Realistic Facial
  Expressions from Photographs,1998
• Expression mapping
• maps one persons expression to another
  person by transferring the convex combination
  coefficients.
• Expression editing
• provides a set of easy-to-use tools and
  interfaces to allow a user to design facial
  expressions interactively.

6
Geometry-Driven Expression Synthesis

• Pighin et al.1998
• Generate photorealistic facial expressions
  through a convex combination of example
  expressions.
• Ei (Gi, Ii), i0, . . .,m

7
System Overview

8
System Overview

9
System Overview

10
System Overview

11
Subregion Expression Synthesis

• Given GR, we want to project it into the convex
  hull of GR0,...,GRm.
• Subregion image IR

12
Blending along Subregion Boundaries

• Weight map
• R blending weight (r)
• G,B index of neighboring
• subregions (i1, i2)
• Pixels blended intensity

13
Inferring Feature Point Motions from a Subset

• Expression editing
• Each time a user moves a feature point, we need
  to determine the most likely movement for the
  remaining feature points.
• Divide the face feature points into hierarchies
• Perform hierarchical principal components
  analysis (PCA) on the example expressions

14
Motion Propagation
Displacement vector (d1, 0)
15
Motion Propagation
Displacement vector (d1, 0)
Project to PC
Displacement vector (d1, d2)
16
Motion Propagation
Displacement vector (d1, d2, 0, 0,)
Displacement vector (d1, d2)
17
Motion Compensation

• Vertex motions include skin deformations and
  global head motion
• 2D similarity transformation T( a, b, tx, ty)
• Minimize
• Displacement vector

18
Results

• Enhanced Expression Mapping
• Expression Editing

19
Limitations

• Lack of extrapolation
• Artifacts due to image blending and pixel
  misalignment make the animation look unnatural
• Does not handle out of-plane head rotations
  because use a 2D motion model

20
Conclusion

• Geometry-driven facial expression synthesis
  system
• Synthesizing facial expression textures from
  facial feature point motions
• Motion propagation algorithm
• Infer facial feature point motions from a small
  subset of tracked points