Recovery of Facial Motion

1
Recovery of Facial Motion

• Jingyu Yan
• Spring 2003, Comp239

2
Motivations

• A morphable model for animation
• C S C is the morphing coefficient and S is
  the shape basis
• Recovery of the shape basis and coefficients of
  face expression from video
• P R(C S) T (weak perspective or
  orthographic)
• Animate the 3D facial model recovered from the
  video

3
Examples

• Video

4
Challenges and Tasks

• Textureless
• Few features can be reliably tracked
• Hard to match corresponding points between frames
• Lighting changes as the surface moves
• Occlusion
• Detection and estimation of occlusion
• Noise
• Video data is noisy.

5
Approaches

• Estimate motion from optical flows
• Y XF (Y temporal image gradients X spacial
  image gradients F optical flows)
• Enforce the constraints imposed by the model
• P R(C S) T, rank(S)lt 3K
• Certainty warp of the video data
• ?F is X, so ?Y is X ?FXT.
• Transform the uncertainty from elliptical
  (Mahalanobis) errors to spherical(Frobenius)
  errors.

6
What is accomplished

• Motion estimation
• For large motion, this relation will not hold.
• Solution build a pyramid of images. The motion
  data from the coarser level provides estimates
  for the finer level.
• Constraints enforcement
• Observation rigid motion is more constrained
  than non-rigid motion.
• Automatic detection of the rigid and non-rigid of
  the motion can further enhance the estimate.

7
Some initial results

• A medium level in the image pyramids

Automatic detection of rigidity of motion
Frame 1
no detection
Frame 29
8
Further tasks

• Retrieve the shape basis and morphing
  coefficients from motion data
• Apply them back to 3D animation.

9
References

• M. Brand. Morphable 3D models from video.
• Trancking and Modelling non-rigid Objects with
  Rank Constraints Lorenzo Torresani, Danny Yang,
  Gene Alexander, Christoph BreglerProc. IEEE CVPR
  2001
• M. Irani. Multi-frame optical flow estimation
  using subspace constraints. In Proc. ICCV, 1999.
• C. Tomasi and T. Kanade. Shape and motion from
  image streams under orthography A factorization
  method. International Journal of Computer Vision,
  9(2)137154, 1992.