A 3D Morphable Model for Correcting Facial Expressions

1
A 3D Morphable Model for Correcting Facial
Expressions

• Chris Boehnen
• Graduate Student University of Notre Dame
• Tanya Peters
• University of Washington
• Technical Advisor Trina Russ
• Security Technology Department, 4128

2
Biometrics

• Use of measurable physiological characteristics
  to authenticate a user and confirm identity
• Examples include fingerprint, retina, iris,
• hand geometry, vein patterns, voice,
• gait, and face

3
Motivation for Biometrics

• Access control to secure facilities or systems
• Identification of terrorists and criminals on
  watch lists is essential to National Security
• Several of the 9/11 highjackers
• were on terrorist watch lists
• Watch lists expected to grow
• in size to millions of
• people in the
• coming years

4
3D Facial Recognition

• 3D images of faces are used as a biometric to
  identify people
• 3D facial recognition has potential advantages
  over 2D facial recognition
• correcting for pose
• lighting variation
• facial expression
• accuracy

5
Problem Facial Expressions

• Facial expressions change the face enough to
  cause problems for 3D face recognition when
  attempting to compare two faces with different
  expressions

6
Our Solution

• Given a 3D scan of a face, we want to morph the
  scan from an unknown input expression to a known
  expression used during enrollment in the gallery
• If our morph is accurate enough, then by
  performing recognition between two faces with the
  same expression performance should improve

Used for comparison to gallery images for
recognition process
Morphed
Original Scan
Our Morphed Scan
Gallery
7
Physical vs. Principle Components Analysis
Approach

• Physically based approaches model the underlying
  facial structural
• This approach was designed for facial animation
  and can produce faces that look good
• Does not require a training set and should be
  able to correct any facial expression
• Requires use of a generic skull model
• Easy to produce expressions from neutral, but
  hard to go towards neutral

8
Physical vs. Principal Component Analysis
Approach

• PCA based approaches represent a face as a linear
  combination of an orthogonal face basis deduced
  from a face training set
• For good generalization, training set should have
  people representing every age/race/gender
  performing the type of expressions you want to be
  able to correct for
• Assumption two people who look similar when
  smiling will also look similar when frowning 1,
  which is not entirely accurate

1 Face identifications across different poses
and illuminations with a 3D Morphable model, V.
Blanz, S. Romdhani, and T. Vetter
9
Previous Work

• Vetter Blanz
• Linear Object Classes there exists a linear
  relationship between elements of the same class
  2
• Given a training set of 3D facial scans, we can
  represent a new face and alternate expressions
  using a linear combination of the PCA basis of
  the training set 1

1 Face identifications across different poses
and illuminations with a 3D Morphable model, V.
Blanz, S. Romdhani, and T. Vetter 2 Linear
Object Classes and Image Synthesis From a Single
Example Image, Thomas Vetter and Tomaso Poggio
PAMI 1997
10
Linear Transform Overview
x
x1
x2
x3
xref
2 Linear Object Classes and Image Synthesis
From a Single Example Image, Thomas Vetter and
Tomaso Poggio PAMI 1997
11
Overview Producing a Training Set

• Segment each face in the set
• Find a correspondence between each segmented face
  and a reference face
• Use Principle Components Analysis to find the
  optimal basis for the matrix comprised of the
  correspondence vectors

12
Segmentation

• First the face must be separated from the rest of
  the image
• A subjects hair, neck, ears, and background make
  the recognition process much more difficult
• This step is accomplished through a combination
  of jump edge detection and binary
  dilation/erosion

13
Correspondence Calculation

• Align the reference image and the input image
  using ICP (Iterative Closest Point 3)
• Find the Corresponding point to the reference
  face on the input face

Examples
Point to Surface Calculation
3 P. Besl and N. Mckay, A Method for
Registration of 3-D shapes , IEEE Transactions on
Pattern Analysis and Machine Intelligence,
February 1992
14
Different Correspondence Searches
15
Reference Face
16
Principle Component Analysis

• Generate an orthogonal basis comprised of the
  most important characteristics of faces in our
  training set

Training Set
Corresponded Data
PCA Basis
Synthesize New Image
17
Find the PCA Basis

• Compute f by subtracting the mean from each
  correspondence vector
• Compute the principal components using singular
  value decomposition
• The matrix V contains the principle
• components needed for reconstruction
• Given the principle components,
• we can then construct a class of
• signals for this training set

18
Synthesizing a New Face

• Given this information, we can then reconstruct
  any input scan within the class of signals
  provided by the training set
• Given an image x and the PCA Basis S, we want to
  determine the set of ? so we minimize the
  distance between x and ? , where
• From this synthesized image, we will then morph
  the image to that of a standard facial expression
  for matching

19
Examples (Nearest Neighbor Search Person
Reference Face)
This image in the training set
50,000 points Training set has 20 faces
Original Image
Reconstruction
Synthesized
This image not in the training set
20
Examples(Not in Training Set)(Nearest Neighbor
Search Person Reference Face)
50,000 points Training set has 150 faces
21
Training Set Size(not in training set) (Nearest
Neighbor Search Person Reference Face)
Synthesized Faces
Original
20 people, 1.09385e-006
50 people, 7.91476e-007
100 people, 5.4477e-007
22
New Correspondence Results
23
Conclusions

• New Normal Search with Synthetic faces produce
  considerably better results and new reconstructed
  faces are faithful to the original
• Smaller training sets are problematic, and
  previous work by Vetter and Blanz suggests that
  the training set of our size should be restricted
  to one age group/race which means ideally we
  should have a larger training set
• Normal Search with Synthetic faces results are
  preliminary and further experimentation is needed

24
Future Work

• Segment each face into multiple regions, and
  reconstruct them individually
• Incorporate larger expression dataset
• Begin synthesis of neutral image from expression
  coefficients
• By incorporating the 3D data and color at the
  location into the vector we give PCA more data to
  deal with. While this will almost certainly
  require a larger training set, it could allow for
  points that are corresponded in 3D that were
  mistakes to not look as close potentially
  creating a better final synthesis
• By looking at the distribution of points
  resulting from the normal intersection and
  manually spreading out clusters of points we may
  create a better vector. This is similar to
  arguing for better triangulation

25
Future Work Continued

• By attempting to determine gausian regions and
  determining correspondence along the different
  types of surface regions (Saddle, Ridge etc) we
  could have some built in markers to assist with
  correspondence
• ICP is using a point to surface nearest neighbor
  metric to determine an alignment. Since we have
  determined that searching for an intersection
  along a normal works better for an alignment,
  utilizing this method for ICPs criterion for a
  good fit as well should produce a better
  alignment in terms of our current correspondence
  criteria
• Normal to surface intersection function doesnt
  work very well (written in VTK). It can be
  improved since it breaks a lot. Also, it works
  within a tolerance which could be increased