Deformation Modeling for Robust 3D Face Matching

1
Deformation Modeling for Robust 3D Face Matching

• Xioguang Lu and Anil K. JainDept. of Computer
  Science Engineering
• Michigan State University

2
Problem

• Although 3D facial scans do not vary with
  lighting or pose changes, nonrigid facial
  deformations can hurt recognition
• Collecting and storing multiple expression
  template scans for each subject is not practical
• Expressions can have differing intensities

3
Proposed Scheme

• A (hierarchical) geodesic sampling is used to
  quantify facial expression
• Expression variations are learned from a small
  control group
• These variations are used to create a deformable
  model from gallery templates
• This deformable model is fit to the target scan
  and matching distance computed

4
(No Transcript)
5
Sampling

• Landmarks are manually selected (nose tip, eye
  corners, mouth corners, and mouth contour)
• Geodesic distance between certain features is
  computed (hierarchically in latest work)
• Geodesics are split into L segments of equal
  length to generate L-1 new feature points

6
Deformation Transfer

• Register non-neutral scan with neutral scan of
  same face to estimate landmark displacement
• Establish a mapping F from the neutral gallery to
  the neutral target face
• Use F to transfer landmarks in the non-neutral
  gallery scan to the (synthesized) non-neutral
  target
• Establish a mapping ? from the neutral to
  non-neutral target
• Interpolate ? using thin-plate-spline mapping
• Boundary constraints are included in
  thin-plate-spline calculation as additional
  landmark points

7
Registration

• Neutral and non-neutral target are aligned using
  features which dont move much with expression
  changes, such as eye corners and nose tip
• This separates rigid transformations from
  nonrigid transformations

8
Thin-Plate Splines

• Goal find a mapping from landmark set U to V
  with known correspondences
• Method imagine V as a thin metal sheet and find
  a function which minimizes bending energy
• Solution F(u) c Au WTs(u)
• s(u) (u u1, u u2, )T
• An analytical solution can be obtained for 3D
  points

9
Deformable Model Construction

• To generate a deformable model, each learned
  expression is simulated on a neutral gallery face
• Face is represented as a combination of shape
  vectors
• M is the number of synthesized templates, ai is
  the weight of each template
• By adjusting the weights ai, various combinations
  of expressions can be generated
• To reduce computational complexity, one
  deformable model per expression is generated

10
Matching

• Coarse alignment performed as during deformation
  transfer
• Alignment refined with iterative closest point
  algorithm
• Associate each point with nearest neighbor,
  calculate transform to minimize distance, repeat
• Minimize a cost function by solving for ais
• R and T are rotation and translation matrices, S
  is the deformable model, and St is the test scan
• Use these ais to compute a new iterative closest
  point distance, and return to step 2 until
  convergence

11
Experiment I

• Self-collected database of 10 subjects at 3
  different poses, with 7 different expressions,
  for 210 total scans and 10 gallery models
• 5 subjects at random chosen as control group,
  leaving 105 scans for recognition
• Results

12
Experiment II

• Control group 10 subjects from Experiment I
• Test group 90 additional subjects, with 6 scans
  each at different viewpoints (in most cases)
• 533 total test scans
• Results

13
Experiment III

• A subset of FRGC v2.0 dataset
• Scans with the earliest timestamp and neutral
  expression are used as templates
• 50 gallery scans, 150 test scans
• 10 subjects in Experiment I used as control group
• Latest results (after publication)

14
Conclusions

• One area for improvement (noted in the paper) was
  the dependence on manual landmark labeling
• Also, I thought that there might be some
  application of geometric invariants to replace
  their registration step (which is subject to
  local minima)

15
Questions?