3D Face Reconstruction from Video

1
3D Face Reconstruction from Video
Unsang Park and Anil K. Jain Pattern Recognition
Image Processing Lab., Computer Science
Engineering

• Experimental Results
• 2D to 3D face reconstruction

Schematic Diagram of Face Recognition utilizing
3D face reconstruction from Video

• Motivation
• Face images in video show large variations in
  pose and lighting, Current state-of-the art face
  recognition engines are not very accurate when
  the probe and gallery images have different pose
  and lighting.

Texture mapping
Reconstructed 3D face model
Coarse reconstruction
Fine reconstruction (TPS)
Generic Model
2D video frames
(a) (b) (a) Pose and
lighting variations appearing in video. (b) Face
recognition performance with and without the
variations in probe data. Face recognition
performance drastically drops as the gallery data
does not contain the variations in the probe data.
Generate 2D face population
Enrollment stage
Recognition stage
Enroll in 2D face database

• Proposed Solution
• We propose a 2D to 3D face reconstruction
  technique to build 3D face models from two video
  streams without using the 3D range sensor.
• Reconstructed 3D face models are used to
  generate 2D face image population for the robust
  face recognition invariant to pose and lighting
  variation.

Match
Probe
Gallery
Identity

• Fine Reconstruction
• The generic model is fitted to the coarse
  reconstruction model by Thin plate spline (TPS)
  process. Let U ui i1, 2, ,n be the
  control points on the generic model and V be the
  control points on the coarse model. The
  non-linear deformation F(u) is obtained by
• The deformation F(u) is applied to all vertices
  in the generic model for the fine reconstruction.

• Coarse Reconstruction
• Coarse 3D reconstruction is performed based on a
  set of corresponding points between two images
  from the two video streams. The set of
  corresponding points are manually labeled.
• A closed form equation is solved to obtain the
  3D coordinates given 2D image coordinates (u,v)
  and calibration matrix (P) as

2D Face recognition utilizing reconstructed 3D
face models and multiple number of probe images
(1, 10, 20 and 30).

• Conclusions and Future work
• Fast and realistic 3D face reconstruction from
  2D video is developed.
• 2D face image population obtained from the
  reconstructed 3D face improved the face
  recognition performance.
• Automatic feature points extraction method for
  the coarse reconstruction process needs to be
  developed.
• A fully automated system from the video input to
  3D face reconstruction and face recognition will
  be developed.

• Generic Face Model Construction
• A generic model with about 5000 vertices is
  constructed from the Morphable model that is
  built using USF 3D face database.