Face Recognition Based on Fitting a 3D Morphable Model Volker Blanz

1
Face Recognition Based on Fitting a 3D Morphable
Model(Volker Blanz Thomas Vetter)

• Billy Tsafack (tbillous_at_web.de)

2
Outline

• Introduction / Current applications
• Paradigms for model-based recognition
• Morphable Model of 3D Face
• Experimental Results / Conclusions
• Targets for the future

3
Introduction

• ? as old as computer vision.
• ? other methods of identification (such as
  fingerprints, or iris scans) could be more
  accurate.
• But because of
• - its non-invasive nature and because,
• - it is people's primary method of person
  identification,
• face recognition always remains a major focus of
  research.

4
Applications of Face Recognition Systems

• Growing numbers of applications are starting to
  use face-recognition as the initial step towards
  interpreting human actions, intention, and
  behavior, as a central part of next-generation
  smart environments
• Entertainment video game, human-robot-interaction
  
• Smart cards drivers licenses, national ID,
  voter registration
• Information security TV Parental Control,
  internet access
• Law Enforcement and Surveillance ...

5
Main Problems in Face Recognition

• Illumination variability ? same object can appear
  dramatically different, even when viewed in fixed
  pose
• How to handle it?
• Employ a representation that is either invariant
  to, or models this variability

6
Influence of the illumination

• The variations between the images of the same
  face due to illumination and viewing direction
  are almost always larger than image variations
  due to change in face identity.
• -- Moses, Adini, Ullman,
  ECCV94

7
Influence of the illumination

• Source Simon Baker (Carnegie Mellon University)

8
Main Problems in Face Recognition

• Variation in Pose ranging from frontal to profile
  views
• 3D Faces can either be generated automatically
  from one or more photographs, or modeled directly
  through an intuitive user interface
• How to handle it?
• Simulation of the process of image formation in
  3D Space.
• Estimation of 3D Shape and Texture of Faces from
  single images.

9
Variation in pose
10
Outline

• Introduction / Current applications
• Paradigms for model-based recognition
• Morphable Model of 3D Face
• Experimental Results / Conclusions
• Targets for the future

11
Face Recognition Based on Fitting a 3D Morphable
Model

• A method for face recognition across variations
  in pose, ranging from frontal to profile views,
• and across a wide range of illuminations,
  including cast shadows and specular reflections.

12
Approach of the algorithm

• construct the morphable models from 3D Scans
• fitting the model to image for 3D Shape
  reconstruction
• measuring similarity of face for face recognition

13
Some definitions

• morph change shape via computer animation
• Training set set of images used for training the
  model
• Gallery set of images that includes all
  individuals who are known to the system
• Probe set of images for testing

14
Identification vs Verification

• Identification
• The system reports which person from the gallery
  is shown on the probe image.
• Verification
• A person claims to be a particular member of the
  gallery. The system decides if the probe and the
  gallery image is the same person.

15
Paradigms for model-based recognition

• Paradigm 1
• fit the model
• recognition based on model coefficients
• (intrinsic shape and texture of faces).
• They are independent of the imaging conditions.

16
Paradigms for model-based recognition

• Paradigm 1 (next)
• Analyse of all gallery images by the algorithm.
  Shape and texture coefficients are finally
  stored.
• Identification Computation of the coefficients
  by the fitting algorithm. Comparison with all
  gallery data in order to find the nearest
  neighbour of the probe image.

17
Paradigms for model-based recognition

• Paradigm 2
• Generate synthetic views from gallery or probe
  images
• Transfer of the synthetic views to a second
  viewpoint-dependent recognition system.

18
Paradigms for model-based recognition

• (For identification, the model coefficients of
  the probe image are compared with the stored
  coefficients of all gallery images.)

19
Outline

• Introduction / Current applications
• Paradigms for model-based recognition
• Morphable Model of 3D Face
• Experimental Results / Conclusions
• Targets for the future

20
Morphable Model of 3D Faces

• Shape vector
• that contains the X, Y, Z- coordinates of its n
  vertices
• Texture vector
• that contains the R, G, B color values of the n
  corresponding vertices.

21
Morphable Model of 3D Faces

• S, T convex combination of shape and texture
  vectors Si and Ti.
• They should describe a realistic human face.
• (continuous change of the parameters generate a
  smooth transition ...)

22
Morphable Model of 3D Faces

• Principal Component Analysis (PCA) performs a
  basis transformation to an orthogonal coordinate
  system formed by the eigenvectors si and ti of
  the covariance matrices
• the eigenvalues of the shape covariance
  matrix
• probability density within free space for
  the coefficients

23
Morphable Model of 3D Faces

• The expressiveness of the model can be increased
  by dividing faces into dependent subregions that
  are morphed independently ( Into eyes, nose,
  mouth, surrounding region) ? Segmentation
• Segmentation subdividing the vector space of
  faces into independent subspaces.

24
Morphable Model of 3D Faces
25
Morphable Model of 3D Faces

• Facial attributes hardly described by numbers.
• How to handle it?
• Record two scans of the same individual with
  different expressions
• and add the differences
• to a different individual in neutral
  expression.
• Distinctiveness
• commonly manipulated in caricatures.

26
Morphable Model of 3D Faces
27
Model based image analysis

• Goal
• represent a novel face in an image by model
  coefficients i and i and provide a
  reconstruction of 3D shape
• Steps
• Synthetise the image (image positions of
  vertices, illumination and color)
• Fitting the Model to an Image

28
Model based image analysis

• ? The fitting process finds shape and texture
  coefficients and describes a three-dimensional
  face model.
• ? Rendering R produces an image Imodel that
  is as similar as possible to Iinput.

29
Face Vectors

• The flow field is used to form shape and texture
  vectors S and T.
• Match between the reference face and the
  corresponding point through the flow field.
• 3D Laser Scans parametrized by cylindrical
  coordinates (h, phi)

30
Outline

• Introduction / Current applications
• Paradigms for model-based recognition
• Morphable Model of 3D Face
• Experimental Results / Conclusions
• Targets for the future

31
Results

• Model fitting and identification were tested on
  two publicly available databases of images
• PIE-CMU Database (for the colored images)
• FERET Database (for the gray-level images)
• Both cover a wide ethnic variety. Partial
  occlusion by hair and glasses. But no
  compensation for these effects.

32
Results Model Fitting

• Up to seven feature points were manually labeled
  in front and in side views, up to eight in
  profile views.

33
Results Model Fitting (FERET)

• ? Reconstructions of 3D shape and texture from
  FERET images (top row).
• ?In the second row, results are rendered into
  the original images with pose and illumination
  recovered by the algorithm.
• ?The third row shows novel views.

34
Results Model fitting (CMU-PIE)

• Top originals
• Middle reconstructions rendered into originals
• Bottom novel views.
• (The pictures shown here are difficult due to
  harsh illumination, profile views, or eye
  glasses. Illumination in the third image is not
  fully recovered, so part of the reflections are
  attributed to texture.)

35
Results Recognition Performance

• The error of pose is within a few degrees (lt5).

36
Results Recognition Performance

• Sum of the Mahalanobis distances of the segments
  shapes and textures
• Variations of model coefficients obtained from
  different images of the same person
• Cosine of the angle between two vectors

37
Results Recognition Performance

• For CMU-PIE images, data were computed for the
  side view gallery.
• Promising results
• Overall Percentage of Successfull Identifications
  to Different Criteria of Comparing Faces

38
Results Recognition Performance
39
Results Recognition Performance

• This table lists the percentages of correct
  identifications on the FERET set, based on front
  view gallery images ba, along with the estimated
  head poses obtained from fitting.

40
Results Recognition Performance

• True Accept (Hit Rate)
• False Accept (False Alarm)
• False Reject
• True Reject

41
Results Recognition Performance
42
Conclusions

• 1) Learning class-specific information about
  human faces from a data set of examples
• 2) Estimating 3D shape and texture, along
• with all relevant 3D scene parameters, from a
  single image at any pose and illumination
• 3) Representing and comparing faces for
  recognition tasks.

43
Conclusions

• The algorithm achieved promising results
• ? 3D morphable model is a powerful and versatile
  representation for human faces
• ? Runtime 4,5 minutes on 2 GHz Pentium 4
  processor.

44
Outline

• Introduction / Current applications
• Paradigms for model-based recognition
• Morphable Model of 3D Face
• Experimental Results / Conclusions
• Targets for the future

45
Next Targets

• extend the morphable model to different ages,
  ethnic groups, and facial expressions by
  including face vectors from more 3D scans,
• improve 3D reconstructions and identification
  (the algorithm currently ignores glasses, beards,
  or strands of hair covering part of face),
• Automated initialization and faster fitting
  procedure.

46
Literature

• 1 J.J. Atick, P.A. Griffin, and A.N. Redlich,
  Statistical Approach to Shape from Shading
  Reconstruction of 3D Face Surfaces from Single 2D
  Images, Computation in Neurological Systems,
  vol. 7, no. 1, 1996.
• 2 J.R. Bergen and R. Hingorani, Hierarchical
  Motion-Based Frame Rate Conversion, technical
  report, David Sarnoff Research Center, Princeton
  N.J., 1990.
• 3 D. Beymer and T. Poggio, Face Recognition
  from One Model View, Proc. Fifth Intl Conf.
  Computer Vision, 1995.
• 4 D. Beymer and T. Poggio, Image
  Representations for Visual Learning, Science,
  vol. 272, pp. 1905-1909, 1996.
• 5 C.M. Bishop, Neural Networks for Pattern
  Recognition. Oxford Univ.ress, 1995.
• 6 V. Blanz, Automatische Rekonstruktion der
  dreidimensionalen Form von Gesichtern aus einem
  Einzelbild, PhD thesis, Tübingen, Germany, 2000.
• 7 V. Blanz, S. Romdhani, and T. Vetter, Face
  Identification across Different Poses and
  Illuminations with a 3D Morphable Model, Proc.
  Fifth Intl Conf. Automatic Face and Gesture
  Recognition, pp. 202-207, 2002.
• ...