3D Face Reconstruction from Monocular or Stereo Images.

1
3D Face Reconstruction from Monocular or Stereo
Images.
Thomas Vetter
University of Basel
Switzerland
http//gravis.cs.unibas.ch
2
Change Your Image ...
3
Analysis by Synthesis
Image
3D World
Image Description
4
Approach Example based modeling of faces
2D Image 2D Face
Examples
w1 w2
w3 w4
. . .
5
Morphing 3D Faces
1
__


2
6
Shape and Texture Vectors
Example i
Reference Head
7
Surface registration Which representation?
8
Registration in different representations

• Implicit
• Triangulated
• Parameterized

9
Database of 3D Faces
10
Vector space of 3D faces.

• A Morphable Model can generate new faces.

a1 a2 a3
a4 . . .

b1 b2 b3
b4 . . .
11
Manipulation of Faces
12
Continuous Modeling in Face Space
13
Modelling the Appearance of Faces
A face is represented as a point in face space.

• Which directions code for specific attributes ?

14
Learning from Labeled Example Faces
Fitting a regression function
15
Facial Attributes
Gender
Original
16
3D Shape from Images
Input Image
3D Head
17
Matching a Morphable 3D-Face-Model

• R Rendering Function
• Parameters for Pose, Illumination, ...
• 
  
• Find optimal a, b, r !

18
Automated Parameter Estimation
150 shape coefficients ai 150 texture
coefficients bi

• Face Parameters

head position head orientation focal length

• 3D Geometry

• Ambient intensity, color
• Parallel intensity, color, direction
• Color contrast, gains, offsets

• Light and Color

19
Image Formation at each Vertex k
bi
ai
20
Error Function

• Image difference (pixel intensity cost function)
  
• Plausible parameters
• Minimize

21
animation by Volker Blanz.
22
Using Multiple Features
?
23
Which Feature to use?
24
Edge Feature
bi
ai
25
Edge Fitting Results
26
Multi-Features Fitting Algorithm
Stage Nb. Features Parameters Nb. of Parameters
1 Anchor, edges rigid 7
2 edges rigid, . 27
3 pixel intensity, prior illumination, . 32
4 edges, pixel intensity, prior, texture constraint 217
5 edges, pixel intensity, prior, texture constraint specular highlight 792
27
Multi-Features Fitting Algorithm
At stage 4
28
Recognition from Images
29
3D Computer Graphics
30
Correct Identification 1 out of 68 ()
gallery
profile
side
front
probe
99.8
front
99.9
side
98.3
profile
total
CMU-PIE database 4488 images of 68 individuals
3 poses x 22
illuminations 66 images per individual
31
Reanimation of Images
V. Blanz, C. Basso, T. Poggio T.
Vetter Reanimating Faces in images and Video
Proc. of Eurographics 2003
32
Expression Transfer
33
Analysis by Synthesis
3D World
Image Description
Image
34
Segmenting hair a general requirement ?
35
Skin segmentation
We need to mask out non-skin regions / outliers
3DMM is not sufficient
36
Shading Problem
Skin regions contain strong intensity gradients
that make a segmentation difficult!
37
Illumination Compensation
38
Illumination Compensation
Local fitting

• Skin Detail Analysis for Face RecognitionJean
  Sebastian Pierrard , Thomas Vetter CVPR 2007

39
(Skin) Texture Similarity
Basic idea Compare image texture with samples
that are known to be skin
40
Skin Segmentation
Texture similarity facilitates simple
segmentation by thresholding method
Get threshold from in seed region
Result still affected by shading
Compute texture similarity on
41
Segmentation Results
Thresholding

• Skin Detail Analysis for Face RecognitionJean
  Sebastian Pierrard , Thomas Vetter CVPR 2007

42
Try New Hairstyles
3D Shape and Texture
43
More Hairstyles
3D Shape and Texture
3D Angle, Position Illumination, Foreground,
Background
44
Using more than a single image ?
45
Model Based Stereo
46
Model Based Stereo
47
Silhouette Term
48
Colour Difference Term
49
Results
50
Results
51
(No Transcript)
52
Results on Flash Data
Ground Truth Monocular Stereo
53
Acknowledgement
Volker Blanz Sami Romdhani Brian Amberg Jaen
Sabastian Pierrard