REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS

1
REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS

• Ginés García Mateos
• Dept. de Informática y Sistemas
• University of Murcia - SPAIN

2
Introduction

• Main objective develop a new technique to track
  human faces and facial features
• Working in real-time fast processing
• Under realistic conditions robust to facial
  expressions, lighting conditions, 3D head pose
  and movements
• With high location accuracy facial features
  location (eyes, nose, mouth)

3
Introduction

• Index of the presentation
• Face integral projections
• Integral projection models
• Alignment of projections
• The tracking process
• Experimental results
• Conclusions

4
Face integral projections

• Definition. Let i(x,y) be an image, and R(i) a
  region in it
• Vertical integral projection
• PVR ymin, ..., ymax ? R
• PVR(y) i(x,y) ? (x,y) ? R(i)
• Horizontal integral projection
• PHR xmin, ..., xmax ? R
• PHR(x) i(x,y) ? (x,y) ? R(i)

PVFACE(y)
FACE
y
EYES
PHEYES(x)
x
5
Face integral projections

• Dimensionality reduction 3D world ? 2D images ?
  1D integral projections
• Advantages
• Fast to compute and to process
• Disadvantages
• Loss of information. Is it relevant for the
  problem?
• What happens when applied to human faces?

6
Face integral projections

• Different individuals

7
Face integral projections

• Different facial expressions

8
Face integral projections

• Different segmented regions

9
Integral project. models

• Face integral projec. is an interesting and
  robust feature for tracking
• It has been applied using heuristic analysis
  max-min search, fuzzy logic, thresholding
  projections
• Proposal define and work with adaptable
  projection models
• How to model a variety of projection patterns?

10
Integral project. models

• A projection model is a pair
• M mmin, ..., mmax ? R (Mean)
• V mmin, ..., mmax ? R (Variance)

11
Integral project. models

• Advantages of working with explicit projection
  models
• The model is learnt from examples. In tracking,
  it is adapted to tracked faces
• We can define a signal to model distance

12
Integral project. models

• Advantages of working with explicit projection
  models
• The model can be reprojected

Reprojection (by outer product) using 1 vertical
IP and 2 horizontal IP
13
Alignment of projections

• Corresponding facial features should be projected
  on the same locations

Before alignment After alignment
14
Alignment of projections

• Alignment with respect to a model
• Problem formulation
• Let S smin, ..., smax ? R be a signal
• Let M,V mmin, ..., mmax ? R be a model
• Let S be a family of scale and translations
  alignments of S
• Find parameters (a,b,c,d,e) which minimize

15
The tracking process

• Tracking is based on the alignment of integral
  projections
• Main steps
• 1. Prediction and segmentation
• 2. Vertical alignment
• 3. Horizontal alignment
• 4. Orientation estimation

16
The tracking process

• Features to track
• Input to the tracker

• Bounding ellipse
• Facial features eyes and mouth

• State of tracking in frame t-1
• Frame t

• Face model

17
The tracking process

• 1. Prediction and segmentation
• Null predictor locations in frame t-1 are used
  to extract the face in frame t

Wrapped
Segmented
Predicted location
18
The tracking process

• 2. Vertical alignment
• Using the vertical projection of the face, and
  the model, align the face vertically

Align PVFACE to MVFACE
Segmented
PVFACE(y)
y
Model
Align using the obtained parameters (a,b,c,d,e)
19
The tracking process

• 3. Horizontal alignment
• Using the horizontal projection of the eyes
  region, align the face horizontally

Segmented after step 2
PHEYES(x)
Align PHEYES to MHEYES
y
Model
x
Align using the obtained parameters (a,b,c,d,e)
20
The tracking process

• 4. Orientation estimation
• Using vertical projections of each eye, estimate
  the orientation of the face

Align PVEYEi to MVEYEi
Segmented after step 3
PVEYE1, PVEYE2
y
Model
21
The tracking process

• Global structure of the tracker

1. Prediction and segmentation
2. Vertical alignment
3. Horizontal alignment
4. Orientation est.
22
Experimental results

• Experiments
• Location accuracy
• Execution time per frame
• Robustness to facial expressions, 3D pose,
  lighting conditions
• Different sources TV, video-conference camera
  and DVD
• Compared with CamShift algorithm

23
Experimental results

• FILE NAME tl5-02.avi SOURCE TV
• FORMAT 640x480 (25 fps) LENGTH 280 frames
• MODEL SIZE (pixels) 97x123

• AVG/MAX ERROR (mm) 3.41 / 12.9
• TIME/FRAME (ms) 4.02

24
Experimental results

• FILE NAME a3-05.avi SOURCE TV
• FORMAT 640x480 (25 fps) LENGTH 541 frames
• MODEL SIZE (pixels) 101x136

• AVG/MAX ERROR (mm) 1.95 / 9.76
• TIME/FRAME (ms) 4.62

25
Experimental results

• FILE NAME a3-2.avi SOURCE TV
• FORMAT 320x240 (25 fps) LENGTH 440 frames
• MODEL SIZE (pixels) 75x95

• AVG/MAX ERROR (mm) -
• TIME/FRAME (ms) 3.74

26
Experimental results

• FILE NAME ggm2.avi SOURCE QuickCam
• FORMAT 320x240 (25 fps) LENGTH 655 frames
• MODEL SIZE (pixels) 70x91

• AVG/MAX ERROR (mm) 1.83 / 9.29
• TIME/FRAME (ms) 3.69

27
Experimental results

• FILE NAME sw2-1.avi SOURCE DVD
• FORMAT 320x240 (30 fps) LENGTH 427 frames
• MODEL SIZE (pixels) 94x115

• AVG/MAX ERROR (mm) -
• TIME/FRAME (ms) 3.57

28
Experimental results

• Location accuracy
• Errors in mm (in the face plane) using a
  ground-truth location of facial features
• Average error below 4 mm, maximum error 14 mm
• With CamShift average error over 10 mm, maximum
  error 30 mm

29
Experimental results

• Execution time, per frame
• Off-the-self PC AMD Athlon at 1.2 GHz
• Average time below 5 ms, with 640x480 resolution,
  face size 100x120 pixels
• With CamShift average time about 10 ms, unable
  to work in one video sequence

30
Conclusions

• The tracking problem is decomposed into three
  main independent steps
• Vertical alignment
• Horizontal alignment
• Orientation estimation
• The process is fast, accurate and robust in the
  tested conditions
• It is exclusively based on integral projections

31
Conclusions

• The tracker is not affected by background
  distractors
• It can be applied either in color and grey-scale
  images
• Main limitation maximum allowed movement
  (approx. 1 m/s, at 25 fps)
• Future work improve the prediction step, e.g.
  with Kalman filters

32
Last

• This work has been supported by Spanish CICYT
  project DPI-2001-0469-C03-01
• Demo videos
• http//dis.um.es/ginesgm/fip
• Grupo PARP web page
• http//dis.um.es/parp
• Thank you very much