Facial Tracking and Animation

1
Facial Tracking and Animation

• Project Proposal
• Computer System Design
• Spring 2004
• Todd Belote David Brown
• Brad Busse Bryan Harris

2
Problem Background

• Speech Driven Facial Animation (PRISM)
• Facial animation from processed speech
• Previous Research (Jablonski Zavala)
• Low cost facial motion and speech processing
  using facial markers
• An infrared camera is used with infrared
  reflectors to pick up facial markers
• Microphone array for audio
• Winnov capture card (640x480 at 30 fps)
• Areas of Desired Improvement
• There is no current recovery method for point
  loss
• Feature points skewed in perspective create
  playback artifacts
• There are only 22 feature points, which cannot
  fully describe a face
• Initialization requires mouse-clicking the
  markers on the first frame
• The current algorithm is costly

3
Problem Statement

• Design a low cost computer system which can
  decode and analyze audio/video data in real-time
  and save the resulting analysis to disk.

4
Design Objectives

• Analyze video at 30 fps generate an FAP(Facial
  Animation Parameter) file.
• Continue with current audio analysis.
• All processing done in real-time.

5
Existing Solutions

• Lin et al, from National Taiwan University use a
  system with mirrors and camera to determine
  (x,y,z) coordinates of feature points
• Essa et al, from MIT Media Lab use computer
  recognition to analyze video in non real-time
  setting

6
Microphone Array, Camera, and Capture Card

• Audio sample rates range from 8 to 48 kHz
• Video sample rate will be 30 fps, and can be
  captured at 640x480 resolution
• Capture card brings audio and video into computer
  in synchrony

7
IBM Pupil Cam

• Camera designed to identify human pupils by
  emmitting infrared light using LEDs
• We will use infrared reflective markers to track
  facial movements

8
Design Constraints and Feasibility

• Cost and Speed
• The system must run in real-time
• Portability
• Hardware specific system
• Quicktime would make a more portable system
• Usability
• Point initialization
• Freedom of movement for each user
• Recovery from point occlusion

9
Alternative Solutions

• Leave system as-is
• Use mirrors to find (x,y,z) coordinates
• Quicktime libraries
• Individual facial templates

10
Design Validation

• Can generate FAP file in real-time
• Lost points can be recovered
• User can rotate her head without data loss
• Audio is analyzed in real-time
• Automatic point initialization

11
Societal, Safety and Environmental Analysis

• Primary use as a research tool
• Low bandwidth face to face phones using texture
  maps
• Low bandwidth phone support for the deaf
• Audio-visual data recording

12
Management

• Todd Belote Data Aquisition
• David Brown Marker Initialization
• Brad Busse Marker Tracking Algorithms
• Bryan Harris Facial Relationships

13
Scheduling

• Mondays 420 to 630 PM
• With Steve Ortiz
• Wednesdays 530 to 700 PM
• With Steve Ortiz and Marco Zavala
• Steve Ortiz Project Advisor
• Marco Zavala Previous Project Owner

14
Scheduling Gant Chart
15
Pert Chart