Facial Tracking and Animation Project Proposal Computer System Design Spring 2004 Todd BeloteDavid Brown Brad BusseBryan Harris

Problem Background Speech Driven Facial Animation (PRISM) –Facial animation from processed speech Previous Research (Jablonski & Zavala) –Low cost facial motion and speech processing using facial markers –An infrared camera is used with infrared reflectors to pick up facial markers –Microphone array for audio –Winnov capture card (640x480 at 30 fps) Areas of Desired Improvement –There is no current recovery method for point loss –Feature points skewed in perspective create playback artifacts –There are only 22 feature points, which cannot fully describe a face –Initialization requires mouse-clicking the markers on the first frame –The current algorithm is costly

Problem Statement Design a low cost computer system which can decode and analyze audio/video data in real-time and save the resulting analysis to disk.

Design Objectives Analyze video at 30 fps; generate an FAP(Facial Animation Parameter) file. Continue with current audio analysis. All processing done in real-time.

Existing Solutions Lin et al, from National Taiwan University use a system with mirrors and camera to determine (x,y,z) coordinates of feature points Essa et al, from MIT Media Lab use computer recognition to analyze video in non real-time setting

Microphone Array, Camera, and Capture Card Audio sample rates range from 8 to 48 kHz Video sample rate will be 30 fps, and can be captured at 640x480 resolution Capture card brings audio and video into computer in synchrony

IBM Pupil Cam Camera designed to identify human pupils by emmitting infrared light using LEDs We will use infrared reflective markers to track facial movements

Design Constraints and Feasibility Cost and Speed –The system must run in real-time Portability –Hardware specific system Quicktime would make a more portable system Usability –Point initialization –Freedom of movement for each user –Recovery from point occlusion

Alternative Solutions Leave system as-is Use mirrors to find (x,y,z) coordinates Quicktime libraries Individual facial templates

Design Validation Can generate FAP file in real-time Lost points can be recovered User can rotate her head without data loss Audio is analyzed in real-time Automatic point initialization

Societal, Safety and Environmental Analysis Primary use as a research tool Low bandwidth face to face phones using texture maps Low bandwidth phone support for the deaf Audio-visual data recording

Management Todd Belote – Data Aquisition David Brown – Marker Initialization Brad Busse – Marker Tracking Algorithms Bryan Harris – Facial Relationships

Scheduling Mondays – 4:20 to 6:30 PM –With Steve Ortiz Wednesdays – 5:30 to 7:00 PM –With Steve Ortiz and Marco Zavala Steve Ortiz – Project Advisor Marco Zavala – Previous Project Owner

Scheduling – Gant Chart

Pert Chart