May 10, 2004Facial Tracking and Animation Todd Belote Bryan Harris David Brown Brad Busse

May 10, 2004Facial Tracking and Animation Problem Background Speech driven facial animation Correlate captured facial movements to audio patterns –Capture facial movements –Analyze corresponding audio

May 10, 2004Facial Tracking and Animation Goals and Objectives Develop an inexpensive, robust real-time system to track facial motion and process corresponding audio. The system must: –Cost around $1000 –Run on a personal computer –Allow for long periods of data acquisition –Handle head movements –Recover from point occlusion –Output only necessary information

May 10, 2004Facial Tracking and Animation System Functionality Capture Live Data Record Video Capture From Video Generate FAP from output file Process audio of.wav file

May 10, 2004Facial Tracking and Animation System Description DATA ACQUISITION FAP GENERATION POINT INITIALIZATION POINT TRACKING AUDIO PROCESSING Top level system organization –Illustrates data flow –Functional block division

May 10, 2004Facial Tracking and Animation Data Acquisition Video Data – 320x240 at 30 fps (bmp) Audio Data – 16 bit at 16kHz (wav) CAMERA MICROPHONE AVI MOVIE FILE EH AUDIO PROCESSING VIDEO PROCESSING FRAMEGRABBER SW TIMER CAPTURECARDCAPTURECARD WAV FILE

May 10, 2004Facial Tracking and Animation Point Initialization and Tracking Process frame data –Locate all areas in the frame which constitute a point Identify points –Initialization – algorithm to determine identity –Tracking – algorithm to match current frame points with previous location of identified points

May 10, 2004Facial Tracking and Animation Point Transform Approach: Criminisi et al. Maps any arbitrary quadrilateral onto any other This can account for all six degrees of freedom as well as perspective distortion, greatly simplifying the computation required to reorient the face When using an orientation square that encompasses most of the face, this algorithm can be made as accurate as necessary

May 10, 2004Facial Tracking and Animation Point Transform: Demo Actual Camera DataRe-oriented Processed Data

May 10, 2004Facial Tracking and Animation FAP Generation and Audio Processing FAP Generation –Process text file of point movement –Output FAE Engine compliant file of point displacements Audio Processing –LPC –MFCC –Pitch –Power

May 10, 2004Facial Tracking and Animation Environmental and Health Considerations All hardware is off the shelf No harm from infrared light No harm from other products –Eg. Reflective markers

May 10, 2004Facial Tracking and Animation Social, Political and Ethical Considerations Provide low cost audiovisual capture –Increase research in field by removing cost barrier –Further advances Eg. Phone for the deaf No Ethical Issues No Political affects

May 10, 2004Facial Tracking and Animation Economics and Sustainability No economies of scale due to narrow scope IBM PupilCAM is hard to locate and therefore sustainability with current hardware is issue –Other cameras could provide the same function

May 10, 2004Facial Tracking and Animation Thanks For the Contributions