Face Animation Overview with Shameless Bias Toward MPEG-4 Face Animation Tools Dr. Eric Petajan Chief Scientist and Founder face2face animation, inc. eric@f2f-inc.com

Computer-generated Face Animation Methods Morph targets/key frames (traditional) Speech articulation model (TTS) Facial Action Coding System (FACS) Physics-based (skin and muscle models) Marker-based (dots glued to face) Video-based (surface features)

Morph targets/key frames Advantages Complete manual control of each frame Good for exaggerated expressions Disadvantages Hard to achieve good lipsync without manual tweeking Morph targets must be downloaded to terminal for streaming animation (delay)

Speech articulation model Advantages High level control of face Enables TTS Disadvantages Robotic character Hard to sync with real voice

Facial Action Coding System Advantages Very high level control of face Maps to morph targets Explicit specification of emotional states Disadvantages Not good for speech Not quantified

Physics-based Advantages Disadvantages Good for realistic skin, muscle and fat Collision detection Disadvantages High complexity Must be driven by high level articulation parameters (TTS) Hard to drive with motion capture data

Marker-based Advantages Disadvantages Can provide accurate motion data from most of the face Face models can be animated directly from surface feature point motion Disadvantages Dots glued to face Dots must be manually registered Not good for accurate inner lip contour or eyelid tracking

Video-based Advantages Disadvantages Simple to capture video of face Face models can be animated directly from surface feature motion Disadvantages Must have good view of face

What is MPEG-4 Multimedia? Natural audio and video objects 2D and 3D graphics (based on VRML) Animation (virtual humans) Synthetic speech and audio

Samples versus Objects Traditional video coding is sample based (blocks of pixels are compressed) MPEG-4 provides visual object representation for better compression and new functionalities Objects are rendered in the terminal after decoding object descriptors

Object-based Functionalities User can choose display of content layers Individual objects (text, models) can be searched or stored for later used Content is independent of display resolution Content can be easily repurposed by provider for different networks and users

MPEG-4 Object Composition Objects are organized in a scene graph Scene graphs are specified using a binary format called BIFS (based on VRML) Both 2D and 3D objects, properties and transforms are specified in BIFS BIFS allows objects to be transmitted once and instanced repeatedly in the scene after transformations

MPEG-4 Operation Sequence

Faces are Special Humans are hard-wired to respond to faces The face is the primary communication interface Human faces can be automatically analyzed and parameterized for a wide variety of applications

MPEG-4 Face and Body Animation Coding Face animation is in MPEG-4 version 1 Body animation is in MPEG-4 version 2 Face animation parameters displace feature points from neutral position Body animation parameters are joint angles Face and body animation parameter sequences are compressed to low bitrates

Neutral Face Definition Head axes parallel to the world axes Gaze is in direction of Z axis Eyelids tangent to the iris Pupil diameter is one third of iris diameter Mouth is closed and the upper and lower teeth are touching Tongue is flat, horizontal with the tip of tongue touching the boundary between upper and lower teeth

Face Feature Points Right eye Left eye Nose Teeth Mouth Tongue y y x z 2.1 2.12 2.11 2.14 2.10 2.13 10.6 10.8 10.4 10.2 10.10 5.4 5.2 5.3 5.1 10.1 10.9 10.3 10.5 10.7 4.1 4.3 4.5 4.6 4.4 4.2 11.1 11.2 11.3 11.4 11.5 x y z 11.5 11.4 11.2 10.2 10.4 10.10 10.8 10.6 2.14 7.1 11.6 4.6 4.4 4.2 5.2 5.4 2.10 2.12 2.1 11.1 x y z Right eye Left eye 3.13 3.7 3.9 3.5 3.1 3.3 3.11 3.14 3.10 3.12 3.6 3.4 3.2 3.8 Nose 9.6 9.7 9.14 9.13 9.12 9.2 9.4 9.15 9.5 9.3 9.1 9.10 9.11 9.8 9.9 Teeth Mouth 8.1 8.9 8.10 8.5 8.3 8.7 8.2 8.8 8.4 8.6 2.2 2.3 2.6 2.8 2.9 2.7 2.5 2.4 Tongue 6.2 6.4 6.3 6.1 Feature points affected by FAPs Other feature points

Face Animation Parameter Normalization Face Animation Parameters (FAPs) are normalized to facial dimensions Each FAP is measured as a fraction of neutral face mouth width, mouth-nose distance, eye separation, or iris diameter 3 Head and 2 eyeball rotation FAPs are Euler angles

Neutral Face Dimensions for FAP Normalization

FAP Groups

Mouth closed if sum of upper and lower lip FAPs = 0

Face Model Independence FAPs are always normalized for model independence FAPs (and BAPs) can be used without MPEG-4 systems/BIFS Private face models can be accurately animated with FAPs Face models can be simple or complex depending on terminal resources

MPEG-4 BIFS Face Node Face node contains FAP node, Face scene graph, Face Definition Parameters (FDP), FIT,and FAT FIT (Face Interpolation Table) specifies interpolation of FAPs in terminal FAT (Face Animation Table) maps FAPs to Face model deformation FDP information included face feature points positions and texture map

Face Model Download 3D graphical models (e.g. faces) can be downloaded to the terminal with MPEG-4 3D model specification is based on VRML Face Animation Table( FAT) maps FAPs to face model vertex displacements Appearance and animation of downloaded face models is exactly predictable

FAP Compression FAPs are adaptively quantized to desired quality level Quantized FAPs are differentially coded Adaptive arithmetic coding further reduces bitrate Typical compressed FAP bitrate is less than 2 kilobits/second

- FAP Predictive Coding + FAP(t) Q Bitstream Arithmetic Coder Frame Delay Q-1

Face Analysis System MPEG-4 does not specify analysis systems face2face face analysis system tracks nostrils for robust operation Inner lip contour estimated using adaptive color thresholding and lip modeling Eyelids, eyebrows and gaze direction

Nostril Tracking

Inner Lip Contour Estimation

FAP Estimation Algorithm Head scale is normalized based on neutral mouth (closed mouth) width Head pitch is approximated based on vertical nostril deviation from neutral head position Head roll is computed from smoothed eye or nostril orientation depending on availability Inner lip FAPs are measured directly from the inner lips contour as deviations from the neutral lip position (closed mouth)

FAP Sequence Smoothing

MPEG-4 Visemes and Expressions A weighted combination of 2 visemes and 2 facial expressions for each frame Decoder is free to interpret effect of visemes and expressions after FAPs are applied Definitions of visemes and expressions using FAPs can also be downloaded

Visemes

Facial Expressions

Free Face Model Software Wireface is an openGL-based, MPEG-4 compliant face model Good starting point for building high quality face models for web applications Reads FAP file and raw audio file Renders face and audio in real time Wireface source is freely available

Body Animation Harmonized with VRML Hanim spec Body Animation Parameters (BAPs) are humanoid skeleton joint Euler angles Body Animation Table (BAT) can be downloaded to map BAPs to skin deformation BAPs can be highly compressed for streaming

Body Animation Parameters (BAPs) 186 humanoid skeleton euler angles 110 free parameters for use with downloaded body surface mesh Coded using same codecs as FAPs Typical bitrates for coded BAPs is 5-10kbps

Body Definition Parameters (BDPs) Humanoid joint center positions Names and hierarchy harmonized with VRML/Web3D H-Anim working group Default positions in standard for broadcast applications Download just BDPs to accurately animate unknown body model

Faces Enhance the User Experience Virtual call center agents News readers (e.g. Ananova) Story tellers for the child in all of us eLearning Program guide Multilingual (same face different voice) Entertainment animation Multiplayer games

Visual Content for the Practical Internet Broadband deployment is happening slowly DSL availability is limited and cable is shared Talking heads need high frame-rate Consumer graphics hardware is cheap and powerful MPEG-4 SNHC/FBA tools are matched to available bandwidth and terminals

Visual Speech Processing FAPs can be used to improve speech recognition accuracy Text-to-speech systems can use FAPs to animate face models FAPs can be used in computer-human dialogue systems to communicate emotions, intentions and speech especially in noisy environments

Video-driven Face Animation Facial expressions, lip movements and head motion transferred to face model FAPs extracted from talking head video with special computer vision system No face markers or lipstick is required Normal lighting is used Communicates lip movements and facial expressions with visual anonymity

Automatic Face Animation Demonstration FAPs extracted from camcorder video FAPs compressed to less than 2 kbits/sec 30 frames/sec animation generated automatically Face models animated with bones rig or fixed deformable mesh (real-time)

What is easy, solved, or almost solved Can we do photorealistic non-animated face models? YES Can we do near-real-time lip sync'ing that is indistinguishable from a human? NO

What is really hard Synthesizing human speech and facial expressions Hair

What we have assumed someone else is solving Graphics acceleration Video camera cost and resolution Multimedia communication infrastructure

Where we need help We have a face with 68 parameters but we need the psychologists to tell us how to drive it autonomously We need to embody our agents into graphical models that have a couple of thousand parameters to control gaze, gesture, body language, and do collision detection-> NEED MORE SPEED

Core functionality of the face Speech Lips, teeth, tongue Emotional expressions Gaze, eyebrow, eyelids, head pose Non-verbal communication Sensory responsivity Technical requirements Framerate Synchronization Latency Bitrate Spatial resolution Complexity Common framework withbody Interaction Different faces should respond similarly to common commands Accessible to everyone

Interaction with other components Language and discourse Phoneme to viseme mapping Given/new Action in the environment Global information Emotional state Personality Culture World knowledge Central time-base and timestamps

Open questions Central vs peripheral functionality Degree of interface commonality Degree of agent autonomy What should the VH be capable of