1. Face Animation Overview with Shameless Bias Toward MPEG-4 Face Animation Tools
Dr. Eric Petajan
Chief Scientist and Founder
face2face animation, inc.

2. Computer-generated Face Animation Methods
Morph targets/key frames (traditional)
Speech articulation model (TTS)
Facial Action Coding System (FACS)
Physics-based (skin and muscle models)
Marker-based (dots glued to face)
Video-based (surface features)

3. Morph targets/key frames
Advantages
Complete manual control of each frame
Good for exaggerated expressions
Disadvantages
Hard to achieve good lipsync without manual tweeking
Morph targets must be downloaded to terminal for streaming animation (delay)

4. Speech articulation model
Advantages
High level control of face
Enables TTS
Disadvantages
Robotic character
Hard to sync with real voice

5. Facial Action Coding System
Advantages
Very high level control of face
Maps to morph targets
Explicit specification of emotional states
Disadvantages
Not good for speech
Not quantified

6. Physics-based Advantages Disadvantages
Good for realistic skin, muscle and fat
Collision detection
Disadvantages
High complexity
Must be driven by high level articulation parameters (TTS)
Hard to drive with motion capture data

7. Marker-based Advantages Disadvantages
Can provide accurate motion data from most of the face
Face models can be animated directly from surface feature point motion
Disadvantages
Dots glued to face
Dots must be manually registered
Not good for accurate inner lip contour or eyelid tracking

8. Video-based Advantages Disadvantages Simple to capture video of face
Face models can be animated directly from surface feature motion
Disadvantages
Must have good view of face

9. What is MPEG-4 Multimedia?
Natural audio and video objects
2D and 3D graphics (based on VRML)
Animation (virtual humans)
Synthetic speech and audio

10. Samples versus Objects
Traditional video coding is sample based (blocks of pixels are compressed)
MPEG-4 provides visual object representation for better compression and new functionalities
Objects are rendered in the terminal after decoding object descriptors

11. Object-based Functionalities
User can choose display of content layers
Individual objects (text, models) can be searched or stored for later used
Content is independent of display resolution
Content can be easily repurposed by provider for different networks and users

12. MPEG-4 Object Composition
Objects are organized in a scene graph
Scene graphs are specified using a binary format called BIFS (based on VRML)
Both 2D and 3D objects, properties and transforms are specified in BIFS
BIFS allows objects to be transmitted once and instanced repeatedly in the scene after transformations

13. MPEG-4 Operation Sequence

15. Faces are Special Humans are hard-wired to respond to faces
The face is the primary communication interface
Human faces can be automatically analyzed and parameterized for a wide variety of applications

16. MPEG-4 Face and Body Animation Coding
Face animation is in MPEG-4 version 1
Body animation is in MPEG-4 version 2
Face animation parameters displace feature points from neutral position
Body animation parameters are joint angles
Face and body animation parameter sequences are compressed to low bitrates

17. Neutral Face Definition
Head axes parallel to the world axes
Gaze is in direction of Z axis
Eyelids tangent to the iris
Pupil diameter is one third of iris diameter
Mouth is closed and the upper and lower teeth are touching
Tongue is flat, horizontal with the tip of tongue touching the boundary between upper and lower teeth

18. Face Feature Points Right eye Left eye Nose Teeth Mouth Tongue y y x z
2.1
2.12
2.11
2.14
2.10
2.13
10.6
10.8
10.4
10.2
10.10
5.4
5.2
5.3
5.1
10.1
10.9
10.3
10.5
10.7
4.1
4.3
4.5
4.6
4.4
4.2
11.1
11.2
11.3
11.4
11.5 x y z
11.5
11.4
11.2
10.2
10.4
10.10
10.8
10.6
2.14
7.1
11.6
4.6
4.4
4.2
5.2
5.4
2.10
2.12
2.1
11.1 x y z
Right eye
Left eye
3.13
3.7
3.9
3.5
3.1
3.3
3.11
3.14
3.10
3.12
3.6
3.4
3.2
3.8
Nose
9.6
9.7
9.14
9.13
9.12
9.2
9.4
9.15
9.5
9.3
9.1
9.10
9.11
9.8
9.9
Teeth
Mouth
8.1
8.9
8.10
8.5
8.3
8.7
8.2
8.8
8.4
8.6
2.2
2.3
2.6
2.8
2.9
2.7
2.5
2.4
Tongue
6.2
6.4
6.3
6.1
Feature points affected by FAPs
Other feature points

19. Face Animation Parameter Normalization
Face Animation Parameters (FAPs) are normalized to facial dimensions
Each FAP is measured as a fraction of neutral face mouth width, mouth-nose distance, eye separation, or iris diameter
3 Head and 2 eyeball rotation FAPs are Euler angles

20. Neutral Face Dimensions for FAP Normalization

21. FAP Groups

22. Mouth closed if sum of upper and lower lip FAPs = 0

23. Face Model Independence
FAPs are always normalized for model independence
FAPs (and BAPs) can be used without MPEG-4 systems/BIFS
Private face models can be accurately animated with FAPs
Face models can be simple or complex depending on terminal resources

24. MPEG-4 BIFS Face Node
Face node contains FAP node, Face scene graph, Face Definition Parameters (FDP), FIT,and FAT
FIT (Face Interpolation Table) specifies interpolation of FAPs in terminal
FAT (Face Animation Table) maps FAPs to Face model deformation
FDP information included face feature points positions and texture map

25. Face Model Download
3D graphical models (e.g. faces) can be downloaded to the terminal with MPEG-4
3D model specification is based on VRML
Face Animation Table( FAT) maps FAPs to face model vertex displacements
Appearance and animation of downloaded face models is exactly predictable

26. FAP Compression FAPs are adaptively quantized to desired quality level
Quantized FAPs are differentially coded
Adaptive arithmetic coding further reduces bitrate
Typical compressed FAP bitrate is less than 2 kilobits/second

27. - FAP Predictive Coding + FAP(t) Q Bitstream Arithmetic Coder Frame
Delay
Q-1

28. Face Analysis System MPEG-4 does not specify analysis systems
face2face face analysis system tracks nostrils for robust operation
Inner lip contour estimated using adaptive color thresholding and lip modeling
Eyelids, eyebrows and gaze direction

29. Nostril Tracking

30. Inner Lip Contour Estimation

31. FAP Estimation Algorithm
Head scale is normalized based on neutral mouth (closed mouth) width
Head pitch is approximated based on vertical nostril deviation from neutral head position
Head roll is computed from smoothed eye or nostril orientation depending on availability
Inner lip FAPs are measured directly from the inner lips contour as deviations from the neutral lip position (closed mouth)

32. FAP Sequence Smoothing

33. MPEG-4 Visemes and Expressions
A weighted combination of 2 visemes and 2 facial expressions for each frame
Decoder is free to interpret effect of visemes and expressions after FAPs are applied
Definitions of visemes and expressions using FAPs can also be downloaded

34. Visemes

35. Facial Expressions

36. Free Face Model Software
Wireface is an openGL-based, MPEG-4 compliant face model
Good starting point for building high quality face models for web applications
Reads FAP file and raw audio file
Renders face and audio in real time
Wireface source is freely available

37. Body Animation Harmonized with VRML Hanim spec
Body Animation Parameters (BAPs) are humanoid skeleton joint Euler angles
Body Animation Table (BAT) can be downloaded to map BAPs to skin deformation
BAPs can be highly compressed for streaming

38. Body Animation Parameters (BAPs)
186 humanoid skeleton euler angles
110 free parameters for use with downloaded body surface mesh
Coded using same codecs as FAPs
Typical bitrates for coded BAPs is 5-10kbps

39. Body Definition Parameters (BDPs)
Humanoid joint center positions
Names and hierarchy harmonized with VRML/Web3D H-Anim working group
Default positions in standard for broadcast applications
Download just BDPs to accurately animate unknown body model

40. Faces Enhance the User Experience
Virtual call center agents
News readers (e.g. Ananova)
Story tellers for the child in all of us
eLearning
Program guide
Multilingual (same face different voice)
Entertainment animation
Multiplayer games

41. Visual Content for the Practical Internet
Broadband deployment is happening slowly
DSL availability is limited and cable is shared
Talking heads need high frame-rate
Consumer graphics hardware is cheap and powerful
MPEG-4 SNHC/FBA tools are matched to available bandwidth and terminals

42. Visual Speech Processing
FAPs can be used to improve speech recognition accuracy
Text-to-speech systems can use FAPs to animate face models
FAPs can be used in computer-human dialogue systems to communicate emotions, intentions and speech especially in noisy environments

43. Video-driven Face Animation
Facial expressions, lip movements and head motion transferred to face model
FAPs extracted from talking head video with special computer vision system
No face markers or lipstick is required
Normal lighting is used
Communicates lip movements and facial expressions with visual anonymity

44. Automatic Face Animation Demonstration
FAPs extracted from camcorder video
FAPs compressed to less than 2 kbits/sec
30 frames/sec animation generated automatically
Face models animated with bones rig or fixed deformable mesh (real-time)

46. What is easy, solved, or almost solved
Can we do photorealistic non-animated face models? YES
Can we do near-real-time lip sync'ing that is indistinguishable from a human? NO

47. What is really hard Synthesizing human speech and facial expressions
Hair

48. What we have assumed someone else is solving
Graphics acceleration
Video camera cost and resolution
Multimedia communication infrastructure

49. Where we need help
We have a face with 68 parameters but we need the psychologists to tell us how to drive it autonomously
We need to embody our agents into graphical models that have a couple of thousand parameters to control gaze, gesture, body language, and do collision detection-> NEED MORE SPEED

50. Core functionality of the face
Speech
Lips, teeth, tongue
Emotional expressions
Gaze, eyebrow, eyelids, head pose
Non-verbal communication
Sensory responsivity
Technical requirements
Framerate
Synchronization
Latency
Bitrate
Spatial resolution
Complexity
Common framework withbody
Interaction
Different faces should respond similarly to common commands
Accessible to everyone

51. Interaction with other components
Language and discourse
Phoneme to viseme mapping
Given/new
Action in the environment
Global information
Emotional state
Personality
Culture
World knowledge
Central time-base and timestamps

52. Open questions Central vs peripheral functionality
Degree of interface commonality
Degree of agent autonomy
What should the VH be capable of