1. Performance Driven Facial Animation
CSCI480/582 Lecture 31 Chap 5.3
Performance Driven Facial Animation
Apr, 13, 2009

2. Outline What differs facial animation from body animation? Face model
Facial motions and FACS
Facial motion synchronization with audio production
Performance-driven techniques

3. Face Model for Facial Animation
In modeling face, Four aspects needs to be considered
Structure: vertex distribution and topology to represent face in geometry
Appearance: color, texture, light reflection of face

4. Face Model - Geometry Dense mesh around the “holes”
The density of the mesh determines the level of details in facial deformation
Eyes, ears, and mouth cavity
Teeth and tongue

5. Face Model - Appearance
Using Texture Mapping
Unwinding the “cylinder” surface by concatenating multiple camera views to form the texture map

6. More Advanced Skin Appearance Model
The skin is translucent
Light is scattered through multiple layers, then reflected from a sub-layer
Bidirectional Reflectance Distribution Function is often used

7. Bidirectional Reflectance Distribution Function(BRDF)
A 4-dimensional function that defines how light is reflected at an opaque surface
Input:
incoming light direction wi
Outgoing direction wo
Output:
Ratio of reflected radiance along wo to irradiance incident along wi

8. Facial Motion Expressions involving Speeches involving
All front facial mesh
Eye ball orientation
Speeches involving
Lip and jaw
Tongue
The combination of expressions and speeches

9. Facial Acting Code System
Originally developed by Ekman and Friessen in 70s
Based on the physiology of facial muscles
A common standard to systematically categorize the physical expression of emotions

10. FACS – The Codes for Action Units
Action Descriptors
Inner Brow Raiser -- Frontalis (pars medialis)
2 Outer Brow Raiser -- Frontalis (pars lateralis)
4 Brow Lowerer -- Corrugator supercilii, Depressor supercilii
5 Upper Lid Raiser -- Levator palpebrae superioris
6 Cheek Raiser -- Orbicularis oculi (pars orbitalis)
7 Lid Tightener -- Orbicularis oculi (pars palpebralis)
9 Nose Wrinkler -- Levator labii superioris alaeque nasi
10 Upper Lip Raiser -- Levator labii superioris …
66 Action descriptors
Demo

11. Lip-Sync by Viseme Morphing
Use reference video to rotoscope the shape of the 3D model for key gestures
Use a set of viseme corresponding to phonemes, then interpolate between viseme gestures
Such approach often is not detailed enough for finer visual speech animation

12. Facial Motion Driven by Mocap Data
Captures subtle details of facial expressions
But, Mocap Data need be to cleaned for good rendering effect. Often this step is labor intensive.
How can we make Mocap more flexible?

13. FACS-Viseme Solver
Use a small set of Mocap data to synthesize a much larger range of facial motions
Each piece of mocap data in the small recording set represent an Action Unit
Use Action Unit analysis to decompose recorded Mocap data for easy post editing
Use audio-visual mapping analysis to align mocap segment for new speech