Performance Driven Facial Animation CSCI480/582 Lecture 31 Chap 5.3 Performance Driven Facial Animation Apr, 13, 2009

Outline What differs facial animation from body animation? Face model Facial motions and FACS Facial motion synchronization with audio production Performance-driven techniques

Face Model for Facial Animation In modeling face, Four aspects needs to be considered Structure: vertex distribution and topology to represent face in geometry Appearance: color, texture, light reflection of face

Face Model - Geometry Dense mesh around the “holes” The density of the mesh determines the level of details in facial deformation Eyes, ears, and mouth cavity Teeth and tongue

Face Model - Appearance Using Texture Mapping Unwinding the “cylinder” surface by concatenating multiple camera views to form the texture map

More Advanced Skin Appearance Model The skin is translucent Light is scattered through multiple layers, then reflected from a sub-layer Bidirectional Reflectance Distribution Function is often used

Bidirectional Reflectance Distribution Function(BRDF) A 4-dimensional function that defines how light is reflected at an opaque surface Input: incoming light direction wi Outgoing direction wo Output: Ratio of reflected radiance along wo to irradiance incident along wi

Facial Motion Expressions involving Speeches involving All front facial mesh Eye ball orientation Speeches involving Lip and jaw Tongue The combination of expressions and speeches

Facial Acting Code System Originally developed by Ekman and Friessen in 70s Based on the physiology of facial muscles A common standard to systematically categorize the physical expression of emotions

FACS – The Codes for Action Units Action Descriptors Inner Brow Raiser -- Frontalis (pars medialis) 2 Outer Brow Raiser -- Frontalis (pars lateralis) 4 Brow Lowerer -- Corrugator supercilii, Depressor supercilii 5 Upper Lid Raiser -- Levator palpebrae superioris 6 Cheek Raiser -- Orbicularis oculi (pars orbitalis) 7 Lid Tightener -- Orbicularis oculi (pars palpebralis) 9 Nose Wrinkler -- Levator labii superioris alaeque nasi 10 Upper Lip Raiser -- Levator labii superioris … 66 Action descriptors Demo

Lip-Sync by Viseme Morphing Use reference video to rotoscope the shape of the 3D model for key gestures Use a set of viseme corresponding to phonemes, then interpolate between viseme gestures Such approach often is not detailed enough for finer visual speech animation

Facial Motion Driven by Mocap Data Captures subtle details of facial expressions But, Mocap Data need be to cleaned for good rendering effect. Often this step is labor intensive. How can we make Mocap more flexible?

FACS-Viseme Solver Use a small set of Mocap data to synthesize a much larger range of facial motions Each piece of mocap data in the small recording set represent an Action Unit Use Action Unit analysis to decompose recorded Mocap data for easy post editing Use audio-visual mapping analysis to align mocap segment for new speech