Muscle Based Facial Animation Jason Jerald April 7, 2004

Overview History Types of Facial Muscle Models Muscle Vectors (Waters 1987) Improvements Breton, Bouville and Pele 2001 Bui and Nijholt 2003

Facial Muscle History 19th century - physiologist Duchenne applied electrical currents to freshly guillotined heads to observe facial contortions Later applied same technique to old inmates of alms houses to create artificial expressions Recorded with photography 1977 - Psychologists Ekman and Friesden Created the Facial Action Coding System (FACS) Notational-based environment that determines emotional states from visible facial distortion Individual muscles are described as Action Units (AU) This work is commonly used in computer facial animation

Facial Muscle History 1980 - Platt published masters thesis on a physically based muscle–controlled facial expression model 1987 - Waters published the seminal paper on muscle based facial animation using muscle vectors 21st century – improvements on Waters model

Overview History Types of Facial Muscle Models Muscle Vectors (Waters 1987) Improvements Breton, Bouville and Pele 2001 Bui and Nijholt 2003

Various Muscle Models Free form deformations Spline psuedo muscles Deforms objects by manipulating control points arranged in a 3d cubic lattice Surface regions corresponding to anatomical descriptions of the muscle actions are defined Displacing control point is analogous to actuating a physically modeled muscle More Intuitive than vector representations but cannot model furrows, bulges, and wrinkles. Spline psuedo muscles Deforming facial mesh in muscle-like fashion Ignores underlying anatomy Supports smooth and flexible deformations Hierarchical splines allow more detail in specified regions

Various Muscle Models Mass-spring methods Layered spring meshes Forces applied to elastic meshes through muscle arcs Muscles represented as collections of functional blocks Action units created by applying muscle forces to deform the spring network Layered spring meshes models skin, fatty tissue, and muscle tied to bones Spring elements connect each mesh node and each layer Realistic but computationally expensive Vector representations What this talk focuses upon

Overview History Types of Facial Muscle Models Muscle Vectors (Waters 1987) Improvements Breton, Bouville and Pele 2001 Bui and Nijholt 2003

Muscle Vector Model What is needed? Few dynamic parameters that emulate the primary characteristics of facial expression Linear/parallel muscles that pull and sphincter muscles that squeeze Factors determining nodal mobility are Tensile strength of the muscle and skin Proximity to the muscle node of attachment Proximity to the bone Elastic bounds of the relaxed tissue Interaction of other muscles

Muscle Vectors Models the actions of muscles upon skin Each muscle has a zone of influence A muscle includes vector field direction, an origin, and an insertion point

Advantages / Disadvantages Independent of facial mesh (facial mesh can be exchanged) Compact representation Expression parameters can control groups of muscles Fast Disadvantages Positioning of muscles can be time consuming Does not take curvature into account Artifacts when a mesh vertex is under the influence of multiple muscle actions

Muscle Vector Parameters Muscle attached at two points Point of Attachment A – the root of the muscle attached to the bone Point of insertion I into the flesh Muscle can therefore be considered as the vector AI flesh Muscle bone With no contraction the points of attachment and insertion do not move and the muscle vector maintains its length Acts like a magnet attracting all the vertices within its zone of influence. The skin contracts more near the muscle.

Muscle Vector Parameters V is the mesh vertex Opening angles β is the opening angle α is the maximum angular limit Muscle contraction is faded as β raises to α Radial distances as a proportion of |AI| S is where the muscle influence starts to fade E is where the muscle influence ends Vertices are faded if they are in the band defined by S and E Muscle

Equations C is the contraction factor (between 0 and 1) is the fading coefficient related to the angular distance between AV and AI is the fading coefficient related to the radial distance between V and S if V is in the fading band SE

Muscle Results

Sphincter / Mouth Muscle Waters models the mouth with a sphincter muscle Described from a single point around which the surface contracts as if drawn together like a string bag Longitudinal and vertical axii allow elliptical shape Sphincter muscle Elliptical Sphincter muscle

Muscle sets

Facial Action Coding System (FACS) Developed by Psychologists Ekman and Friesden in 1977 FACS is Description of facial muscles and jaw/tongue derived from analysis of facial anatomy Notational-based system that determines emotional states from visible facial distortion Action Units (AU) correspond to muscle vectors

Waters Results

Overview History Types of Facial Muscle Models Muscle Vectors (Waters 1987) Improvements Breton, Bouville and Pele 2001 Bui and Nijholt 2003

Breton et al Non-muscle parametric animation mixed with muscle vectors (jaw, eyes, eyelids, neck) Opening of mouth with a muscular system requires distinction between lower and upper lips

The eyes Eyes and Eyelids Modeled as spheres and hemispheres Simple rotations Random blinking Gaze looking forward when speaking Random gaze direction when not speaking

Jaw and Neck Jaw Neck Single axis of rotation Lower lips not within jaw influence Neck Three axii of rotation Center of rotation is the center of the neck bounding box Vertices of the head are fully rotated Rotations of the neck linearly faded with distance Jaw boundary Neck boundary

Lips Distinction between upper and lower lips must be made in order to open mouth Distinction between upper and lower lips determined at load time

Breton et al 2001 results

Overview History Types of Facial Muscle Models Muscle Vectors (Waters 1987) Improvements Breton, Bouville and Pele 2001 Bui and Nijholt 2003

Bui et al Multiple muscle action artifacts removed by simulating parallelism Division into regions Wrinkles

Muscle action artifacts The problem Problem when a mesh vertex is under the influence of multiple muscle actions Muscle actions are independent Actual nodal displacement determined by a succession of muscle actions Unnatural results occur when a vertex is shifted outside the zone of influence of adjoining muscle vectors The solution Combining muscle contractions done by simulating parallelism For a vertex inside multiple muscles’ zone of influence, small units of contraction levels are applied until no more contraction to apply Step sizes of 20% of full contraction found to have good results

Region Division Allows easier rendering of special parts of the face such as lips and eyebrows Reduces artifacts generated by displacement of vertices in regions that are not affected by a muscles contraction

Muscle Action Artifacts Two muscles with no parallelism Two muscles with parallelism Three muscles with parallelism

Wrinkles Assume muscles lie parallel to the facial skin and heights of wrinkles are the same Height and number of wrinkles are predefined for each muscle To make wrinkles more visible use triangular flat shading at vertex where wrinkle starts

Bui et al results (2003) happy neutral Sad with close up of wrinkles surprise

Demo

Conclusion Waters is the basic model Additional tricks can be added to improve appearance Facial expressions can be defined by muscle groups while individual muscles can also be controlled. Simple yet effective and fast

References Breton, G., Bouville, C., and Pel, D. 2001. FaceEngine a 3d Facial Animation Engine for Real Time Applications. in Proceedings of 6th International Conference on 3D Web Technology, pp. 15-22. Bui, T. D., Heylen, D., and Nijholt, A. 2003. Improvements on a Simple Muscle-Based 3d Face for Realistic Facial Expressions. in Proceedings of 16th International Conference on Computer Animation and Social Agents, pp. 33-40. Noh, J.-Y. and Neumann, U. 1998. A Survey of Facial Modeling and Animation Techniques. USC Technical Report No. 99-705. Parke, F. I. and Waters, K. (1996) "Chapter 7 Skin and Muscle-Based Facial Animation." In Computer Facial Animation, pp. 223-257. Waters, K. 1987. A Muscle Model for Animating Three-Dimensional Facial Expression. in Proceedings of 14th Annual Conference on Computer Graphics and Interactive Techniques, pp. 17-24.