Facial Animation Wilson Chang Paul Salmon April 9, 1999 Computer Animation University of Wisconsin-Madison

Papers Used §Bregler C.,Covell M.,Slaney M., Video Rewrite: Driving Visual Speech with Audio. In SIGGRAPH 97 Conference Proceedings. ACM SIGGRAPH, August 1997 §Guenter B.,Grimm C.,Wood D., Malvar H., Pighin F, Making Faces. In SIGGRAPH 98 Conference Proceedings. ACM SIGGRAPH, July 1998 §Pighin F, Hecker J., Lischinski D., Szeliski R., Salesin D., Synthesizing Realistic Facial Expressions from Photographs. In SIGGRAPH §Waters K., A Muscle Model for Animating Three-Dimensional Facial Expression. In SIGGRAPH 1987.

Motivation §Creation of Virtual Characters §Teleconferencing & Video Compression §Simulated Movement §Facial Surgery Planning

Why facial animation is hard. §Humans are very good at reading expressions. §Any slight deviation from a “correct” expression will be immediately noticed. §Deep-rooted instinct.

Three general catagories §2-D Facial Model §3-D Facial Model §Muscular Model

2-D Facial Animation §Video Rewrite - modify and sync an actors’ lip motion to a new soundtrack. §Keyframe approach. §Uses vision techniques to track mouth movement.

Video Rewrite registration §Hand annotation of 26 images with 54 eigenpoints each. §Morph pairs to 351 images. §Learn eigenpoint model. §Warp images to standard reference plane. §Eigenpoint analysis.

Audio Analysis §Video Rewrite uses TIMIT speech database. §Triphones - emphasize middle. §“teapot” = /SIL-T-IY/, /T-IY-P/, /IY-P-AA/, /P-AA-T/, /AA-T-SIL/

Video Synthesis §Triphone Footage selection error =  D p + (1-  )D s §D p phoneme-context distance. §D s distance between lip shapes. l Overall Lip Width & Height l Inner Lip Height l Height of Visible Teeth

Finish Synthesis §Compress and Stretch video. §Align and blend mouth to face.

Results §Good Sync and natural articulation. §Missing Triphones result in unnatural speech

Making Faces §Motion capture. §3D mesh via Cyberware Laser scanner. §Deformed by l Position of 128 Dots Manual identification - 1st frame Tracked by vision techniques §Texture Extraction l Dot removal. l Cylindrical map.

Synthesizing Realistic Facial Expressions from Photographs §3D facial models derived from photographs. §Smooth transitioning between model expressions. §Adaptation from one model to another.

Model Fitting §Generic 3D mesh model. §Pose Recovery - using multiple subject views: l Identify feature points. l Deduce camera pose. l Iteratively refine the generic face model.

Model Fitting §Scattered Data Interpolation: l Interpolate mesh between feature points. l Uses radial basis functions. §Correspondence based shape refinement: l Use less accurate correspondences. l Polylines for eyebrows, eyelids, lips, etc. l Not used in pose processing due to error.

Texture Extraction §View independent vs View dependent. §Weight maps- bias selection of original photograph: l Self-occlusion. l Smoothness. l Positional certainty. l View similarity.

View Dependent Texture Extraction §Select best photographs. §Draw model for each photograph. §Blend rendered image. §Pros l adds detail. §Cons l sensitive to original photo. l More memory, slower.

View Independent Texture Extraction §Blend photographs to form single texture. l Map onto virtual cylinder.

View Independent Texture Extraction §Blurry

Special Case Textures §Fine Detail - hair. §Occlusion - eyes, teeth. §Intricate Projection - ears. §Shadowing - eyes, teeth §Solutions l Use photo with highest visibility. l Simulate shadowing

Expression Morphing §Simplified by common mesh. §Linearly interpolated vertices. §Blend result of rendering with each texture. §Synthesize new expressions via: l Global blend. l Regional blend. l Painterly interface.

Results §Smooth transitioned expressions:

Results §Applied transitions to different human subject:

Our conclusions §Good results between models. §Relatively inexpensive equipment. §Notable manual processing.

Muscular Modeling §Easy generalized across models. §22 muscle groups §Facial Action Coding System (Ekman, Wallace) - Action Unit parameterization

Anatomy

Skin as Mesh §Nodal mobility l Tensile Strength of skin l Proximity to muscle attachment l Depth of tissue & proximity to bone l Elasticity & interaction with other muscles §Network of springs l p = F/k

Mesh expression examples

Muscle types modeled §Linear/parallel muscles §Sphincter muscles

Linear/parallel muscles

Sphincter muscles

Animating §Not in paper §Build a library §Abstract language §Keyframe