Facial Type, Expression, and Viseme Generation Josh McCoy, James Skorupski, and Jerry Yee

Introduction  Virtual Human Faces –Hard to generate –Easy to criticize  Motivation –Movies –Games  Problems –Hand-made models take time –Physically-based models look weird

Contribution  Data-driven facial face generation  User-guided categorization  Real-time pose generation from data

Related Work: Face Retargeting  V. Blanz, C. Basso, and T. Vetter  Reanimating Faces in Images and Video. –Use a morphable model to synthesize a 3D face of the 2D image. –Capture 35 scans of static face poses (expressions and visemes in neutral expression) from a source actor. –Find dense point-to-point correspondences –Retarget facial movements to the 3D face. –Render the 3D face back into the 2D image.

Related Work: Face Retargeting  Problems –Does not generate new expressions that are not in the source data set. –Does not combine and retarget expressions and visemes together.

Related Work: Bilinear Model  E. Chuang and C. Bregler  Mood Swings: Expressive Speech Animation –Capture a video of an actor reading script under three different expressions (happy, angry, neutral) –Create a bilinear model, factoring expressions and visemes into two separate components. –Synthesize new facial movements with any expression and viseme.

Related Work: Bilinear Model  Problems –Requires a full Cartesian product of facial expressions and visemes. –Does not generate new expressions that are not in the source data set. –Does not change the facial characteristics (identity).  Pres Videos\Jerry\moodswings.mov Pres Videos\Jerry\moodswings.mov Pres Videos\Jerry\moodswings.mov

Related Work: Multilinear Model  D. Vlasic, M. Brand, H. Pfister, & J. Popovic  Face Transfer with Multilinear Models –Capture videos of 16 actors, each performing 5 visemes under 5 different expressions. –Create a multilinear model, factoring expressions, visemes, and identity into three separate components. –Synthesize new facial movements with any expression, viseme, and identity

Related Work: Multilinear Model  Problems –Requires a full Cartesian product of facial expressions, visemes, and identity. –Limitations in the missing data imputation process. –Does not generate new expressions that are not in the source data set.  Pres Videos\Jerry\vlasic-2005-ftm-sing.mp4 Pres Videos\Jerry\vlasic-2005-ftm-sing.mp4 Pres Videos\Jerry\vlasic-2005-ftm-sing.mp4

Methods  Acquire and Categorize  Learn  Generate

Acquire and Categorize  Three data sets are needed to fill the model space –Set of many neutral faces –Set of one face in many poses –Set of Visemes with reference face  Vertex Correspondence  User “rates” attributes of each face  Video

Acquire and Categorize

Learn Expression deformation Viseme deformation Type deformation Reference Face  Analyze each triangle and transform type separately

Learn Low-dimensional subspace (PCA) polygons individuals  Compare each pose to reference face  Principle Component Analysis (PCA) –Apply to each axis of variation –Analyze transformation of every face in mesh  Infer variation of single attribute from combination of many

Generate  Same sliders as categorization UI  Generate any combination of attributes  Runs in real-time

Results

Conclusion  Realistic face poses from real-world basis data  Arbitrary faces from sparse data set  Future Work –Use high res data to drive low res morphing –Incorporate more biologically accurate face model