Vision-based Control of 3D Facial Animation Jin-xiang Chai Jing Xiao Jessica Hodgins Carnegie Mellon University

Our Goal Interactive avatar control Designing a rich set of realistic facial actions for a virtual character Providing intuitive and interactive control over these actions

+ High quality- Expensive - Intrusive - Noisy - Low resolution + Inexpensive + Non-intrusive Control InterfaceQuality Control Interface vs. Quality Vision-based animation Online motion capture

Our Idea Vision-based interface Motion capture database Interactive avatar control +

Motion capture Making faces [Guenter et al. 98] Expression Cloning [Noh and Neumann 01] Vision-based tracking for direct animation Physical markers [Williams 90] Edges [Terzopoulos and Waters 93, Lanitis et al. 97] Dense optical flow with 3D models [Essa et al. 96, Pighin et al. 99, DeCarlo et al. 00] Data-driven feature tracking [Gokturk et al. 01] Vision-based animation with blendshape Hand-drawn expression [Buck et al. 00] 3D model avatar model [FaceStation] Related Work

Video Analysis Avatar animation Preprocessed Motion Capture Data Expression Control and Animation Expression Retargeting Act out expressions System Overview Video Analysis

Video Anal y sis Vision-based tracking 3D Head Poses [Xiao et al. 2002] 2D facial features Video Analysis

Expression Control Parameters Extracting 15 expression control parameters from 2D tracking points Distance between two feature points Distance between a point and a line Orientation and center of the mouth Expression control signal t

Avatar animation Preprocessed Motion Capture Data Expression Control and Animation Expression Retargeting Act out expressions Video Analysis System Overview

Motion Capture Data Preprocessing 3D Poses Expression separation Expression control parameter extraction frames (10 minutes) including: 6 basic facial expressions typical everyday facial expressions speech data

Avatar animation Expression Control and Animation Expression Retargeting Act out expression Video Analysis Preprocessed Motion Capture Data System Overview

Expression Control 2D tracking data Vision-based interface Motion capture database 19*2 dofs Expression control parameters 15 dofs 76*3 dofs 3D motion data

Challenges Visual expression control signals are very noisy One to many mapping from expression control parameter space to 3D motion space Temporal coherence Control parameter space3D motion space 15 dofs76*3 dofs

Data-driven Dynamic Filtering Nearest Neighbor Search Noisy control signal Online PCA K=120 closest examples W = 0.33s 7 largest Eigen- curves (99.5 % energy) Filtered control signal Filter by eigen-curves

Expression Mapping Nearest Neighbor Search From expression control parameter space to 3D motion data space d1d1 d2d2 dKdK...  w(d 2 ) w(d K ) w(d 1 )... Filtered control signal Synthesized motion

Avatar animation Act out expression Video Analysis Preprocessed Motion Capture Data Expression Control and Animation System Overview Expression Retargeting

Expression Retarget Synthesized expressionAvatar expression

Expression Retarget xsxs xtxt Learn the surface mapping function using Radial Basis Functions such that x t =f(x s ) Transfer the motion vector by local Jacobian matrix Jf(x s ) by  x t =Jf(x s )  x s xsxs xtxt ? Run time computational cost depends on the number of vertices

Precompute Deformation Basis … T0T0 T1T1 T2T2 T3T3 T4T4 T5T5 … S0S0 S1S1 S2S2 S3S3 S4S4 S5S5 PCA Precompute deformation basis 25 source motion bases –99.5% energy 25 precomputed avatar motion bases

Target Motion Synthesis … … Synthesized expression Avatar expression 0,…. N S0S0 T0T0 S1S1 T1T1 T2T2 S2S2 iSiiSi iTiiTi Run time computational cost is O(N) N is the number of bases S3S3 T3T3 TNTN SNSN

Avatar animation Act out expression Video Analysis Preprocessed Motion Capture Data Expression Control and Animation System Overview Expression Retargeting

Results

Conclusions Developed a performance-based facial animation system for interactive expression control Tracking real-time facial movements in video Preprocessing the motion capture database Transforming low-quality 2D visual control signal to high quality 3D facial expression An efficient online expression retarget

Formal user study on the quality of the synthesized motion Controlling and animating 3D photorealistic facial expression Size of database Future Work