1. Use and Re-use of Facial Motion Capture M. Sanchez, J. Edge, S. King and S. Maddock

2. 0. Motivation Facial Animation in the Computer Graphics industry is a mainly human-driven process, requiring a lot of time and resources Other aspects of Character Animation, such as Skeletal Animation, have been successfully automated by the use of human motion capture technology Applying the same approach to Facial Animation could sensibly reduce the workload involved, and would lead to a corresponding increase in the realism of Facial Animation However, the differences in the nature of the captured content requires the development of specific techniques to make Facial Motion Capture applicable

3. 1. System overview

4. 1.1 Facial Motion Capture System input: Problems: It doesn’t analyze the motion over the full geometry of the face (just at the markers); The captured face may not correspond with the face to be animated; Noise and missing data. 3D tracking of a predefined set of markers attached to the skin surface

5. 2. Animating the skin

6. How to reconstruct the deformation of the complete skin surface when only the movement of a few points is known? Direct interpolation of the movement of the markers over the skin (Kshirsagar et al. 00, Pasquariello and Pelachaud. 01) Dirichlet Free Form Deformations (Escher et al. 98) Radial Basis Functions (Fidaleo, Noh et al. 00) We use Planar Bones (Sanchez and Maddock 03)

7. 2.1. Planar Bones Extended formulation of Surface-oriented Free Form Deformations (Kokkevis and Singh, 00) Define a parameterisation of every vertex over a control mesh, used to drive a deformation Preserve a “distance relation” between the control structure and the deformed geometry Replicate proper transmission of motion across the skin without the need of surface metrics

8. 4. Retargeting Facial Motion Capture The dimensions of the face are different, and so is the scale of the motion; Conventional full-body Motion Capture retargeting is not applicable; The correspondence between different faces is highly non-linear.

9. 4. Building a mapping between faces

10. Retargeting FMC requires: 1) Adapting the Planar Bones control mesh to the target geometry; 2) Scaling the motion of the markers according to the change of physiognomy. Ideally, both processes should be performed automatically In practical terms, we need some user input.

11. 4.1. Fitting the control mesh

12. 3 stages: 1) Radial Basis Functions - produce initial approximation 2) Cylindrical projection – Constraining the markers to the target surface 3) Mesh fitting – Blind constrained optimisation Additional parameters of the Planar Bones method are also retargeted by this process: Extents of the deformation (affection volume) Discontinuity maps

13. 4.1. Fitting the control mesh RBF stage: Build an interpolant of the offset between the markers labelled on the target face and their equivalents in the reference model Evaluate this function at the non-hand-labelled markers to obtain their image on the target geometry Mesh fitting stage: Finding the optimal distribution of control points that: a) Minimizes the “distance” between the reference face deformed by the retargeted control mesh and the target geometry b) Preserves the general shape represented by a deformation energy function c) Stays on the surface of the target face (enforced through the cylindrical mapping) Simplex downhill method

14. 4.2. Scaling Facial Motion

15. The two faces are labelled with the same markers After fitting the control mesh We can extend this mapping to the whole space the faces are given in By interpolating the initial displacement at every control point using RBFs This interpolant is used to compute the mapping on the target space of the captured markers during the animation

16. 4.2. Scaling Facial Motion An evaluation the 2-norm of the metric tensor of the mapping shows how infinitesimal displacements are scaled green: positive scaling (>1) blue: negative scaling The Planar Bones algorithm computes the final deformation, driven by the retargeted control mesh This procedure implicitly scales the movement of the markers in the target space, according to the initial correspondence that is given as reference.

17. 5. Processing Motion Capture Input Limitations in the marker tracking technology lead to deficiencies in the captured data:

18. 6. Sample results: lip tracking

20. 7. Conclusions and future work We have introduced a novel method for the retargeting and animation of faces from motion capture data Current research: Provide a better model for the tracking of the inner contour of the lips: Marker-less image processing of the video capture Physical model using a mass-spring system attached to the outer contour Introduce furrowing and wrinkling in the skin animation A posteriori deformation analysis on the deformation induced by Planar Bones

21. Questions? m.sanchez @dcs.shef.ac.uk j.edge @dcs.shef.ac.uk

22. Additional samples: chorus