1. Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

2. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-42 Outline  Definition of the Problem  Introducing MPEG-4 FBA  Facial Structure Overview  Numerical Simulation  Results and Conclusions

3. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-43 Definition of the Problem MPEG-4 FBA standard mainly focuses on networking capabilities for sending/ receiving animations. It allows employing model-based coding of human-like faces in a systematic manner. It does not provide any solution on how to achieve the movements of the face/body.

4. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-44 Definition of the Problem We use some already developed techniques to build a face model conforming to the anatomical structure of the human head. Our contribution is in devising a fast and automatic building process using MPEG-4. The resulting virtual talking head is used to perform general facial animation coming from an encoded MPEG-4 FAB stream.

5. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-45 Definition of the Problem MPEG-4 FBA Encoded Stream MPEG-4 Player Animatable Face Model Modeling Process 3D Mesh + Info MPEG-4 FBA Output

6. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-46 Outline  Definition of the Problem  Introducing MPEG-4 FBA  Facial Structure Overview  Numerical Simulation  Results and Conclusions

7. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-47 Introducing MPEG-4 FBA MPEG-4 Standard dealing with efficient transmission of multimedia on the net FBA Efficient coding of shape and animation of human-like faces and bodies FDP Control points used to define the shape of a proprietary face model FAP Set of animation parameters suitable to define facial movements

8. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-48 MPEG-4 Facial Definition Pts

9. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-49 MPEG-4 Facial Animation Pts 68 low-level FAPs used to express basic actions that the face can perform 20 high-level FAPs used to express more complex movements like expressions and visemes

10. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-410 Outline  Definition of the Problem  Introducing MPEG-4 FBA  Facial Structure Overview  Numerical Simulation  Results and Conclusions

11. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-411 Facial Structure Overview From a VRML modelto a mass-spring structure Skin structure { Masses [or nodes] from VRML vertices Springs from VRML topology Acting on the skin model through muscle actions, movement is achieved

12. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-412 Skin is composed by 3 layers Skin – Prism Element subdivided in little triangular prisms

13. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-413 Skin – Prism Element

14. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-414 From FDPs Muscles – Mapping muscles are mapped automatically Green points are where the muscle is attached to the skull Red points are where the muscle is inserted in the skin Two kinds of muscle: linear and sphincter

15. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-415 Muscle Mapping - Examples

16. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-416 Outline  Definition of the Problem  Introducing MPEG-4 FBA  Facial Structure Overview  Numerical Simulation  Results and Conclusions

17. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-417 Semi-Implicit Integrations A ordinary differential equation system governs the movement of the spring-mass network. Explicit methods are simple but unstable if applied to stiff systems [e.g. Euler steps]. Implicit methods are precise and stable but computationally expensive [e.g. Backward Euler]. Semi-Implicit Integration = implicit integration approximated with some explicit steps.

18. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-418 Skin nodes can be displaced by muscular force and by the elastic wave coming from the muscle movement. Adaptive Displacement

19. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-419 Outline  Definition of the Problem  Introducing MPEG-4 FBA  Facial Structure Overview  Numerical Simulation  Results and Conclusions

20. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-420 Achieving Facial Animation MPEG-4 FBA data stream MPEG-4 player (PC, PDA, mobile phones) Animatable Face Model anAtomic Audio FDPs Static Model Output One action set for each MPEG-4 FAP

21. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-421

22. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-422 Conclusions - Pros This method generates realistic facial animation encoded in a MPEG-4 FAP stream of general face meshes using only a standard PC. It is automatic and fast. It can be used to animate talking heads in MPEG-4 compliant applications.

23. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-423 Conclusions - Cons Animation becomes inaccurate if the polygonal resolution of the face mesh is not enough fine (too few springs!) and/or the mesh topology is not regular (e.g. not simmetric). Solution: Motion Cloning

24. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-424 Main References Y. Lee, D. Terzopoulos and K. Waters. ”Constructing Physics-Based Facial Models of Individuals” K. Waters. ”A Muscle Model for Animating Three- Dimensional Facial Expressions”. D. Baraff and A. Witkin. ”Large Steps in Cloth Simulation”. I.S. Pandzic and R. Forchheimer, editors. ”MPEG-4 Facial Animation – The Standard, Implementation and Applications”.

25. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-425 Acknowledgements Image Coding Group, Department of Electrical Engineering, Linköping University, Sweden Visage Technologies AB [www.visagetechnologies.com]

26. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-426

27. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-427 Skull when otherwise is the net force on node i is the nodal normal of node i

28. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-428 Skull - Example

29. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-429 Skull – Lower Jaw

30. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-430 Muscle Mapping - Examples

31. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-431 Linear Muscles

32. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-432 Linear Muscles

33. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-433 Sphincter Muscles

34. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-434 Semi-Implicit Integration Starting from the differential system governing the net: Equations for one mass m i :

35. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-435 Semi-Implicit Integration Implicit method (Lagrangian), it is obtained: Non linear functions  very expensive extract x i n+1 for each time step x i n+1 is approximated using an Euler step (explicit method): Trade-off between stability of the implicit and speed of the explicit

36. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-436 Adaptive Displacement

37. 09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-437