1. University of Washington v The Hebrew University * Microsoft Research Synthesizing Realistic Facial Expressions from Photographs Frederic Pighin Jamie Hecker Dani Lischinski v David Salesin Richard Szeliski *

2. Realistic facial animation One of the most challenging problems in computer graphics: Facial surface is subtle Facial movements are complex Faces familiar

3. MusclesBones [Parke and Waters, 96]

4. Our approach 3 steps in synthesizing realistic faces: 1. Modeling from photographs 2. Creating new expressions 3. Animating expression changes

6. “surprised” “worried” “sad”

7. “neutral”“joy”

8. Previous work: Modeling Hand digitizing 3D scanning Image-based techniques

9. [Kleiser, 95] Hand digitizing

10. [Lee et al., 95] 3D scanning

11. [Parke, 74] Image-based techniques

12. Previous work: Animation Performance-driven animation Physically-based animation 2D morphing 3D morphing

13. [Williams, 90] Performance-driven animation

14. Physically-based animation [Lee et al., 95]

15. [Bregler et al., 97][Beier and Neeley, 92] 2D morphing

16. [Chen et al., 95] 3D morphing

17. 1. Modeling from photographs

18. Overview of modeling 1. Take multiple photographs of a person 2. Establish corresponding feature points 3. Recover 3D points and camera parameters 4. Deform generic face model to fit points 5. Extract textures from photos

19. Animating expression changes Varying the proportion of different expressions over time creates animation

20. Pose recovery Solve for: 3D locations of feature points Camera orientation Camera position Focal length Robust estimation by linear least squares

22. Mesh deformation Involves two steps: Compute displacement of feature points Apply scattered data interpolant

23. Generic modelDisplacementDeformed model

24. Texture map extraction The color at each point is a weighted combination of the colors in the photos Texture can be: View-independent View-dependent

26. View-independentView-dependent

27. 2. Creating new expressions

28. Creating new expressions In addition to global blending we can use: Regional blending Painterly interface

29. “neutral” “happy” “fake smile” + X X

30. Design of a drunken smile

31. 3. Animating expression changes