1. SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: http://cs.nyu.edu/~bregler/sig-course-06-face/

2. SIGGRAPH Course 30: Performance-Driven Facial Animation Section: Markerless Face Capture and Automatic Model Construction Part 1: Chris Bregler, NYU

3. Markerless Face Capture

4. Markerless Face Capture - Overview - Single / Multi Camera Input 2D / 3D Output Real-time / Off-line Interactive-Refinement / Face Dependent / Independent Make-up / Natural Flow / Contour / Texture / Local / Global Features Hand Crafted / Data Driven Linear / Nonlinear Models / Tracking

5. Markerless Face Capture – History – Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models. Single Camera Input 2D Output Off-line Interactive-Refinement Make-up Contour / Local Features Hand Crafted Linear Models / Tracking

6. Tracking = Error Minimization Err(u,v) =  || I(x,y) – J(x+u, y+v) ||

7. Tracking = Error Minimization In general: ambiguous using local features

8. Tracking = Error Minimization Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

9. Tracking = Error Minimization Error = Feature Error + Model Error

10. Tracking = Error Minimization Error = Optical Flow + Model Error

11. - Optical Flow (Lucas-Kanade) Intensity x v ? i I(x ) - J(x + v ) iii 2 I (x ) - I(x ) v i t i 2  i linearize I J E(V)

12. V - = E(V) V Model I (1) -  I(1) v t 1 I (2) -  I(2) v t 2 I (n) -  I(n) v t n... 2    Optical Flow + Model

13. V - = E(V) V Model I (1) -  I(1) v t 1 I (2) -  I(2) v t 2 I (n) -  I(n) v t n... 2    V = M (   ) Optical Flow + Model

14. V - V Model Optical Flow + linearized Model V = M  2 Z + H V 2 Z + C 

15. Optical Flow + Hand-Crafted Model DeCarlo, Metaxas, 1999Williams et a,l 2002

16. Optical Flow and PCA Eigen Tracking (Black and Jepson)

17. PCA over 2D texture and contours Active Appearance Models (AAM): (Cootes et al)

18. PCA over 2D texture and contours

19. PCA over texture and 3D shape 3D Morphable Models (Blanz+Vetter 99)

20. Affine Flow and PCA

21. 3D Model Acquisition - Multi-view input: Pighin et al 98

22. Solution for Rigid 3D Acquisition Structure from Motion: - Tomasi-Kanade-92 Factorization 3D Pose 3D rigid Object

23. Acquisition without prior model ? No Model available ? Model too generic/specific ? Stock-Footage only in 2D ?

24. Solution based on Factorization - We want 3 things: - 3D non-rigid shape model - for each frame: - 3D Pose - non-rigid configuration (deformation) -> Tomasi-Kanade-92: W = P S Rank 3

25. Solution based on Factorization - We want 3 things: - 3D non-rigid shape model - for each frame: - 3D Pose - non-rigid configuration (deformation) -> PCA-based representations: W = P non-rigid S Rank K

26. 3D Shape Model Linear Interpolation between 3D Key-Shapes: S 1 S 2 S

27. Basis Shape Factorization Complete 2D Tracks or FlowMatrix-Rank <= 3*K

28. Nonrigid 3D Kinematics from point tracks -

29. - Nonrigid 3D Kinematics from dense flow

30. -

31. -

32. Motion Capture Modeling Synthesis Nonrigid 3D Kinematics from dense flow

33. Markerless Face Capture - Summary - Single / Multi Camera Input 2D / 3D Output Real-time / Off-line Interactive-Refinement / Face Dependent / Independent Make-up / Natural Flow / Contour / Texture / Local / Global Features Hand Crafted / Data Driven Linear / Nonlinear Models / Tracking