1. SIGGRAPH Course 30: Performance-Driven Facial Animation Section: Marker-less Face Capture and Automatic Model Construction Part 1: Chris Bregler, NYU Part 2: Li Zhang, Columbia University

2. Face Tracking Approaches Marker-based hardware motion capture systems Tom Tolles (House of Moves) presentation 9:00 (earlier) Parag Havaldar (Sony Pictures Imageworks) presentation at 2:15 pm

3. Marker-based Face Capture:

4. Marker-less Face Capture:

5. Early Computer Face Capture Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models. Single Camera Input 2D Output Off-line Interactive-Refinement Make-up Contour / Local Features Hand Crafted Linear Models / Tracking

6. Disney:

7. Early “Markerless Facecapture” Disney: Step-Mother  Eleanor Audley

8. Early Computer Face Capture Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models. Single Camera Input 2D Output Off-line Interactive-Refinement Make-up Contour / Local Features Hand Crafted Linear Models / Tracking

9. Markerless Face Capture - Overview - Single / Multi Camera Input 2D / 3D Output Off-line / Real-time Interactive-Refinement / Face Dependent / Independent Make-up / Natural Flow / Contour / Texture / Local / Global Features Hand Crafted / Data Driven Linear / Nonlinear Models / Tracking

10. Common Framework Error = Feature Error + Model Error Tracking = Error Minimization

11. Difference: Error = Feature Error + Model Error Tracking = Error Minimization

12. Difference: Error = Feature Error + Model Error Tracking = Error Minimization

13. Difference: Error = Feature Error + Model Error Tracking = Error Minimization

14. Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

15. Tracking = Error Minimization Error = Feature Error + Model Error

16. Tracking = Error Minimization Error = Optical Flow + Model Error Most general feature:

17. Tracking = Error Minimization Err(u,v) =  || I(x,y) – J(x+u, y+v) ||

18. - Basics in Optical Flow: Lucas-Kanade 1D Image Intensity x u ? FG Linearization: Spatial GradientTemporal Gradient

19. Spatial GradientTemporal Gradient ROI (u,v) FG Lucas-Kanade: 2D Image

20. Minimize E(u,v): => C DC D Lucas-Kanade: Error Minimization: 2D Image

21. Marker-less Face Capture: In general: ambiguous using local features

22. - = E(V) Optical Flow I (1) -  I(1) v t 1 I (2) -  I(2) v t 2 I (n) -  I(n) v t n... 2

23. V - = E(V) Optical Flow I (1) -  I(1) v t 1 I (2) -  I(2) v t 2 I (n) -  I(n) v t n... 2

24. V - = E(V) V Model Optical Flow + Model I (1) -  I(1) v t 1 I (2) -  I(2) v t 2 I (n) -  I(n) v t n... 2

25. V - = E(V) V Model I (1) -  I(1) v t 1 I (2) -  I(2) v t 2 I (n) -  I(n) v t n... 2 V = M (   ) Optical Flow + Model

26. V - V Model Optical Flow + linearized Model V = M  2 Z + H V 2 Z + C 

27. Optical Flow + 3D Model DeCarlo, Metaxas, 1999Eisert et al 2003

28. Optical Flow + MPEG4 Model --> MediaPlayer (Eisert et al)

29. High-End Production: Optical Flow + 3D Model Disney Gemeni-Project Williams et al 2002 EA Universal Capture Borshukov et al 2002-2006

30. More “forgiving” Error Norm - Faces change appearance L2 D

31. More “forgiving” Error Norm - L2 Norm vs Robust Norm L2robust DD

32. - Robust Error with EM layers I (1) -  I(1) v t 1 I (2) -  I(2) v t 2 I (n) -  I(n) v t n... 2

33. - Robust Error with EM layers I (1) -  I(1) v t 1 I (2) -  I(2) v t 2 I (n) -  I(n) v t n... 2 0.1 0.2 0.9

34. - Lucas-Kanade + changing Appearance FG Learned PCA:

35. Optical Flow and PCA Eigen Tracking (Black and Jepson)

36. 2D texture and contours + PCA Active Appearance Models (AAM): (Cootes et al)

37. 2D texture and mesh + PCA

38. Lucas-Kanade + Apearance Models Lucas-Kanade AAMs: (Baker & Matthews)

39. Affine Flow + PCA + Robust Norm Disney: Gemeni-Project

40. Solution based on Factorization - We want 3 things: - 3D non-rigid shape model - for each frame: - 3D Pose - non-rigid configuration (deformation) -> Tomasi-Kanade-92: W = P S Rank 3

41. Solution based on Factorization - We want 3 things: - 3D non-rigid shape model - for each frame: - 3D Pose - non-rigid configuration (deformation) -> PCA-based representations: W = P non-rigid S Rank K

42. Space-Time Factorization Complete 2D Tracks or FlowMatrix-Rank <= 3*K Nonrigid flow or Markerset -> “Rigid Stabilization + Blendshapes”

43. Space-Time Factorization Irani, 1999 Bregler, Hertzmann, Biermann, 2000 Torresani, Yang, Alexander, Bregler, 2001 Brand, 2001 Xiao, Kanade, 2004 Torresani, Hertzmann, 2004

44. From Pixels to 3D Blend Shapes (Torresani et al 01,02)

45. Trajectory Constraints t=2 t=1 t=F.... =.... 3D positions of point i for the K modes of deformation fra mes Q’Q’ mimi w i : full trajectory Space-Time Tracking (Torresani Bregler 2002)

46. Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al) – Rank Based Tracking – 3D Basis Shapes – Probabilistic Tracking / Models – Occlusion – Dynamical Systems Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al) – Rank Based Tracking – 3D Basis Shapes – Probabilistic Tracking / Models – Occlusion – Dynamical Systems From Pixels to 3D Blend Shapes (Torresani et al 01,02)

47. Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al) – Rank Based Tracking – 3D Basis Shapes – Probabilistic Tracking / Models – Occlusion – Dynamical Systems Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al) – Rank Based Tracking – 3D Basis Shapes – Probabilistic Tracking / Models – Occlusion – Dynamical Systems p ( I(pj,t ) | “point pj,t is visible”) = N ( I(pj,t )| µj ;  2 ) p ( I(pj,t ) | “pixel pj,t is an outlier”) = c From Pixels to 3D Blend Shapes (Torresani et al 01,02) z t = A * z t-1 + nt

48. From Pixels to 3D Blend Shapes (Torresani et al 01,02)

50. Disney Gemeni Project

51. Markerless Face Capture - Summary - Single / Multi Camera Input 2D / 3D Output Real-time / Off-line Interactive-Refinement / Face Dependent / Independent Make-up / Natural Flow / Contour / Texture / Local / Global Features Hand Crafted / Data Driven Linear / Nonlinear Models / Tracking