2. Unsupervised Learning for Speech Motion Editing Eurographics/SIGGRAPH Symposium on Computer Animation (2003) Yong Cao 1,2 Petros Faloutsos 1 Frederic Pighin 2 University of California, Los Angeles 1 Institute for Creative Technologies, University of Southern California 2

3. Problem ■ Motion Capture is convenient but lacks flexibility ■ Problem: How to extract the semantics of the data for intuitive motion editing?

4. Related Work 1. Face motion synthesis ■ Physics-based face model Lee, Terzopoulos, Water ( SIGGRAPH 1995) Kähler, Haber, Seidel (Graphics Interface 2001) ■ Speech motion synthesis Bregler, Covell, Slaney (SIGGRAPH 1997) Brand (SIGGRAPH 1999) Ezzat, Pentland, Poggio (SIGGRAPH 2002) 2. Separation of style and content Brand, Hertzmann (SIGGRAPH 2000) Chuang, Deshpande, Bregler (Pacific Graphics 2002)

5. Our Contribution ■ New statistical representation of facial motion Decomposition into style and content Intuitive editing operations

6. Our Contribution Original Neutral MotionEdited Sad Motion

7. Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components ■ Motion editing

8. New representation or

9. Independent Component Analysis (ICA) ■ ICA Linear transformation Components are independent ■ Example: Blind Source Separation ICA From: “http://www.cis.hut.fi/projects/ica/fastica/”

10. Independent Component Analysis (ICA) ■ Statistical technique ■ Linear transformation ■ Components are maximally independent

11. ■ Preprocessing (PCA) ■ Centering ■ Whitening Decomposition: Reconstruction: ■ ICA decomposition Steps of ICA

12. ICA vs. PCA ■ The components of PCA are uncorrelated ■ The components of ICA are independent

13. Components of PCA Can NOT separate Mouth motion and Eye-brow motion

14. Components of ICA Mouth motion and Eye-brow motion being separated

15. Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components ■ Motion editing

16. Speech motion of 113 sentences in 5 emotion moods: Frustrated 18 sentences Happy 18 sentences Neutral 17 sentences Sad 30 sentences Angry 30 sentences Speech motion Dataset Each motion: 109 motion capture markers 2 – 4 seconds

17. Facial Motion and ………… Facial motion Components in ICA space DecompositionReconstruction

18. Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components ■ Motion editing

19. Interpretation of independent components ■ Qualitatively ■ Quantitatively ■ Goal: Find the semantics of each component Classify each component into: Style (emotion) Content (speech) ■ Methodology

20. Qualitatively Style (emotion)Content (speech) changing

21. Style: Emotion Same speech, different emotion ………… HappyFrustrated Quantitatively

22. ■ Eyelid motion■ Eyebrow motion■ Mouth motion Speech Content Grouping of motion markers

23. Content: speech related motion Step1: Using each independent component to reconstruct facial motion Reconstruct ………… 0 0 0 0

24. Step2: Compare according to certain region Content: speech related motion

26. Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantic meaning of components ■ Motion editing

27. Motion Editing with ICA ■ Edit the motion in intuitive ways ■ Translate ■ Copy and Replace ■ Copy and Add

28. Results ■ Changing emotional state by translating

31. Conclusion ■ New statistical representation of facial motion Decomposition into content and style Intuitive editing operations

32. The End Thanks to Wen Tien for his help on this paper, Christos Faloutsos for useful discussions, and Brian Carpenter for his excellent performance. Thanks to the USC School of Cinema – Television and House of Moves for motion capture.