1. Helsinki University of Technology Laboratory of Computational Engineering Modeling facial expressions for Finnish talking head Michael Frydrych, LCE, 11.6.2004

2. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Finnish talking head Computer animated model of a talking person Synchronized A/V speech Model of emotional facial expressions

3. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 User interface of “old” talking head

4. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 What has been done with it? Studies in audiovisual speech perception Kiosk-interface at the University of Tampere Cultural activities Major role in play Kyberias at Kellariteatteri (2001) Talking Head

5. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Content Talking heads – why? Animation methods Controlling animation Making them speak Practicals -------------------------------------------------- Making the head smile Emotions –why? Practicals

6. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Why talking heads? Entertainment Information services Ananova, information kiosks Education services Learning foreign languages,… Agents in spoken dialogue systems nonverbal signals, comfort

7. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Tampere museums Tampere museums

8. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Aids in communication Speech is both heard and seen Improve intelligibility in noisy env. Aid for hearing impaired people Synface

9. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Synface (telephone -> animated face) Figure by KTH Stockholm

10. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 … applications Language training speech training for profoundly deaf Diagnostics and therapy EU: VEPSY, VREPAR (assess and treat naxiety disorders and specific phobia)

11. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Audiovisual speech integration = combining auditory and visual percepts into a single speech percept Strength of integration is demonstrated by McGurk- effect: combining sound /pa/ to a face ”telling” /ka/, speech percept is often /ta/ (McGurk & MacDonald, 1976, Nature) /pa/ /ka/ /ta/

12. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Result:Computer animated talking face improves intelligibility of auditory speech A study in audio-visual speech perception

13. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Psychophysical and psychophysiological experiments Audiovisual speech perception Emotion research … Benefits Natural stimuli may contain unwanted features Full controllability Quick creation of stimuli … application in research

14. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Bulding on realism Realism: 1) Objective  topography, animation, texture, synchronization,... 2) Subjective (communication)  Audio-visual speech  Facial expressions, nonverbal behavior (prosody, eye movements) Evaluation: Objective  Subjective

15. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Making the head speak Issues: Voice - speech synthesizer Animation – parameterization Synchronization

16. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Acoustic Speech Generation Based on Festival platform. Developed at The Centre for Speech Technology Research, University of Edinburg, England. Scheme programming language, allows to program behaviour Finnish voice, prosody, expansion (numerals, etc.) Department of Phonetics, University of Helsinki Issues: production of articulatory parameters, synchronization

17. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Animation methods - representation Polygonal Keyframing libraries of postures, interpolation Parametric deformations deformations are grouped under parameters meaningful to the animator Muscle-Based deformations Interactive deformations numerous control points, deformation propagation Free Form deformations deformation associated with a deformation box

18. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Splines Implicit surfaces Physics-based models Physical models of the skin Volume preservation Deformations by inducing forces

19. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Hooks to data Need the geometry of faces Rendering properties Deformation of facial expression or speech How? 2D and 3D techniques

20. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 3D Input 3D digitizer is the most direct way, fairly automatic (Optotrack) 3D trackers – digitizing of projected/marked mesh, rather manual CT (Computer Tomography) and MRI (Magnetic Resonance Imaging) and … 3D modeling programs

21. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 2D Input Photogrammetry Two images of an object are taken from different viewpoints, corresponding points are found The 3D shape of faces can be determined from a single 2D image after projecting of regular pattern Generic facial model is prepared and transformed to “match” a photograph 3rd dimension can be approximated by acquiring face model (set priors) and Bayesian inference

22. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Texture mapping

23. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 break

24. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Data for articulation and expressions Keyframing -> expression libraries Real-time/performance data Parameterization Articulatory parameters – jaw opening, lip rounding, lip protrusion, … Facial expressions – FACS Statistical models from expression libraries or real-time data

25. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Statistical parameterization Parameterized model learned from 3D performance data (Reveret) Figure by ISCP Grenoble

26. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 … three control parameters Figure by ISCP Grenoble

27. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 … and the results Jaw Rounding Opening Raising Figure by ISCP Grenoble

28. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Video by ISCP Grenoble

29. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Finnish talking head Audiovisual database Using MaxReflex 3D optical tracker (at Linköping Univ.) Multiple IR cameras, reflexive markers  reconstruction from stereo Coarticulation, lips, visual prosody

30. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Point-lights positions

31. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Demo – live recording at Linköping

32. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 How to create “visemes” ?

33. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Demo – reconstructed motion 10 fps 40 fps

34. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 Figure by ISCP Grenoble

35. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004

36. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004

37. Laboratory of Computational Engineering Michael Frydrych, 11.6.2004 End of 1st part