Helsinki University of Technology Laboratory of Computational Engineering Modeling facial expressions for Finnish talking head Michael Frydrych, LCE,

Laboratory of Computational Engineering Michael Frydrych, Finnish talking head Computer animated model of a talking person Synchronized A/V speech Model of emotional facial expressions

Laboratory of Computational Engineering Michael Frydrych, User interface of “old” talking head

Laboratory of Computational Engineering Michael Frydrych, What has been done with it? Studies in audiovisual speech perception Kiosk-interface at the University of Tampere Cultural activities Major role in play Kyberias at Kellariteatteri (2001) Talking Head

Laboratory of Computational Engineering Michael Frydrych, Content Talking heads – why? Animation methods Controlling animation Making them speak Practicals Making the head smile Emotions –why? Practicals

Laboratory of Computational Engineering Michael Frydrych, Why talking heads? Entertainment Information services Ananova, information kiosks Education services Learning foreign languages,… Agents in spoken dialogue systems nonverbal signals, comfort

Laboratory of Computational Engineering Michael Frydrych, Tampere museums Tampere museums

Laboratory of Computational Engineering Michael Frydrych, Aids in communication Speech is both heard and seen Improve intelligibility in noisy env. Aid for hearing impaired people Synface

Laboratory of Computational Engineering Michael Frydrych, Synface (telephone -> animated face) Figure by KTH Stockholm

Laboratory of Computational Engineering Michael Frydrych, … applications Language training speech training for profoundly deaf Diagnostics and therapy EU: VEPSY, VREPAR (assess and treat naxiety disorders and specific phobia)

Laboratory of Computational Engineering Michael Frydrych, Audiovisual speech integration = combining auditory and visual percepts into a single speech percept Strength of integration is demonstrated by McGurk- effect: combining sound /pa/ to a face ”telling” /ka/, speech percept is often /ta/ (McGurk & MacDonald, 1976, Nature) /pa/ /ka/ /ta/

Laboratory of Computational Engineering Michael Frydrych, Result:Computer animated talking face improves intelligibility of auditory speech A study in audio-visual speech perception

Laboratory of Computational Engineering Michael Frydrych, Psychophysical and psychophysiological experiments Audiovisual speech perception Emotion research … Benefits Natural stimuli may contain unwanted features Full controllability Quick creation of stimuli … application in research

Laboratory of Computational Engineering Michael Frydrych, Bulding on realism Realism: 1) Objective  topography, animation, texture, synchronization,... 2) Subjective (communication)  Audio-visual speech  Facial expressions, nonverbal behavior (prosody, eye movements) Evaluation: Objective  Subjective

Laboratory of Computational Engineering Michael Frydrych, Making the head speak Issues: Voice - speech synthesizer Animation – parameterization Synchronization

Laboratory of Computational Engineering Michael Frydrych, Acoustic Speech Generation Based on Festival platform. Developed at The Centre for Speech Technology Research, University of Edinburg, England. Scheme programming language, allows to program behaviour Finnish voice, prosody, expansion (numerals, etc.) Department of Phonetics, University of Helsinki Issues: production of articulatory parameters, synchronization

Laboratory of Computational Engineering Michael Frydrych, Animation methods - representation Polygonal Keyframing libraries of postures, interpolation Parametric deformations deformations are grouped under parameters meaningful to the animator Muscle-Based deformations Interactive deformations numerous control points, deformation propagation Free Form deformations deformation associated with a deformation box

Laboratory of Computational Engineering Michael Frydrych, Splines Implicit surfaces Physics-based models Physical models of the skin Volume preservation Deformations by inducing forces

Laboratory of Computational Engineering Michael Frydrych, Hooks to data Need the geometry of faces Rendering properties Deformation of facial expression or speech How? 2D and 3D techniques

Laboratory of Computational Engineering Michael Frydrych, D Input 3D digitizer is the most direct way, fairly automatic (Optotrack) 3D trackers – digitizing of projected/marked mesh, rather manual CT (Computer Tomography) and MRI (Magnetic Resonance Imaging) and … 3D modeling programs

Laboratory of Computational Engineering Michael Frydrych, D Input Photogrammetry Two images of an object are taken from different viewpoints, corresponding points are found The 3D shape of faces can be determined from a single 2D image after projecting of regular pattern Generic facial model is prepared and transformed to “match” a photograph 3rd dimension can be approximated by acquiring face model (set priors) and Bayesian inference

Laboratory of Computational Engineering Michael Frydrych, Texture mapping

Laboratory of Computational Engineering Michael Frydrych, break

Laboratory of Computational Engineering Michael Frydrych, Data for articulation and expressions Keyframing -> expression libraries Real-time/performance data Parameterization Articulatory parameters – jaw opening, lip rounding, lip protrusion, … Facial expressions – FACS Statistical models from expression libraries or real-time data

Laboratory of Computational Engineering Michael Frydrych, Statistical parameterization Parameterized model learned from 3D performance data (Reveret) Figure by ISCP Grenoble

Laboratory of Computational Engineering Michael Frydrych, … three control parameters Figure by ISCP Grenoble

Laboratory of Computational Engineering Michael Frydrych, … and the results Jaw Rounding Opening Raising Figure by ISCP Grenoble

Laboratory of Computational Engineering Michael Frydrych, Video by ISCP Grenoble

Laboratory of Computational Engineering Michael Frydrych, Finnish talking head Audiovisual database Using MaxReflex 3D optical tracker (at Linköping Univ.) Multiple IR cameras, reflexive markers  reconstruction from stereo Coarticulation, lips, visual prosody

Laboratory of Computational Engineering Michael Frydrych, Point-lights positions

Laboratory of Computational Engineering Michael Frydrych, Demo – live recording at Linköping

Laboratory of Computational Engineering Michael Frydrych, How to create “visemes” ?

Laboratory of Computational Engineering Michael Frydrych, Demo – reconstructed motion 10 fps 40 fps

Laboratory of Computational Engineering Michael Frydrych, Figure by ISCP Grenoble

Laboratory of Computational Engineering Michael Frydrych,

Laboratory of Computational Engineering Michael Frydrych,

Laboratory of Computational Engineering Michael Frydrych, End of 1st part