2. WP 6: Emotion in Interaction Catherine Pelachaud, U Paris 8 Plenary, 4-6 June 2007, Paris

3. 2 humaine wp6 : Emotion in Interaction WP 6: The area Research theme: role of emotion in interaction. Three domains of study Perception domain: how certain aspects related to cognition may influence agent’s actions Interaction domain: how to create relations between users and agents; how the agent can provide feedback Generation domain: how to show expressive behaviours consistently and naturally across modalities

4. 3 humaine wp6 : Emotion in Interaction WP 6: the main teams University of Paris8 DIST - University of Genova EPFL - Lausanne France-Telecom ICCS - Athens Limsi CNRS OFAI - Wien T-systems - Berlin KTH - Stockholm MIRALab - Geneva DFKI - Saarburcken University of Ausburg University of Hertfordshire University of Paris 8 University of Sheffield Twente University INESC-ID - Lisbon TCD - Dublin University of Bari ISTC-CNR - Rome

5. 4 humaine wp6 : Emotion in Interaction PHIPS Definition of an Affective Interactive Embodied Conversational Agent that encompasses the capabilities: 1. Cognitive Influences on Action 2. Creating Affective Awareness 3. Backchannel properties and architecture 4. Coordination of signs in multi modalities 5. Expressive behaviour and speech

6. 5 humaine wp6 : Emotion in Interaction Element 1: Cognitive influence on actions Agent perceptual attention (UP8) Agents with real-time synthetic vision, attention and memory capabilities Model of attention and emotion aspects related to facial expression and novelty relation (WP3 / WP6) Evaluation study of the visual perception model GPU-based visual attention speed-up for real-time perception model (WP6 / WP7)

7. 6 humaine wp6 : Emotion in Interaction Element 1: Cognitive influence on actions: DEMO - UA Reaction to Agent‘s expressions (UA) Integration of tangible input device, speech recognition, emotional behavior control Analysis of user‘s gaze behavior

8. 7 humaine wp6 : Emotion in Interaction Element 1: Visual attention in affective agents Cooperation between UA and NII, Tokyo: Investigation of the relationship between visual attention and affect  Combining bio sensor with eye tracking technology Conduction of an empirical study under the leadership of Helmut Prendinger to investigate the potential benefits of attentive presentation agents Prendinger, Bee, Nischt, 2006

9. 8 humaine wp6 : Emotion in Interaction Element 2: Creating Affective Awareness Investigation of the level of user’s engagement with one another and with an ECA in an emotionally rich context (UA, HU, UP8, DIST, ICCS, KTH) create affective relationship with others humans / objects study of user’s engagement –when initiating, maintaining, ending an interaction –through music, emotion and movement –detection and imitation: ability to replicate emotional state

10. 9 humaine wp6 : Emotion in Interaction Element 2: Creating Affective Awareness Expressive control of music and visual media by full-body movement Collaboration between InfoMus Lab-DIST (University of Genova) and KTH (Royal Institute of Technology, Stockholm) Development of a system allowing users to express themselves through their full-body movement and to control in real-time the generation of an audio-visual feedback System based on the integration of two different software platforms: EyesWeb (for movement analysis and visual feedback generation) and pDM (to synthesize in real-time expressive music performances)

11. 10 humaine wp6 : Emotion in Interaction Element 2: Creating Affective Awareness: DEMO The real-time audio-visual feedback consists of (i) the rendering of a music performance with different emotional characterisations by manipulating acoustic parameters → the dynamic variations of the motor cues control the dynamics of acoustic cues such as tempo, sound level, articulation (ii) the rendering of the user's silhouette on a big screen in front of them coloured depending on the expressivity of their movement

12. 11 humaine wp6 : Emotion in Interaction Element 2: Creating Affective Awareness Development of realtime continuous emotion recognition from the speech signal (UA) Implementation of system to mirror the user‘s affective state by using: the Greta agent (UA) the empathic anthropomorphic robot (Collaboration between UA and Bielefeld University)

13. 12 humaine wp6 : Emotion in Interaction Element 3: Backchannels Three communication levels: establishing and maintaining engagement (contact, perception, attention) (WP4) comprehension (understand, interest) reaction (believability, attitude, agreement) Three different dimensions to characterise backchannel signals: cognitive/reactive (signals done with/without explicit planning) sincere/deceptive (sincerity/goal to deceive one’s reaction) imitation/dictionary (signal of alignment, positive/negative signals) Backchannel forms: verbal and nonverbal signals

14. 13 humaine wp6 : Emotion in Interaction Element 3: Backchannels Insight Data collection and analysis Theory and models Perceptual tests of affective bursts and facial expressions Modeling and Implementation Recognition Decision Generation Testing and evaluation DFKI, UTwente, URoma, UParis8, ISTC-CNR

15. 14 humaine wp6 : Emotion in Interaction Element 3: Backchannels: Dialogue Management Integration of the various components of a dialogue system capable of non-verbal expressivity: a visual renderer (Greta), an audio renderer (MARY), and a dummy dialogue system capable of generating non-verbal behaviour (Conversational Dialogue Engine / DFKI) Using OpenAIR

16. 15 humaine wp6 : Emotion in Interaction Element 3: Backchannels

17. 16 humaine wp6 : Emotion in Interaction Element 4: Coordination of signs in multiple modalities Models of coordination between modalities built from Automatic analysis of instructed/acted behaviors (ICCS) Manual annotation of spontaneous behaviors (CNRS-LIMSI, UP8) Perceptual studies: Comparison of the original video with 4 animations: –basic emotion 1 (e.g. Anger) –basic emotion 2 (e.g. Despair) –multiple levels replay –facial blending replay (UP8) 40 subjects No-audio #3 #41 Audio #3 #41

18. 17 humaine wp6 : Emotion in Interaction Element 4: Coordination of signs in multiple modalities Models of individual expressive behaviors in each modality: example of reaction movements (EPFL) Semantic representations : find concepts and relationships among them Morphological Descriptors: height, gender, age, etc. Individuality: personality, emotional state, cultural background, etc. Body: geometry, skeletal structure Behavior Controllers: inputs required for algorithm to work and output it produces. Reaction behavior Inverse Kinematics

19. 18 humaine wp6 : Emotion in Interaction Element 5: Expressivity Expressive behaviour: analysis/synthesis of expressive behaviours (DIST, OFAI, UP8, ICCS) Expressive speech synthesis: blending of emotions, control of voice quality in speech synthesis, copy synthesis of emotional speech (DFKI, FT, T-S) Model of complex emotions (UP8) Reliable features of sadness Fake joySadness masked by joy + =+ Neutral expression JoySadnessSuperposition of Sadness and Joy = + EmoTV

20. 19 humaine wp6 : Emotion in Interaction Element 5: Expressivity Effects of Expressivity parameters over head, facial expression and gesture over different time span: gesture phase, whole gesture, whole sequence behavior mimicry (ICCS-UP8)

21. 20 humaine wp6 : Emotion in Interaction Element 5: Expressivity GEMEP: Corpus of acted emotional performances created by WP3/ Geneva. Feature Extraction from Audio Channel (OFAI) Phonetic segmentation into Phonemes Syllables Pitch Extraction Features from Video Channel (OFAI, DIST, UP8) Face detection Silhouettes & Bounding Boxes Hand tracking Manual annotation and replay (UP8)

22. 21 humaine wp6 : Emotion in Interaction Element 5: Expressivity Restitution of salient information in human-machine interactions (FT) Prosodic copy (F0 + duration) of the focused part (words…) from dedicated corpus to neutral synthesized utterances TD-PSOLA copy synthesis on the focused part Use focused part as target in unit selection Synergies with national project PAVOQUE on parameterisation of prosody and voice quality for expressivity in speech synthesis (DFKI) Spectral interpolation using LSF Voice adaptation with HMM synthesis Emofilt: emotional speech synthesis by prosody transformation (T-S) Interface to DFKI‘s MARY TTS Available in 34 languages Meant as a pragmatic tool Screenshot of Emofilt GUI sad anger

23. 22 humaine wp6 : Emotion in Interaction Conclusion Creation of affective ECA able to: Perceive, adapt, respond affectively to events, objects, people in real/virtual world Create affective bonds Provide affective feedback Be multimodal and expressive