1 IUT de Montreuil Université Paris 8 Emotion in Interaction: Embodied Conversational Agents Catherine Pelachaud

2 Emotion in Interaction: Humaine Three domains of study –Perception domain: how certain aspects related to perception may influence agent’s actions –Interaction domain: how to create relations between users and agents; how the agent can provide feedback –Generation domain: how to show expressive behaviours consistently and naturally across modalities

3 Perception and Attention Non obtrusive agent: based on the interest shown by the addressee decides Non obtrusive agent: based on the interest shown by the addressee decides –when to start a communication –how long to maintain it –when to finish it Perception of attentive and emotional behaviors Perception of attentive and emotional behaviors –where to look at –how to interpret other’s behaviors

4 Perception and Attention Endow an ECA with the capabilities to: Endow an ECA with the capabilities to: –have a visual attention and perception model –show attention and interest –attract the attention of others

5 Interaction Interaction: two or more parties exchange messages. Interaction: two or more parties exchange messages. Interaction is by no means a one way communication channel between parties. Interaction is by no means a one way communication channel between parties. Within an interaction, parties take turns in playing the roles of the speaker and of the addressee. Within an interaction, parties take turns in playing the roles of the speaker and of the addressee.

6 Interaction Speaker and addressee adapt their behaviors to each other Speaker and addressee adapt their behaviors to each other –Speaker monitors addressee’s attention and interest in what he has to say –Addressee selects feedback behaviors to show the speaker that he is paying attention

7Interaction Agent as adressee ought to have the capabilities: Agent as adressee ought to have the capabilities: attention: pay attention to the signals produced by speaker to perceive, process and memorize them attention: pay attention to the signals produced by speaker to perceive, process and memorize them perception: of signals perception: of signals comprehension: understand meaning attached to signals comprehension: understand meaning attached to signals internal reaction: the comprehension of the meaning may create cognitive and emotional reaction internal reaction: the comprehension of the meaning may create cognitive and emotional reaction decision: decision to communication the internal reaction decision: decision to communication the internal reaction generation: display behaviors generation: display behaviors

8Interaction Providing feedback is a fundamental aspect of engagement Providing feedback is a fundamental aspect of engagement Backchannel: signals of Backchannel: signals of –1- perception –2- attention -- interest –3- understanding –4- attitude: belief (T/F); like (T/F); –5- acceptance positive/negative positive/negative any combination of the above any combination of the above –pay attention but not understand; understand but not believe

9 Expressive Behaviors Behaviors encode: Behaviors encode: –content information (the ‘What is communicating’) –expressive information (the ‘How it is communicating’) highly synchronized highly synchronized multimodal multimodal

10 Expressive Behaviors Basic emotions Basic emotions –Anger, fear, sadness, happiness, disgust, surprise –Universally recognized (Ekman) –Basic emotion = family of related states (Ekman 75) Real life emotions are often complex and involve several emotions Real life emotions are often complex and involve several emotions

11 Expressive Behaviors Blends may occur either as Blends may occur either as –superposition of emotions –masking of one emotion by another emotion –quick succession of different emotions –suppression / overacting of an emotion Superposition of 2 emotions: evaluation of an event can happen under different angles Superposition of 2 emotions: evaluation of an event can happen under different angles Mask an emotion by another one : consideration of social context Mask an emotion by another one : consideration of social context

12Examples Disappointment Masked by Joy Disappointment Masked by Joy Superposition of Sadness and Joy Superposition of Sadness and Joy DisappointmentjoyDisappointment masked by joy Joy Sadness Sadness and Joy

13Callas Creation of a virtual agent able to Creation of a virtual agent able to –perceive and be attentive –be aware of and adapt to the surroundings (physical and social context) –interact with users/other agents –communicate and listen –show ‘real’ emotions –use expressive gestures

14Callas Extension and refinement of the results obtained in Humaine Extension and refinement of the results obtained in Humaine Real-Time integrated system Real-Time integrated system Integration of virtual and real perception models Integration of virtual and real perception models Real situations Real situations