Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, March 2005, Paris

2 humaine WP6 workshop WP6 Emotion in Interaction Embodied Conversational Agents WP6 core task: describe an interactive ECA system with capabilities beyond those of present day ECAs Conceptual framework consists of three key domains: perception, interaction and generation Advance ECA capabilities in the key domains of perception, interaction and generation

3 humaine WP6 workshop WP6 Emotion in Interaction Perception: perception from audio and visual information; Agent perception of the users / other agents / world. human perceptual attention Interaction: role of emotions in interaction; Balanced perception and generation Interactants model of speaker(s) and listener(s) Agent Interpretation of discourse signals Active interpretation of signals Social attention for speakers and listeners based on facial expressions during social interaction Create affective awareness

4 humaine WP6 workshop WP6 Emotion in Interaction Generation: design of expressive signs. Produce dynamic expressive visual and auditory behaviours Achieve coordination of signs of emotion in multiple modalities Define representation language to encode information and use it to drive animation of ECAs

5 humaine WP6 workshop Current conception ECAs with more or less sophisticated presentation facilities. ECAs are to a certain extent able: to compute what to say based on discourse plans to generate synchronized behaviours to talk with emotional appropriate intonation to show some emotional expressions to perform gesture and body movement

6 humaine WP6 workshop What needs to be done improve the naturalness of ECAs multimodal integration and the display of emotional behaviours improve the believability of ECAs consistency of an ECA’s behaviour in terms of personality, cultural context and situation. improve the interactivity of ECAs take into account the particular user and the particular context markes up its own model of user and of context

7 humaine WP6 workshop ECA ECA ought to be able to interact with the User not only through written text, but with all modalities: words, voice, gesture, gaze, facial expression, body movements, body posture (even through touch?) conceive, represent and convey all the possible meanings that natural language and multimodal interaction convey in humans

8 humaine WP6 workshop ECA - Capabilities Capabilities required from an ECA corresponds to steps to move research forward WP6 exemplar: Definition of Affective Interactive Embodied Conversational Agents with several capabilities covering the 3 domains: perception interaction generation

9 humaine WP6 workshop ECA - Capabilities Cognitive influences on action Emotion related attention shifts Adapt politeness behaviours to the user’s emotional state Creating Affective Awareness Creating Affective Bonds Imitation Adaptation Backchanneling

10 humaine WP6 workshop ECA - Capabilities Coordination of signs in multiple modalities From multimodal emotional corpora to models of coordination between modalities Multimodal behaviour Gesture repositories Expressivity Behaviour expressivity Speech Expressivity Context dependent emotional body gesture

11 humaine WP6 workshop Phases of Research Empirical research Modelling Implementation Evaluation

12 humaine WP6 workshop Phases of Research Phase 1 – Empirical Research Find out regularities in mind and behavior of Humans Define lexicons for all modalities: face, gaze, gesture, … through: empirical research (questionnaires) analysis of observational data Cooperation with WP3 (Theories and Models) and WP5 (Data and Database)

13 humaine WP6 workshop Phases of Research Phases 2 and 3 – Modelling and Implementation formalisation and representation of the rules found out in the previous phase Agent made of mind and body represent signals properties (shape and temporal dynamism) represent mind model Cooperation with WP4 (Signs of Emotion), WP7 (Emotion in Cognition and Action), WP8 (Communication and Emotion)

14 humaine WP6 workshop Phases of Research Phase 4 – Evaluation similarity of ECA behavior with human behavior believability of ECA animation added-value of ECA in interaction system how ECAs fit user’s need Cooperation with WP9 (Usability)

15 humaine WP6 workshop Example of ECA Greta: ECA system expressive agent individuality specification based on a model of communicative functions by I. Poggi

16 humaine WP6 workshop Model of Discourse and Conversation Based on Castelfranchi & Parisi and Poggi A sentence performs an action. It aims at a goal, and possibly at one or more supergoals A discourse is made of several sentences which aims at the same supergoals A conversation is a sequence of discourse where interactants “adopt” or “adhere” to each other’s goals adopt: I 1 acts to fulfil I 2 ’s goal even if I 2 has not asked I 1 to do so adhere: I 1 acts to fulfil I 2 ’s goal that I 2 has explicitly asked I 1 to do

17 humaine WP6 workshop Multimodal Communication Taxonomy of Communicative Functions (I. Poggi) The speaker may provide three broad types of information about: Information about the world: deictic, iconic (adjectival),… Information about the speaker’s mind: belief (certainty, meta cognitive) Goal (performative, rheme/theme, turn-system, belief relation) emotion Information about speaker’s identity (sex, cutlure, age…)

18 humaine WP6 workshop Information about Speaker’s Belief Speaker has the goal that Addressee believes some specific speaker’s beliefs Certainty markers: Indicate how reliable the provided information is Meta-cognitive marker: Meta information about the source or cognitive states of information the speaker is taking about: gaze of thought. Goal S Bel A (Bel S b)

19 humaine WP6 workshop Information about Speaker’s Goal Information about intention of single communicative act: performative of a sentence Information about a whole hierarchy of intentions: planning of a sentence Information about overall arrangement of conversation: turn-taking Information about the relation between beliefs Goal S Bel A Intend S Ag_Act Ag_Act: agent’s action

20 humaine WP6 workshop Performative Marker Communicate illocutionary force of speech act I order you to put on your coat I implore you to put on your coat I advise you to put on your coat Communicative intention of speaker that includes: Her goals (her reason to communicate to the addressee, what she wants the addressee to do) Her social relationship she wants to establish with addressee

21 humaine WP6 workshop Performative 3 main classes of performative (General Goals): request, inform, ask Request: command, implore, advise… Inform: warn, tell, announce… Ask: question, interrogate… Propositional content: set of beliefs speaker is mentioning and what she has to say about them.

22 humaine WP6 workshop Performative Elements distinguishing performatives within a same General Goal: In whose interest is the action requested: S or A * Advise vs command, inform vs warn Degree of certainty of S’s beliefs: certain, uncertain * Suggest vs assure Power relationship between S and A: * Command vs implore

23 humaine WP6 workshop Performative: Peremptory Order Example: In whose interest is the action: S Interest S a Degree of certainty: unmarked Power relationship: S has power on A Bel S (Power-on S A) Performative: S orders A to do action a. Ex: I order you to come here right away Goal S (Do A a) Signal of peremptory order: S looks down on A and frowns Bel S (if (not Do A a) then (Angry S))

24 humaine WP6 workshop Performative: Implore Example: In whose interest is the action: S (interest S a) Degree of certainty: unmarked Power relationship: A has power on S Bel S (Power-on A S) Performative: S implores A to do action a Ex: I implore for your help; I can’t do it otherwise Goal S (Do A a) Signal of imploration: S bends head aside and raised inner eyebrow Bel S (if (not Do A a) then (Sad S))

25 humaine WP6 workshop Information about Speaker’s Affective State Emotion are triggered by an event, action, person One can feel emotion toward another person (love, scorn) Emotion triggered by an event and not directed toward someone (fear, surprise) Signal: facial expressions of emotion Display of emotion are regulated Display or not of the emotion based on context Goal S Bel A (Feel S e)

26 humaine WP6 workshop Language of control of agent’s behaviors Based on APML: Affective Presentation Markup Language XML-based mark-up language Describe the communicative functions (I. Poggi) Control the agent’s animation Provide a mapping between meaning and signal

27 humaine WP6 workshop Expression Meaning deictic: this, that, here, there adjectival: small, subtle, difficult,…; or big, long, great,… certainty: certain, uncertain, … metacognitive: I'm planning, I'm thinking, … performative: greet, request (implore, order, suggest...), inform (warn, approve...), topic comment: this is the topic, this is the comment Belief relation: contrast,… turn allocation: take turn, give turn affective: anger, fear, happy, sorry-for, envy, relief, …

28 humaine WP6 workshop Expression signals Deictic: gaze direction, head direction Certainty: Certain: small intensity frown Uncertain: raised eyebrow adjectival: Small, tiny…: small eye aperture, small opening of fingers Belief relation: contrast: raised eyebrow; sequential symmetrical hand palm up Performative: Suggest: small raised eyebrow, head aside Emotion: Sorry-for: head aside, inner eyebrow up …

29 humaine WP6 workshop Example As far as vitamins are concerned research has shown that eating the recommended levels of vitamin A and C can have beneficial effects for your appearance and health

30 humaine WP6 workshop

31 humaine WP6 workshop Overall Architecture Björn Hartmann, Myriam Lamolle, Maurizio Mancini

32 humaine WP6 workshop Input Input text marked with APML Communicative Function tags and their coefficient of communicative strength Agent Definition Expressivity of agent Spatial, temporal, fluidity, power, repetitivity, overallActivity Agent-specific hierarchy of modalities (Which modalities the Agent tends to prefer?) Face, gaze, gesture, posture, … Predisposition factors (How expressive are each modalities?) Face, gaze, gesture, posture, … Database of

33 humaine WP6 workshop Video

34 humaine WP6 workshop Conclusion Creation of an Affective Interactive Embodied Conversational Agent perception interaction generation Collaboration requires within and across Workpackges