1 MPML and SCREAM: Scripting the Bodies and Minds of Life-Like Characters Soft computing Laboratory Yonsei University October 27, 2004

2 Agenda sub-title Introduction Related work –Scripting the Bodies of Life-Like Characters –Scripting the Minds of Life-Like Characters Multi-modal Presentation Markup Language (MPML) Scripting Emotion-based Agent Minds (SCREAM) Conclusion

3 Introduction Communication of humans –Language –Body gesture –Facial expressions Character-based interfaces –An effective tools to enhance the interaction between humans and computers Two categories in designing life-like characters –To instruct the “body” of an animated character –To design the mental make-up of a character

4 Scripting the Bodies of Life-Like Characters Scripting –Low level Manipulating a character’s facial and body motion models –High level The synchronization of a character’s synthetic speech with appropriate gestures The expression of emotions The synchronization of body motion sequences of an ensemble of characters Existing scripting languages –CML (Character Markup Language) –VHML (Virtual Human Markup Language) –BEAT (Behavior Expression Animation Toolkit) –APML (Affective Presentation Markup Language) –Hard to use for non-experts MPML (Multi-modal Presentation Markup Language) –Designed for non-experts

5 Scripting the Minds of Life-Like Characters A task complementary to scripting a character’s visual appearance –To author a character’s mental state –The way the character perceives its surrounding world –How the character reacts emotionally SCREAM (Scripting Emotion-based Agent Minds) –Similar to Reilly’s EM architecture –But more flexible in the sense –Emotion-based synthetic characters can be run in a web browser like the Extermpo and IMP systems

6 Multi-Modal Presentation Markup Language What is MPML? –A scripting language to instruct the body-related behavior of animated agents The Features of MPML –Ease of use –Intelligibility –Believability –Extensibility –Easy Distribution

7 The MPML Family MPML 2.2a –MPML 2.2a supports sequential and parallel behavior of multiple characters an interface to Macromedia Flash DWML –DWML is concerned with scripting dynamic relations between web-based media objects MPML-VR –MPML-VR enables presentations in 3D space

8 MPML System Architecture Fig 3

9 MPML System Architecture (cont.) MPML Script text file –Containing the tagging structures specifying character and environment control MPML 3.0 Visual Editor –Script Loader module Load the text file Check the script for syntactical errors –Graph module Visualize the script by generating a graphical representation of the presentation –Script Saver module Convert the graph to a textual MPML script –Converter module Transforms the MPML script to JavaScript

10 MPML Specification Using DTD (document type definition) for readability Element –Consisted the name and child elements –If no child elements exist for a tag, it is called “empty” Attribute –Name-value pairs separated by “=” –An attribute value may be required or fixed, denoted by #REQUIRED and #FIXED Entity declarations –(e1, e2, …, eN) : An ordered list of child elements –(e1 | e2 | … | eN) : An unordered list of child elements –e? : zero or one time –e+ : one or more times –e* : zero or more times –e : exactly on time

11 Tagging Structures for Character Control DTD for character-related elements

12 Tagging Structures for Presentation Control DTD for presentation-related elements

13 MPML Visual Editor

14 Using MPML to Script an Interactive Presentation

15 Scripting Emotion-based Agent Minds What is SCREAM? –A tool to script the interactive behavior of affective agents The key properties of SCREAM –Ease of use –Intelligibility –Believability –Extensibility –Granularity

16 SCREAM System Architecture

17 Communicative Acts Emotion-based agents receive input in the form of communicative acts Written as com_act (S, H, Concept, Sit) –S: speaker –H: the addressee (interlocutor) –Concept: the information conveyed by S to H –Sit: situation SCREAM System Architecture

18 Emotion Generation Appraisal Module –Emotion types 22 classes: joy, distress, happy for, sorry for, resent, angry at, and so on. Intensity: δ i ∊ {0, …, 5} –Personality Two dimensions: agreeableness and extraversion  ∊ {-5, …, 5} Emotion Resolution Module –Select the active emotions in Sit –from a set {,, …, } Emotion Maintenance –Decrease the intensity by the agent’s personality SCREAM System Architecture

19 Emotion Regulation Social Variables –Social power social-power(L2,L1,θ P,Sit) agent L2 is θ P ranks higher than L1 in Sit, as perceived by L1 –social distance social-distance((L1,L2,θ D,Sit) The closeness between two agents(L1,L2) Lower values indicate more closeness, as perceived by L1 SCREAM System Architecture

20 Emotion Regulation (cont.) Control Variables –Personality An agent is more likely to express negative emotions if it is either disagreeable or extrovert –Other agent’s personality If the agent assumes that the interlocutor’s personality is disagreeable, it will rater not express a negative emotion –Reciprocal feedback There are “negative” feedback loops where agents adjust to linguistic unfriendly behavior Emotion Filter Module –ε = δ – (θ +  ) SCREAM System Architecture

21 Emotion Expression Expressive responses –Somatic responses (flushing) –Behavioral responses (fist-clenching, throwing objects) –Two types of communicative responses, verbal and non-verbal (for instance, frowning) Information-processing responses –The agent’s diversion of attention and evaluations SCREAM System Architecture

22 Agent Model Character profile Affect processing – –  ∊ { +, - }, L : agent, I : interlocutor –If (δ + - δ - ) > 0, agent L like agent I –Otherwise agent L dislike agent I SCREAM System Architecture

23 Using SCREAM to Script a Conversational Character

24 Using SCREAM to Script a Conversational Character Dialogue for five consecutive rounds where the user never follows Djinn’s advice. The figure does not show the user’s interaction with the dealer

25 Conclusion MPML –A powerful and easy-to-use character scripting tool for web content experts who are non-programmers SCREAM –A practical technology for scripting the mental processes underlying a character’s affective behavior Future research –The design of an architecture that allows for a tighter integration of character and environment scripting –The development of a plot manager that produces narratively meaningful interactive stories