1. Toward a Unified Scripting Language 1 Toward a Unified Scripting Language : Lessons Learned from Developing CML and AML Soft computing Laboratory Yonsei University October 1, 2004

2. Toward a Unified Scripting Language 2 1. Introduction Two approaches to specifying scripting languages for character animation –Character Markup Language(CML) Top-down approach by defining high-level attributes for character personality, emotion and behavior that are integrated to form the specification of synchronized animation script –Avatar Markup Language(AML) Bottom-up approach in which the language provides a generic mechanism for the selection and synchronized merging of animations Need for powerful yet generic scripting languages to bridge the gap between behavior generation and animation tools

3. Toward a Unified Scripting Language 3 Visual Behavior Definition –A third factor that governs character behavior The actions an agent needs to perform in a session to achieve given tasks Personality and current mental state The role the agent is given –The behaviors are defined as XML tags 2. Scripting with the Character Markup Language CML Processor High-level behavior tags (map appropriate action point parameters) Animation Script

4. Toward a Unified Scripting Language 4 2. Scripting with the Character Markup Language Classification of Motion –Derived from Blumberg and Russell’s architecture (three- layer structure : geometry, motor, behavior system) –The initial set of CML base motions is classified by the goal of the motion as follows: Movement defines motion that require the rotation or movement of a character from one position to another. Positions are defined by exact coordinates, an object position, or a character position. (move-to, turn-to) Pointing defines a pointing gesture toward a coordinate, object, or character. (point-to) Grasping defines motions that require the character to hold, throw, or come in contact with an object or another character (grasp, throw, touch) Gaze defines the movements related to the head and eyes. (Gaze are gaze, track, blink, look-to, and look-at. ) Gesture includes motions that represent known gestures like hand movement to convey an acknowledgment, a wave, etc. (gesture-at)

5. Toward a Unified Scripting Language 5 2. Scripting with the Character Markup Language CML Specification –Animated character behavior is expressed through the interpretation of XML Schema structures –The language contains Low-level tags (specific character gesture representations defining movements, intensities, and explicit expression) High-level tags (commonly used combinations of these low- level tags –Synchronization between the audio and visual modalities is achieved through the use of SMIL (, )

6. Toward a Unified Scripting Language 6 2. Scripting with the Character Markup Language CML Representation Language –Head Gesture Taxonomy Symbolic gestures, Iconic gestures, Deictic gesture –Hand Gestures Taxonomy Posture, Motion, Orientation, ‘Gestlets’, Fingers –Body Gestures Taxonomy Natural, Relax, Tense, Iconic, Incline –Emotions (based on the OCC theory of emotion) Class, Valence, Subject, Target, Intensity, Time-stamp, Origin

7. Toward a Unified Scripting Language 7 2. Scripting with the Character Markup Language CML Scripting Language –Face Animation Scripting Head Movement tilt : movement in a slant with often subtle or superficial neck movement turn : require more profound movement of the neck

8. Toward a Unified Scripting Language 8 2. Scripting with the Character Markup Language Head Gesture Deictic Symbolic Face Movement and Gesture Define the elements and behaviors for specific parts of the face including the Brow, Gaze, and Mouse … …

9. Toward a Unified Scripting Language 9 2. Scripting with the Character Markup Language –Body Animation Scripting A set of defined body elements which are low level tags based on MPEG-4 BAPs A set of high-level tags representing body parts which are grouped from a set of respective low-level tags Base elements are.. Movement (moving, bending, turning) Gesture defines body postures that include motions representing common Iconic, Symbolic, or Deictic body gestures Posture (expression) defines a set of high-level tags representing general body gesture »Natural,Relax,Tense, Incline

10. Toward a Unified Scripting Language 10 CML Generation Animation Generation 2. Scripting with the Character Markup Language Generating Script State and Context Input Emotion Signal Behavior/Action Speech Text PDTD EDTD BDB FDB CML Processor Action Composition Utterance Composition Synchronisation CML Decoder CML Script FAP Audio/TTS FAP BAP Audio/TTS Face &Body

11. Toward a Unified Scripting Language 11 3. Avatar Markup Language Objective : to design and develop a full end-to-end MPEG-4 multimedia framework to support, amongst other features, 3D avatar-based multi-user chat rooms and autonomous synthetic characters Three component –A database of basic facial and body animation units(can extended by third party) –A rendering system capable of merging multiple face and body animation units and text to speech input in realtime –A high-level scripting language designed to allow animators to specify which animations to use together with timing, priority, and decay information

12. Toward a Unified Scripting Language 12 3. Avatar Markup Language AML Specification –Flexiblity : as many Expression Tracks as required, each containing as many Expressions as required …… … mm:ss:mmm ”name” …… …

13. Toward a Unified Scripting Language 13 3. Avatar Markup Language … mm:ss:mmm|autosync|autoafter normal|slow|fast float, 0 to fn integer, 0 to n … … float, target’s X coordinate in meters float, target’s Y coordinate in meters float, target’s Z coordinate in meters … …

14. Toward a Unified Scripting Language 14 4. CML and AML Applied CML –Sample CML – Happy Move and Point Script

15. Toward a Unified Scripting Language 15 4. CML and AML Applied AML –Sample AML – Walk and Point ”Let me show you another phone over here” 25 00:06:000.\Expressions\.\Speech\ 00:00:800 smile.ex … … ….

16. Toward a Unified Scripting Language 16 5. Discussion and Lessons Learned A Comparison

17. Toward a Unified Scripting Language 17 5. Discussion and Lessons Learned Towards a Unified Language –Objectives of a Unified Language Define a framework to decouple embodied agent animation tools and the underlying affect and planning engines Establish a formal specification for unified /consistent interpretation Provide for modular development Create a markup language based on XML (providing semantic and scripting annotations) –Language Requirements High Level Usability Extensibility Parameterized Synchronization Support Consistency Domain

18. Toward a Unified Scripting Language 18 6. Conclusion There is a trade-off to be made between higher levels of control (high granularity) and the complexity of the resulting language Striving toward a unified scripting and representation language may be the catalyst for much needed agreement