1. RRL: A Rich Representation Language for the Description of Agent Behaviour in NECA Paul Piwek, ITRI, Brighton Brigitte Krenn, OFAI, Vienna Marc Schröder, DFKI, Saarbrücken Martine Grice, IPUS, Saarbrücken Stefan Baumann, IPUS, Saarbrücken Hannes Pirker, OFAI, Vienna

3. NECA Duration: 2.5 years Start: October 2001 A new generation of mixed multi-user / multi agent virtual spaces for the internet Populated by affective conversational agents

4. Affective Conversational Agents Express themselves through –Emotional speech and –synchronised non-verbal expression

5. Application Scenarios Socialite –a multi-user web-application in the social domain eShowRoom –a novel approach to the presentation of products in e-Commerce applications The NECA Platform will be evaluated in two concrete application scenarios

6. Socialite

8. NECA’s Architecture Scene Generator User Input Scene Description Affective Reasoner (AR)

9. NECA’s Architecture Scene Generator User Input Scene Description Multi-modal Output Multi-modal Natural Language Generator (M-NLG) Affective Reasoner (AR)

10. NECA’s Architecture Scene Generator Text/Concept to Speech Synthesis (CTS) User Input Scene Description Multi-modal Output Multi-modal Natural Language Generator (M-NLG) Phonetic+Prosodic Information Affective Reasoner (AR) Emotional Speech

11. NECA’s Architecture Scene Generator Text/Concept to Speech Synthesis (CTS) User Input Scene Description Multi-modal Output Multi-modal Natural Language Generator (M-NLG) Gesture Assignment Module (GA) Phonetic+Prosodic Information Affective Reasoner (AR) Emotional Speech Animation directives

12. NECA’s Architecture Scene Generator Text/Concept to Speech Synthesis (CTS) User Input Scene Description Multi-modal Output Multi-modal Natural Language Generator (M-NLG) Gesture Assignment Module (GA) Animation Control Sequence Phonetic+Prosodic Information Affective Reasoner (AR) Emotional Speech Player-Specific Rendering Animation directives

13. NECA’s Architecture Scene Generator Text/Concept to Speech Synthesis (CTS) User Input Scene Description Multi-modal Output Multi-modal Natural Language Generator (M-NLG) Gesture Assignment Module (GA) Animation Control Sequence Phonetic+Prosodic Information Affective Reasoner (AR) Emotional Speech Player-Specific Rendering Animation directives RRL

14. Requirements for RRL Application Domain –Represent combinations of different types of information –Expressivity Processing Modules –Ease of manipulation/search (incremental/fast) Developers (Maintainability) –Predictability –Locality –Conciseness –Intelligibility

15. Scene Description SG M-NLG GA TTS/CTS What is a Scene? I Theatr. 1 A subdivision of (an act of) a play, in which the time is continuous and the setting fixed, …; the action and dialogue comprised in any one of these subdivisions. (New Shorter Oxford English Dictionary, 1996)

16. Scene Descriptions in a Nutshell Network representations: –Flat, uniform –Use the Description Logical T and A-box distinction. T-box defines types, subtypes, attributes and constants –Can emulate CFGs, so we can include, e.g., semantic representation languages: Discourse Representation Theory (Kamp & Reyle, 1994) –Reification of expressions in the network provide useful handles for interleaving different types of information –Lends itself well for graphical representation

17. Scene Descriptions in a Nutshell Further Features of (RRL) Scene Descriptions –For communication between modules: XML syntax –Temporal relations are explicitly represented. –Meta-conditions used in DRT for WH-questions, Topics and Bridging Anaphora

18. eShowRoom Example

22. Multimodal Output SG M-NLG GA TTS/CTS Multimodal Natural Language Generation (M- NLG) supplies –Information on emotional state –Conceptually rich input for Speech Synthesis –Initial specification of gestures and facial expressions for later use in Gesture Assignment

23. Neca’s Speech Synthesis: Emotions SG M-NLG GA TTS/CTS Not restricted to prosody (pitch, duration) Several voice databases –diphon-inventories for different voice qualities (modal, loud, soft) Emotive interjections Gradual emotional states –Shades of emotion / changing over time

24. Neca’s Speech Synthesis: Concept-to-Speech SG M-NLG GA TTS/CTS Concept-to-Speech instead of Text-to- Speech approach: –Part of Speech tags –Syntactic structure –Information status (given/new) –Information structure (theme/rheme)

25. CTS specific information SG M-NLG GA TTS/CTS This car has leather seats.

26. CTS specific information SG M-NLG GA TTS/CTS This car has leather seats.

27. CTS specific information SG M-NLG GA TTS/CTS This car has leather seats.

28. CTS specific information SG M-NLG GA TTS/CTS This car has leather seats.

29. CTS specific information SG M-NLG GA TTS/CTS This car has leather seats.

30. Prosodic/Phonetic Information for GA SG M-NLG GA TTS/CTS Phonetics –exact timing of speech sounds, pauses and interjections Prosody –boundarie locations for syllables words prosodic phrases

31. Prosodic/Phonetic Information for GA SG M-NLG GA TTS/CTS –information on: syllables bearing word-stress position and type of sentence accents position and type of prosodic boundaries

32. Animation directives SG M-NLG GA TTS/CTS Phonetic information (phonemes) used for specifying –Visemes –breathing

33. Animation directives SG M-NLG GA TTS/CTS Prosodic information (stress, accents, phrasing) used for specifying –synchronization of gestures with speech –eye-blinking –gaze

34. Conclusions RRL is representation language for wide range of expert knowledge required at interfaces of NECA modules. Scene Descriptions: uniform representation/integration of different types of information (illustrated with integration of DRT); using handles;… Speech Synthesis: conceptually rich input as opposed to text Gesture Assignment: access to exact timing of speech