1. Österreichisches Forschnungsinstitut für Artificial Intelligence Representational Lego for ECAs Brigitte Krenn

2. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Motivation Background representations for multimodal behaviour generation use representations at the interfaces of system components

3. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Motivation Wish reusable, flexible representational “standards” to devise interface representations that ease exchange of system components integration of new modules

4. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Motivation Current everybody does their own language there is a wealth of different representations partially overlapping partially differing

5. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at up to date attempts to design a standard representation language for ECAs have failed

6. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Goal to have reusable extendable mappable bits and pieces of representations of ECA relevant information

7. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Formal Requirements separation of declarative and procedural information mapping between high-level concepts and their low-level representations mapping across concepts extendibility granularity of descriptions incorporation of new concepts ability to embed existing XML representations

8. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Content Side ? units of information common to existing ECA systems information ECA systems ideally should have allow for optionality

9. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Current State

10. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Current State Terminology Markup Languages Representation Languages Scripting Languages

11. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Current State Markup Languages for non-expert users to annotate text with high-level expert information e.g. VoiceXML for creating voice enabled applications VHML for creating interactive applications with ECAs APML for annotating text with high-level ECA controls

12. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Current State Representation Languages technically detailed annotations of theory-specific information high- and low-level concepts for expert use function as data representation formats inside a system e.g. RRL

13. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Current State Scripting Languages combine declarative and procedural knowledge comparable to high-level programming languages e.g. STEP/XSTEP, ABL, PAR/EMOTE

14. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Current State Summary Markup languages high-level concepts are indispensable for application development Representation languages mix high- and low-level concepts are crucial in research contexts Scripting languages add procedural information

15. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Current State Summary Representation Language Scripting Language Markup Language

16. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Current State Summary standardisation efforts up to now concentrate on markup languages they are application oriented to design representations for ECAs in the spirit of VoiceXML text/utterance oriented to design multimodal behaviour control as markup for text

17. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Pros for XML-based Representation Languages as Interfaces in ECA systems

18. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Advantages of XML Encoding XML is flexible easy to share tools for XML processing standardization efforts (w3c)

19. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Advantages of the Use of Representation Languages to encode information flow between system components to map between high-level concepts and low-level realizations to ease integration/replacement of system components to support a plug-and-play approach to support the development of mockup systems

20. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Affective, Interactive ECA Relevant Components and Concepts

21. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Content Side Current Foci speaking ECA simulation of mm-dialogues no/little “true” interactivity APML, RRL moving ECA XSTEP, MURML, (?MiraLab) speaking and moving ECA there are some gaps to bridge interacting ECA approaching PML, ABL

22. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Content Side Representations & Architectures information relevant for a certain ECA system depends on the architecture and system components used ?Is it possible to identify a common core of relevant components and concepts ?Is it possible to provide reusable representations for these concepts !Allow for flexibility of the representations

23. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Content Side Relevant Parts World parameters Scenes and story lines MM-dialogue generation Speech Animation (body, face) Affect (emotion, personality traits) + Temporal control and synchronization ?Interactivity ?MM-comprehension

24. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Content Side Temporal Control and Synchronization time-alignment of mm-behaviour of an agent temporal ordering of the actions/behaviours of agents interacting with the outside world agent-object agent-agent agent-user

25. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Content Side Time-Alignment of MM-Behaviour speech as guiding medium (phoneme durations) motion: beats as smallest units (e.g. XSTEP) synchronization of speech rate and motor activity motor activity can also constrain voice quality

26. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Content Side Interactivity What are the desired smallest communicative units? speech dialogue interactive drama What are the technically manageable smallest units? What are the technological challenges?

27. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at The Content Side Interactivity Multimodal understanding ? What is relevant information How do we manage the information flow → agent technology (Lola et al.) Models of the listening ECA

28. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Next Steps examine existing languages for a common core compare their representations consider architectural aspects define XML representations for bits and pieces make them publicly available

29. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at talk me off join in !

30. Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 brigitte@oefai.at Affective, Interactive ECA Relevant Components and Concepts how to incorporate an affective component what aspects of affect are modelled emotion, mood, interpersonal stance, attitude, personality traits what theories are applied affect representations are distributed over various components crosscutting modalities