Österreichisches Forschnungsinstitut für Artificial Intelligence Representational Lego for ECAs Brigitte Krenn

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Motivation Background representations for multimodal behaviour generation use representations at the interfaces of system components

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Motivation Wish reusable, flexible representational “standards” to devise interface representations that ease exchange of system components integration of new modules

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Motivation Current everybody does their own language there is a wealth of different representations partially overlapping partially differing

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 up to date attempts to design a standard representation language for ECAs have failed

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Goal to have reusable extendable mappable bits and pieces of representations of ECA relevant information

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Formal Requirements separation of declarative and procedural information mapping between high-level concepts and their low-level representations mapping across concepts extendibility granularity of descriptions incorporation of new concepts ability to embed existing XML representations

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Content Side ? units of information common to existing ECA systems information ECA systems ideally should have allow for optionality

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Current State

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Current State Terminology Markup Languages Representation Languages Scripting Languages

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Current State Markup Languages for non-expert users to annotate text with high-level expert information e.g. VoiceXML for creating voice enabled applications VHML for creating interactive applications with ECAs APML for annotating text with high-level ECA controls

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Current State Representation Languages technically detailed annotations of theory-specific information high- and low-level concepts for expert use function as data representation formats inside a system e.g. RRL

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Current State Scripting Languages combine declarative and procedural knowledge comparable to high-level programming languages e.g. STEP/XSTEP, ABL, PAR/EMOTE

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Current State Summary Markup languages high-level concepts are indispensable for application development Representation languages mix high- and low-level concepts are crucial in research contexts Scripting languages add procedural information

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Current State Summary Representation Language Scripting Language Markup Language

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Current State Summary standardisation efforts up to now concentrate on markup languages they are application oriented to design representations for ECAs in the spirit of VoiceXML text/utterance oriented to design multimodal behaviour control as markup for text

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Pros for XML-based Representation Languages as Interfaces in ECA systems

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Advantages of XML Encoding XML is flexible easy to share tools for XML processing standardization efforts (w3c)

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Advantages of the Use of Representation Languages to encode information flow between system components to map between high-level concepts and low-level realizations to ease integration/replacement of system components to support a plug-and-play approach to support the development of mockup systems

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Affective, Interactive ECA Relevant Components and Concepts

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Content Side Current Foci speaking ECA simulation of mm-dialogues no/little “true” interactivity APML, RRL moving ECA XSTEP, MURML, (?MiraLab) speaking and moving ECA there are some gaps to bridge interacting ECA approaching PML, ABL

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Content Side Representations & Architectures information relevant for a certain ECA system depends on the architecture and system components used ?Is it possible to identify a common core of relevant components and concepts ?Is it possible to provide reusable representations for these concepts !Allow for flexibility of the representations

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Content Side Relevant Parts World parameters Scenes and story lines MM-dialogue generation Speech Animation (body, face) Affect (emotion, personality traits) + Temporal control and synchronization ?Interactivity ?MM-comprehension

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Content Side Temporal Control and Synchronization time-alignment of mm-behaviour of an agent temporal ordering of the actions/behaviours of agents interacting with the outside world agent-object agent-agent agent-user

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Content Side Time-Alignment of MM-Behaviour speech as guiding medium (phoneme durations) motion: beats as smallest units (e.g. XSTEP) synchronization of speech rate and motor activity motor activity can also constrain voice quality

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Content Side Interactivity What are the desired smallest communicative units? speech dialogue interactive drama What are the technically manageable smallest units? What are the technological challenges?

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 The Content Side Interactivity Multimodal understanding ? What is relevant information How do we manage the information flow → agent technology (Lola et al.) Models of the listening ECA

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Next Steps examine existing languages for a common core compare their representations consider architectural aspects define XML representations for bits and pieces make them publicly available

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 talk me off join in !

Österreichisches Forschnungsinstitut für Artificial Intelligence HUMAINE Workshop Paris 10./11. March 2005 Affective, Interactive ECA Relevant Components and Concepts how to incorporate an affective component what aspects of affect are modelled emotion, mood, interpersonal stance, attitude, personality traits what theories are applied affect representations are distributed over various components crosscutting modalities