1. Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI

2. Our Background: Generation of ECA Behavior  Believable ECA: multimodality, affect, personality, human-like, etc.  Two approaches: Bari APML (Affective Presentation Markup Language) NECA RRL (Rich Representation Language)

3. Overview  Markup Schema Definition What to Annotate  What is a Markup Language What is XML  Dealing with complexity of human communication  Labelling Tools

4. Markup Scheme Definition  What to annotate? (verbal, non-verbal behavior, emotions, context, …)  Units of consideration (utterances/ turns/ paragraphs/ time units/…)  Descriptive Scheme Categorical Dimensional Functional …

5. The Question of Coding  How to Share and Access Data Annotation? Standardization Format Readable for Computers (statistics, inference, …)  XML-based Markup Languages!

6. What is a Markup-Language  Dictionary Definition: a set of symbols and rules for their use when doing a markup of a document nomenclature, terminology, language - a system of words used in a particular discipline;nomenclatureterminology language  Computer Domain Examples: HTML LATEX RTF … But … proprietary language not generic not standard fixed syntax and semantics

7. What is XML XML (eXtensible Markup Language) is  a language for creating markup languages that describe data thus, a meta language used to define new markup languages (domain- or industry-specific).language  Simple, open standard, extensible, ready to use, … XML has a lot of friends that makes computation easy (XSL, DOM, SAX, …) and interoperable (RDF, RDF Schema, OWL, …) No No No! I said turn left!!!

8. Dealing with complexity of human communication  Communication at different non-exclusive abstraction levels Mental state (Beliefs, Goals, Intentions, Affective State, …) Mental state Discourse (Communicative Goals, Focus, Relations) Discourse Meanings (Performative, Affect, Turn-Taking, Emphasis) Meanings Signals (Physical Communicative Expressions) Signals  Defining Different Markup Layers Separate levels of meaning into different layers Define a Markup Language for each layer Define Interaction between layers  Develop Easy-to-use tools

9. Labelling tools: Text-centered annotation: MMAX http://mmax.eml-research.de

10. Labelling tools: Video-centered annotation: ANVIL http://www.dfki.de/~kipp/anvil

11. Feedback into Generation Systems  Annotated Corpora can be used for improving ECA systems: extracting knowledge markup language for specifying ECA behaviors …

12. Overview of the session  Introduction to XML structure  Demos Use of APML in an ECA system XML markup in MARY Text-to-speech  Hands-on: The ANVIL tool and the resulting XML markup