1. www.amiproject.org Collaborative Annotation of the AMI Meeting Corpus Jean Carletta University of Edinburgh

2. www.amiproject.org Carletta20 June 2007 2 AMI Partners

3. www.amiproject.org Carletta20 June 2007 3 NXT Major Development Sites

4. www.amiproject.org Carletta20 June 2007 4 AMI's aim aim: to develop technologies for browsing meetings and to assist people during meetings interdisciplinary: signal processing, language engineering, theoretical linguistics, human-computer interfaces, organizational psychology,...

5. www.amiproject.org Carletta20 June 2007 5 Why annotation? For basic scientific understanding - e.g., How do people choose a next speaker? What is the relationship between speech and gesture during deixis? For machine learning Hand-code e.g. statement vs. question Identify features for each like word sequences and prosody Use the data to fit a statistical classifier that codes new data automatically

6. www.amiproject.org Carletta20 June 2007 6

7. www.amiproject.org Carletta20 June 2007 7

8. www.amiproject.org Carletta20 June 2007 8 AMI Meeting Rooms 4 close- and 2 wide-view cameras, 4 head-set and 8 array microphones, presentation screen capture, whiteboard capture, pen devices, plus extra site-dependent devices TNOEdinburghIDIAP

9. www.amiproject.org Carletta20 June 2007 9 IS1004d, 3:07 - 4:11

10. www.amiproject.org Carletta20 June 2007 10 Corpus Overview 100 hrs of well-recorded meetings orthographically transcribed with word timings by forced alignment ASR output heavily annotated by hand for communicative behaviours Creative Commons Share-Alike licensing, with demo DVD

11. www.amiproject.org Carletta20 June 2007 11 Hand Annotations transcription with word-level timings from forced alignment (100%) timestamping against signal (10-30%) head gestures; hand gestures for addressing and interactions with objects; location in room; gaze; emotion? discourse structure (70%) dialogue acts (some w/ addressing), named entities, topic segments, linked extractive and abstractive summaries

12. www.amiproject.org Carletta20 June 2007 12 Costs in person-hrs/hr transcription30 topic segments + abstractive summaries6-10 dialogue acts w/ some relations20 addressing12 extractive summaries linked to abstract1 named entities2-5 hand gestures (rough timings)6 head gestures (rough timings)6 head gestures (precision timings)20 movement around room4

13. www.amiproject.org Carletta20 June 2007 13 Core Problems How do we represent all of these kinds of annotation on the same base data, including both structural relationships and timing? How do we allow for multiple (human and machine) annotations of the same property, so that we can compare them?

14. www.amiproject.org Carletta20 June 2007 14

15. www.amiproject.org Carletta20 June 2007 15

16. www.amiproject.org Carletta20 June 2007 16 NITE XML Toolkit Mature toolkit for handling annotations with temporal ordering and full structural relations Data storage format designed to support distributed corpus development Libraries for data handling, query, and writing graphical user interfaces End user annotation tools for common tasks Command line utilities for analysis, feature extraction Open source

17. www.amiproject.org Carletta20 June 2007 17 NXT corpus design data model is multi-rooted tree with arbitrary graph structure over the top each node has one set of children, multiple parents annotations often naturally map to a tree corpus design to decide where trees intersect NXT can represent arbitrary graphs but the more the data has this character, the less useful the query language is

18. www.amiproject.org Carletta20 June 2007 18 extract from Bdb001.A.words.xml time - line extract from Bdb001.A.speech-quality.xml Stand-off XML

19. www.amiproject.org Carletta20 June 2007 19 Metadata file Like set of DTDs for the XML files plus: connections between the files list of "observations" (coded dialogues/group discussions/texts) catalog for finding signals and data on disk

20. www.amiproject.org Carletta20 June 2007 20 Simple example query ($w word)($r reference): ($w@POS = “NN”) && ($r ^ $w) Return list of 2-tuples of words and referring expressions where the word’s part of speech is NN and the word is in the referring expression.

21. www.amiproject.org Carletta20 June 2007 21 General features of the language Match variable by no type, single type, or disjunctive type Attribute and content tests for existence, ordering, equality, match to regexp The usual boolean combinators Quantifiers forall and exists Filtering by passing results to another query to create a result tree (not list)

22. www.amiproject.org Carletta20 June 2007 22 Uses for queries Exploring the data in a browser Basic frequency counts Verifying data quality Indexing complexes for further use Finding things for screen rendering in GUI

23. www.amiproject.org Carletta20 June 2007 23 Only configuration needed to: search/index data in NXT format display data in a standardized (ugly) way Set up annotation tools for some common tasks dialogue act named entity time-stamped labelling

24. www.amiproject.org Carletta20 June 2007 24 [named entity demo]

25. www.amiproject.org Carletta20 June 2007 25 Programming tailored interfaces development time is 1.5 days - 2 weeks depending on how clear the spec is complexity of the interface and whether our "transcription view" middleware fits familiarity with Swing

26. www.amiproject.org Carletta20 June 2007 26 Named entity coder

27. www.amiproject.org Carletta20 June 2007 27

28. www.amiproject.org Carletta20 June 2007 28

29. www.amiproject.org Carletta20 June 2007 29

30. www.amiproject.org Carletta20 June 2007 30

31. www.amiproject.org Carletta20 June 2007 31

32. www.amiproject.org Carletta20 June 2007 32

33. www.amiproject.org Carletta20 June 2007 33

34. www.amiproject.org Carletta20 June 2007 34

35. www.amiproject.org Carletta20 June 2007 35 Summary NXT provides infrastructure for collaborative annotation that Is distributed Provides structural relationships Provides timing w.r.t signals Works for large-scale projects NXT’s best current demonstration is in the AMI Meeting Corpus