ISO Project Semantic Annotation Framework, Part 2: Dialogue Acts Editorial Group first meeting Pisa, September 2008 TC 37/SC 4/WG 2 Kiyong Lee, convenor

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up (lunch time Tuesday?)

Project status -Launched as ISO project at SC 4 meeting in Marrakech, 25 May Documents: New Work Item Proposal ISO/TC 37/SC 4 N442; Working Draft WD ISO/TC 37/SC 4 N442 rev 00; Results of voting on new work item proposal ISO/TC 37/ SC 4 N458; -Editorial group: -Jan Alexandersson (Germany) -Harry Bunt (Netherlands/Belgium) (PL) -Jean Carletta (UK) -Alex Chengyu Fang (China/HK) -Jae-Woong Choe (Korea) -Koiti Hasida (Japan) -Olga Petukhova (Netherlands) -Andrei Popescu-Belis (Switzerland) -Claudia Soria (Italy) -David Traum (USA)

Background in ISO Lisbon, May 2004: formation of ISO TC 37/SC 4/TDG 3: Thematic Domain Group on Semantic Content Objective: “To prepare activities for possibly developing international standards and guidelines for semantic annotation” eContent project LIRICS ( ): To explore the needs, requirements, and possibilities of international standards for semantic annotation; to define a set of preliminary concepts for semantic annotation, certified by ISO TC 37/SC 4/TDG 3, in the form if entries in the ISO Data Category Registry.

ISO/LIRICS Data categories (Joint work by Tilburg U, U of Pisa, DFKI Saarbruecken, UPF Barcelona) Data categories for: - semantic role annotation - dialogue act annotation - reference annotation - temporal annotation certified by ISO TC 37/SC 4 Thematic Domain Group 3 (Semantic content)

LIRICS WP 4 Deliverables D4.1”Methodological aspects of semantic annotation and representation” (Harry Bunt & Amanda Schiffrin). Methodological foundations for metamodeling; comparative analysis of semantic annotation efforts. D4.3 “Documented compilation of semantic data categories ”. (Harry Bunt & Amanda Schiffrin) Set of data categories for the annotation of temporal information, reference, semantic roles and communicative functions. D4.4 “Multilingual test suites for semantically annotated data.” (Harry Bunt, Olga Petukhova & Amanda Schiffrin) Description of application of data categories from D4.3 and their evaluation for Dutch, English, Italian, Spanish, German, approved by TDG 3.

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up

Proposal NWIP ISO/... N442 Scope: “provide well-defined concepts for identifying dimensions of interaction that dialogue acts may address; for functional dialogue segmentation in multiple dimensions; for the definition of communicative functions. The standard will specify data categories for a range of core communicative functions, starting from proposals made jointly by the EU LIRICS project and the ISO TC 37/SC 4 Thematic Domain Group TDG 3 on Semantic Content.”

Proposal NWIP ISO/... N442 Purpose: “... provide annotation guidelines and examples. (...) The theoretical foundation of the LIRICS data categories (..) provides a basis for segmenting dialogue in multiple dimensions and allowing markables to be discontinuous and to overlap. The project will provide guidelines for how to effectively perform such segmentation. While it seems feasible, given the current state of the art, to develop standard annotation concepts for a range of core dialogue acts, researchers and applications designers should also be supported in adding their own concepts for specific domains or purposes. The standard will provide general principles and guidelines for extending its core concepts.”

N458: Results of voting on N442 Positive votes from 7 countries; no negative votes UK and Korea: Working Draft acceptable as Committee Draft Nominated experts: Pavel Smrz (CZ), Thierry Declerck (D), Aesun Yoon (Korea), UK to be nominated later Specified time schedule

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD 4 N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up

Project time schedule Submission of standards proposal in phases with maximal deadlines: as CD: 31 September 2009 as DIS: 31 March 2011 as FDIS: 31 October 2011 for publication as IS: 15 June 2012 Hopefully we can deliver much faster than that!

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up

Resonance group Purpose: 1.to accomodate nominated national experts 2.to obtain input from a wider circle of researchers For 1: Pavel Smrz, Thierry Declerck, Aesun Yoon,.. For 2: expressions of interest from various people, including James Allen, Laurent Romary, Gilles Francopoulo Proposal: list invitation to participate in meetings/workshops, such as January 2009 meeting

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up

LIRICS Results Evaluation of data categories for dialogue acts Inter-annotator agreement measurements for English and Dutch; 2 trained annotators working on raw text/audio Results: almost perfect agreement (Rietveld & van Hout, 1993: kappa ≥ 0.80)

LIRICS Results Function classEnglishDutchaverage Information-seeking Assistance-providing Feedback Interaction management Social obligations management 0.94

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up

Discussion items/WD comments At what level of generality should we aim to standardize? How should the standard relate to existing annotation schemas (DAMSL, SWBD-DAMSL, LIRICS, DIT++, AMI, Coconut, Verbmobil,...)? Specific technical comments Specific textual/editorial comments and suggestions Multimodality

Level of generality Principles for annotation schema design, illustrated with an instantiation; DiaML as an interlingua for relating annotation schemas; Core dialogue acts and their use.

Level of generality Principles for annotation schema design, illustrated with an instantiation (possibly the LIRICS schema or a version of that); DiaML as an interlingua for relating annotation schemas; Core dialogue acts and their use.

Level of generality Principles for annotation schema design, illustrated with an instantiation (possibly the LIRICS schema or a version of that); DiaML as a resource (an “interlingua”) for relating existing annotation schemas; Core dialogue acts and their use: a (multidimensional) taxonomy of dialogue acts which are present in many schemas in some form or another, and under some name or another, defined as ISO data categories (preferably with a specification of their semantics), and with principles for extending this set/ taxonomy with additional dialogue acts.

Relation of standard to existing annotation schemes Standard should not include specific annotation schema, but provide mappings between existing schemas; Standard should specify “interlingua” schema with standard concepts (as data categories), explaining their relation to existing schemas; Standard should include new annotation schema, based on the preliminary ISO/LIRICS work, as an improvement of DAMSL (etc.); this schema should be open, and allow the use of only a subset of its dialogue act types.

Specific technical comments Indirect speech acts as indirect forms of direct dialogue acts, or as different types of dialogue acts (David Traum)? Semantic content allowed in determining communicative function (David Traum, Jean Carletta)? Multidimensional annotation schemas versus flat lists of tags (Jean); Purpose/intention/consciousness (Jean) Acknowledge Bales’ Interaction Process Analysis (Jean) Aims and purposes (DIN comment) `Functional segment’ is not in the metamodel; what about things that aren’t on a dimension (Jean)

Specific textual comments/suggestions Section Purpose and justification not clear enough (Claudia, Jean); move material from beginning Section 5 and from Section 7 there (Claudia); Section 9 should also be place earlier (Claudia); Definition of “dialogue act” missing (Claudia, Jean) Figure 1 would be clearer if it had words (Jean); a worked example would help Sections 6 and 7 are somewhat confusing; the document should lead the reader from background to theoretical justification to core model to possible extensions (Claudia) Section 8 is very nice; DiaML could be a nice operationalization of the meta-model (Claudia) A concrete example would help to understand the XML example on p. 16 (Claudia).

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up

Meetings in 2009 January 5-6, 2009 in Tilburg, the Netherlands, preceding IWCS-8 (7-9 January) May/June 2009, Boston or Colorado October 2009, Berlin January 2010, ICGL-2 Hong Kong

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Comments on WD N442 rev 00 7.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up

Draft agenda 1.Background of the project 2.ISO formalities and procedures (Kiyong Lee) 3.Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of voting (ISO/TC 37/SC 4 N 458) 4.Project time schedule 5.Editorial group and resonance group 6.Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it 7.Comments on WD N442 rev 00 8.Planning of future meetings -- next meeting: Tilburg, January Actions and procedures for October January Any other business and wrap-up