1. Multidimensional dialogue act annotation using ISO 24617-2 Harry Bunt Tilburg University ISO 24617-2 Project Leader IJCNLP 2011 tutorial, November 8, Chiang Mai

2. ISO Standard 24617-2 Semantic Annotation Framework, Part 2: Dialogue Acts ISO TC 37/SC 4

3. Project status -Launched in May 2008 -Accepted as International Standard ISO 24617-2, January 2011 -Project team: -Harry Bunt (Netherlands), Project Leader -Jan Alexandersson (Germany) -Jean Carletta (UK) -Alex Fang (China/HK) -Jae-Woong Choe (Korea) -Koiti Hasida (Japan) -Olga Petukhova (Netherlands) -Andrei Popescu-Belis (Switzerland) -Claudia Soria (Italy) -David Traum (USA)

4. Expert Consulting Group Current members:Jens AllwoodCarlos Martinez-Hinarejos James AllenMarieke van Erp Thierry DeclerckDavid Novick Nick CampbellTim Paek Roberta Catizone Patrizia Paggio Anna Esposito Massimo Poesio Raquel Fernández German Rigau Gil Francopoulo Laurent Romary Dirk Heylen Nicla Rossini Julia HirschbergMilan Rusko Kristiina JokinenCandice Sidner Maciej KarpinskiIelka van der Sluis Staffan LarssonPavel Smrz Oliver Lemon Krister Thorisson Paul Mc KevittAesun Yoon Michael McTear Yorick Wilks

5. Outline of this tutorial 1Introduction 2Multifunctionality, multidimensionality and dimensions 3Comunicative functions 4Dialogue segmentation 5Function qualifiers 6Relations between units in dialogue 7Annotation practice: examples of using DiAML, the ISO Dialogue Act Markup Language 8Concluding remarks; further information

6. 1. Introduction

7. Dialogue act: specimen of communicative activity of a dialogue participant, interpreted as having a certain communicative function and a semantic content. Semantic content: specification of objects, relations, actions, propositions,... that the dialogue act is about. Communicative function: specification of how the dialogue act's is intended to change the information state of an addressee.

8. Background - Previous dialogue act annotation schemes: TRAINS, HCRC Map Task, Verbmobil, DIT, SPAAC, C-Star, MUMIN, MRDA, AMI,... - Efforts towards domain-independence, interoperability and standardization: DAMSL (1997), MATE (1999), DIT++ (2005), LIRICS (2007)

9. Dialogue act analysis frameworks Speech Act Theory Communication as Cooperation (Grice) (Austin, Searle) Communicative Activity Analysis (Allwood) HCRC TRAINS MRDA … GBG-IMDIT Verbmobil-2 DAMSL + der. MATEDIT++ LIRICS ISO 24617-2 (DIS) DIT++ Release 5

10. ISO standard for dialogue act annotation Features: ♥ Domain-independent ♥ Concepts defined as data categories (following ISO 12620 standard) and stored in the ISOcat online registry ♥ Multidimensional ♥ Annotation language DiAML (Dialogue Act Markup Language) with: abstract and concrete syntax semantics in terms of information-state update operators defined for abstract syntax concrete syntax defining XML representations

11. 2. Multifunctionality, multidimensionality and dimensions

12. Multifunctionality A: Henry, could you take us through these slides? H: O..w..k..ay.. just ordering my notes

13. Multifunctionality A: Henry, could you take us through these slides? Turn Assign to Henry; Request H: O..w..k..ay.. just ordering my notes

14. Multifunctionality A: Henry, could you take us through these slides? Turn Assign to Henry; Request H: O..w..k..ay.. just ordering my notes Turn Accept; Stalling; Accept Request; Inform

15. Multifunctionality A: Henry, could you take us through these slides? Turn Assign to Henry; Request H: O..w..k..ay.. just ordering my notes Turn Accept; Stalling; Accept Request; Inform Dimensions of communication in dialogue:  Turn Management  Time Management  Task performance  Feedback giving .....

16. Multifunctionality ==> Multidimensional annotation Functional segments often have multiple functions therefore annotation must be “multidimensional” i.e. multiple tags must be assigned or single tags with a syntactic/semantic complexity

17. Dimensions, a conceptual view Participants in a dialogue act in order to perform a certain task or activity

18. Dimensions, a conceptual view Participants in a dialogue act in order to perform a certain task or activity and they also provide and elicit feedback;

19. Dimensions, a conceptual view Participants in a dialogue act in order to perform a certain task or activity and they also provide and elicit feedback; manage the use of speaking turns and time;

20. Dimensions, a conceptual view Participants in a dialogue act in order to perform a certain task or activity and they also provide and elicit feedback; manage the use of speaking turns and time; edit their own and their partner's speech;

21. Dimensions, a conceptual view Participants in a dialogue act in order to perform a certain task or activity and they also provide and elicit feedback; manage the use of speaking turns and time; edit their own and their partner's speech; open and close topics and subdialogues;

22. Dimensions, a conceptual view Participants in a dialogue act in order to perform a certain task or activity and they also provide and elicit feedback; manage the use of speaking turns and time; edit their own and their partner's speech; open and close topics and subdialogues; deal with social obligations (greet, thank, apologize...)

23. Dimensions, a conceptual view Participants in a dialogue act in order to perform a certain task or activity and they also provide and elicit feedback; manage the use of speaking turns and time; edit their own and their partner's speech; open and close topics and subdialogues; deal with social obligations (greet, thank, apologize...) These different kinds of communicative activity, concerned with different information categories, are called dimensions.

24. Dimensions in dialogue act analysis Criteria for distinguishing dimensions: Each dimension should correspond to observed forms of communicative behaviour (be empirically justified) correspond to a well-established class of communicative activities (be theoretically justified) be recognizable with acceptable precision by humans and machines be addressable independent of other dimensions (be ‘orthogonal' to other dimensions) be commonly represented in existing dialogue act annotation schemes (Petukhova & Bunt, 2009)

25. Dimensions in Dialogue Task: dialogue acts moving the underlying task forward

26. Dimensions in Dialogue Task: dialogue acts moving the underlying task forward Auto-Feedback: providing information about speaker's processing of previous utterances Allo-Feedback: providing or eliciting information about addressee's processing of previous utterances

27. Dimensions in Dialogue Task: dialogue acts moving the underlying task forward Auto-Feedback: providing information about speaker's processing of previous utterances Allo-Feedback: providing or eliciting information about addressee's processing of previous utterances Turn Management: allocation of speaker role

28. Dimensions in Dialogue Task: dialogue acts moving the underlying task forward Auto-Feedback: providing information about speaker's processing of previous utterances Allo-Feedback: providing or eliciting information about addressee's processing of previous utterances Turn Management: allocation of speaker role Time Management: managing use of time

29. Dimensions in Dialogue Task: dialogue acts moving the underlying task forward Auto-Feedback: providing information about speaker's processing of previous utterances Allo-Feedback: providing or eliciting information about addressee's processing of previous utterances Turn Management: allocation of speaker role Time Management: managing use of time Discourse Structuring: explicitly structuring the dialogue

30. Dimensions in Dialogue Task: dialogue acts moving the underlying task forward Auto-Feedback: providing information about speaker's processing of previous utterances Allo-Feedback: providing or eliciting information about addressee's processing of previous utterances Turn Management: allocation of speaker role Time Management: managing use of time Discourse Structuring: explicitly structuring the dialogue Own Communication Management: editing one's own speech Partner Communication Man.: editing addressee's speech

31. Dimensions in Dialogue Task: dialogue acts moving the underlying task forward Auto-Feedback: providing information about speaker's processing of previous utterances Allo-Feedback: providing or eliciting information about addressee's processing of previous utterances Turn Management: allocation of speaker role Time Management: managing use of time Discourse Structuring: explicitly structuring the dialogue Own Communication Management: editing one's own speech Partner Communication Man.: editing addressee's speech Social Obligations Management: dealing with social conventions (greeting, thanking, apologizing,..)

32. 3. Communicative functions

33. Criteria for distinguishing communicative functions: each communicative function should correspond to observed forms of communicative behaviour (be empirically justified) have a well-established semantics in terms of information-state updates (be theoretically justified) be recognizable with acceptable precision by humans and machines be important for achieving a good coverage of the phenomena in a given dimension be commonly present in existing dialogue act annotation schemes preferably be either mutually exclusive with the other functions available in a given dimension, or be a specialization of one

34. Communicative functions (taken from DIT++) Dimension-specific communicative functions, e.g.: T urn Release (Turn Management) Stalling (Time Management) Self-Correction (Own Communication Management) Completion (Partner Communication Management) Dialogue opening (Discourse Structuring) Thanking (Social Obligations Management) General-purpose functions, applicable in any dimension, e.g.: Information-seeking functions: Propositional Question, Set Question, Check Question, Choice Question Information-providing functions: Inform, Agreement, Disagreement, Correction Commissive functions: Promise, Offer, Accept Suggestion, Decline Suggestion,... Directive functions: Request, Instruct, Suggestion, Accept Offer, Decline Offer

35. Dialogue act analysis frameworks Speech Act Theory Communication as Cooperation (Grice) (Austin, Searle) Communicative Activity Analysis (Allwood) HCRC TRAINS MRDA … GBG-IMDIT Verbmobil-2 DAMSL + der. MATEDIT++ LIRICS ISO 24617-2 (DIS) DIT++ Release 5

36. General-purpose communicative functions Information-transfer functions action-discussion functions

37. General-purpose communicative functions Information-transfer functions action-discussion functions Info-seeking info-providing commissives directives

38. General-purpose communicative functions Information-transfer functions action-discussion functions Info-seeking info-providing commissives directives Offer Address Suggest Request Question Inform Suggest Promise Instruct Prop. Choice Q Set Q Disagreement Agreement Accept Decline Question Answer Suggest Suggest Address Request Address Offer Check Confirm Disconfirm Correction Accept Decline Question Accept Request Decline Request Offer Offer

39. Dimension-specific communicative functions in ISO 24617-2 and in DIT++ Auto- Allo- Contact Time Partner Turn Own Discourse Social Feedback Feedback Speech Speech Structuring Obligations Man. Management Managem.

40. Dimension-specific communicative functions in ISO 24617-2 and in DIT++ Auto- Allo- Contact Time Partner Turn Own Discourse Social Feedback Feedback Speech Speech Structuring Obligations Establish C. Man. Management Managem. Check C. Opening Positive Positive Stalling Completion Error signal Pre-closing I-Greeting Pos. Attention Pos. Attention Pausing Correct- Retraction R-Greeting Pos. Perception … misspeaking Self-correction I-Self-Introd. … Negative R-Self-Introd. Pos. Execution,,, Apology Negative Elicitation Accept-Apology Neg. Attention … turn-initial turn-final Thanking Neg. Perception Elic. Execution Accept-Thanking Neg. Understanding I-Goodbye Neg. Evaluation Turn Take Turn Keep R-Goodbye Neg. Execution Turn Grab Turn Release Turn Accept Turn Assign

41. Dimension-specific functions examples Dimension Comm. Function Example Auto-feedback Auto-Positive “Okay” Allo-feedback EvaluationElicitation “Okay?” Turn management TurnRelease “Yes...” Time management Stalling “Ehm..”, “Well...” Own comm. man’t Self-correction “I mean...” Partner comm.man't Completion [... completion] Dialogue structuring DA-announcement “Question:” Social oblig. man’t Valediction “Bye”

42. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task

43. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task – I see what you mean. Auto-feedback

44. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task – I see what you mean. Auto-feedback – You misunderstood me. Allo-feedback

45. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task – I see what you mean. Auto-feedback – You misunderstood me. Allo-feedback – I would like to hear Peter’s opinion. Turn management

46. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task – I see what you mean. Auto-feedback – You misunderstood me. Allo-feedback – I would like to hear Peter’s opinion. Turn management – I need a moment to check that. Time management

47. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task – I see what you mean. Auto-feedback – You misunderstood me. Allo-feedback – I would like to hear Peter’s opinion. Turn management – I need a moment to check that. Time management – I’m not sure you understood me correctly. Partner communication management

48. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task – I see what you mean. Auto-feedback – You misunderstood me. Allo-feedback – I would like to hear Peter’s opinion. Turn management – I need a moment to check that. Time management – I’m not sure you understood me correctly. Partner communication management –... I mean Toronto. Own communication management

49. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task – I see what you mean. Auto-feedback – You misunderstood me. Allo-feedback – I would like to hear Peter’s opinion. Turn management – I need a moment to check that. Time management – I’m not sure you understood me correctly. Partner communication management –... I mean Toronto. Own communication management – I would like to ask you something. Discourse structuring

50. General-purpose functions examples in different dimension s Inform acts in various dimensions: – The KL204 leaves at 12.30. Task – I see what you mean. Auto-feedback – You misunderstood me. Allo-feedback – I would like to hear Peter’s opinion. Turn management – I need a moment to check that. Time management – I’m not sure you understood me correctly. Partner communication management –... I mean Toronto. Own communication management – I would like to ask you something. Discourse structuring – I’m very grateful for you help. Social obligations management

51. Communicative functions 55 communicative functions - 25 general-purpose functions: 4 information-seeking functions 7 information-providing functions 8 commissive functions 6 directive functions - 30 dimension-specific functions 2 auto-feedback functions 3 allo-feedback functions 6 turn management functions 2 time management functions 3 discourse structuring functions 2 own communication management functions 2 partner communication management functions 10 social obligation management functions

52. Communicative functions All communicative functions: have a definition as ISO data category, following ISO 12620 standard for concept definitions will eventually be entered in ISOCat registry at http://www.isocat.org/ currently available at http://semantic-annotation.uvt.nl Example of data category: /acceptRequest Definition - Source - Note Example Communicative function of a dialogue act where the speaker commits himself to perform an action requested by the addressee DIT (http://dit.uvt.nl)http://dit.uvt.nl Related terminology in other schemes: Accept (DAMSL, TRAINS, Verbmobil) A: Could you take us through these slides Craig? C: Sure

53. 4. Dialogue segmentation

54. Multidimensional segmentation INFORM Propositional Question Positive: A1 Set Question Assign to C Positive: B2 Negative:A1 Propositional Question Accept A1: We’re aiming a fairly young market Task B1: Do you think then we should really consider voice recognition Task Auto-Feedback B2: What do you think Craig Task Turn Man.'t C1: Well did you not say it was the adults that we’re going for Auto-Feedb. Turn Man't

55. Full-blown characterization of dialogue acts Full-blown characterization of dialogue act involves, besides communicative function and semantic content, also: -Communicative function qualifiers: -certainty -conditionality -sentiment -Relations between dialogue acts and other units in dialogue -Functional dependence relations -Feedback relations -Rhetorical relations -among dialogue acts -among the semantic contents of dialogue acts

56. 5. Function qualifiers

57. Communicative function qualification Dialogue acts do not always have simple communicative functions: A: Do you know when and where the next meeting will be? B: I think it's somewhere early in September.

58. Communicative function qualification Dialogue acts do not always have simple communicative functions: A: Do you know when and where the next meeting will be? conditional request: “Please tell me … if you know” B: I think it's somewhere early in September.

59. Communicative function qualification Dialogue acts do not always have simple communicative functions: A: Do you know when and where the next meeting will be? conditional request: “Please tell me … if you know” B: I think it's somewhere early in September. uncertain answer (“I think... somewhere...”)

60. Communicative function qualification Dialogue acts do not always have simple communicative functions: A: Do you know when and where the next meeting will be? conditional request: “please tell me … if you know” B: I think it's somewhere early in September. uncertain answer (“I think... somewhere...”) A: Would you like to have some tea? B: That would be lovely!

61. Communicative function qualification Dialogue acts do not always have simple communicative functions: A: Do you know when and where the next meeting will be? conditional request: “please tell me … if you know” B: I think it's somewhere early in September. uncertain answer (“I think... somewhere...”) A: Would you like to have some tea? offer B: That would be lovely! happy accept offer

62. Communicative function qualifiers qualification aspect qualifierscommunicative function class certaintyuncertain, certain information-providing functions conditionalityconditional, unconditional action-discussion functions sentiment[open class] (happy, surprised, irritated,...) all communicative functions

63. 6. Relations between dialogue acts and other units in dialogue

64. Relations within a dialogue functional dependence relation: A dialogue act depends for its meaning on a previous dialogue act feedback dependence relation: A feedback act depends for its meaning on something that was previously said rhetorical relation: a. the performance of a dialogue is semantically related to the performance of another dialogue act b. the semantic content of a dialogue act is semantically related to that of another dialogue act

65. Functional dependence relations Functional dependence on a previous dialogue act: Answer → Question Accept Request → Request Disconfirmation → Check Question Decline Offer → Offer Accept Apology → Apology ….. Example: Answer: B: I certainly expect Alice and Bob, and maybe Charlie

66. Functional dependence relations Functional dependence on a previous dialogue act: Answer → Question Accept Request → Request Disconfirmation → Check Question Decline Offer → Offer Accept Apology → Apology ….. Example: Answer: B: I certainly expect Alice and Bob, and maybe Charlie Question: 1) A: Who do you expect at the meeting? 2) A: Which of the people from New York do you expect at the meeting?

67. Feedback dependence relations A feedback act has a feedback dependence on something that was previously said Example: 1. A: Is this flight also available on Tursday? 2. B: On Thursday you said? feedback dependence on 1

68. Dependence relations A feedback act has a feedback dependence on something that was previously said Example: 1. A: Is this flight also available on Tursday? 2. B: Yes, it's available on Thursday as well. functional dependence on 1 (2. B: On Thursday you said? feedback dependence on 1)

69. Rhetorical relations (Examples from the AMI corpus, participants discussing design of a remote control) 1. A: You keep losing them. 2. A: They easily slip behind or under the couch.

70. Rhetorical relations (Examples from the AMI corpus, participants discussing design of a remote control) 1. A: You keep losing them. 2. A: They easily slip behind or under the couch. Cause(content 2, content 1)

71. Rhetorical relations (Examples from the AMI corpus, participants discussing design of a remote control) 1. A: You keep losing them. 2. A: They easily slip behind or under the couch. Cause(content 2, content 1) 1. A: You keep losing them. 2. B: That’s because they don’t have a fixed location.

72. Rhetorical relations (Examples from the AMI corpus, participants discussing design of a remote control) 1. A: You keep losing them. 2. A: They easily slip behind or under the couch. Cause(content 2, content 1) 1. A: You keep losing them. 2. B: That’s because they don’t have a fixed location. Explain(content 2, content 1)

73. Rhetorical relations (Examples from the AMI corpus, participants discussing design of a remote control) 1. A: You keep losing them. 2. A: They easily slip behind or under the couch. Cause(content 2, content 1) 1. A: You keep losing them. 2. B: That’s because they don’t have a fixed location. Explain(content 2, content 1) 1. A: Where do you physically want to position the buttons? 2. A: I think that would have some impact on many things.

74. Rhetorical relations (Examples from the AMI corpus, participants discussing design of a remote control) 1. A: You keep losing them. 2. A: They easily slip behind or under the couch. Cause(content 2, content 1) 1. A: You keep losing them. 2. B: That’s because they don’t have a fixed location. Explain(content 2, content 1) 1. A: Where do you physically want to position the buttons? 2. A: I think that would have some impact on many things. Motivate(dialogue act 2, dialogue act 1)

75. (7.) A Methodology for designing semantic annotation schemes

76. ISO requirements for annotation standards ISO Linguistic Annotation Framework (Ide & Romary, 2005): annotations: the linguistic information that is added to segments of language data, independent of the format in which the information is represented; representations: particular formats in which annotations are rendered, (e.g. in XML, in typed feature structure AVMs, or in graphs in graphical form) independent of their content.  Standards should be formulated at the level of annotations.

77. Annotation schema design methodology 1.Conceptual analysis. Inventory of categories of concepts and relations. “Metamodel”. 2.Specification of abstract syntax of annotation language. Formal specification of types of concepts and their combinations (“annotations”) in set- theoretical terms. 3.Specification of semantics for abstract syntax. Formal specification of how to compositionally compute the meanings of annotation structures. 4.Specification of concrete syntax. Definition of concrete, e.g. XML-based, renderings of annotations in some representation format (“representations”).

78. Annotation/representation distinction: abstract/concrete syntax abstract syntaxconcrete syntax semantics

79. Ideal Concrete Syntax An ideal concrete syntax defines an ideal representation format: 1.For every annotation structure, defined by the abstract syntax, the concrete syntax defines a representation; 2.Every representation, defined by the concrete syntax, is the rendering of a unique annotation structure according to the abstract syntax.

80. Ideal concrete syntax abstract syntax ideal concrete syntax-1 semantics F 1 F 1 -1 IaIa ideal concrete syntax-2 F 2 -1 F2F2 C 12 C 21

81. DiAML abstract syntax Conceptually, an annotation structure is a set of entity structures and link structures, which connect entity structures.  Entity structures contain semantic information about a segment of primary data.  Link structures describe semantic relations between segments of primary data. Most important type of entity structure in DiAML: dialogue act structure, consisting of: - speaker S; - addressee A; - dimension D; - communicative function f or a pair with qualifier(s) q Link structures represent functional, feedback, and rhetorical relations.

82. 7. DiAML Dialogue Act Markup Language

83. DiAML example P1: What time does the next train to Utrecht leave? P2: The next train to Utrecht leaves at 8:32.

84. DiAML example - segmentation P1: What time does the next train to Utrecht leave? = functional segment fs1 P2: The next train to Utrecht leaves at 8:32.

85. DiAML example P1: What time does the next train to Utrecht leave? fs1 [Set Question, in Task dimension] P2: The next train to Utrecht leaves at 8:32.

86. DiAML example P1: What time does the next train to Utrecht leave? fs1 [Set Question] P2: The next train to Utrecht leaves at 8:32. AuFB The next train to Utrecht functional segment fs2 [Auto-Positive]

87. DiAML example P1: What time does the next train to Utrecht leave? fs1 [Set Question] P2: The next train to Utrecht leaves at 8:32. AuFB The next train to Utrecht fs2 [Auto-Positive] feedback dependence on fs1

88. DiAML example P1: What time does the next train to Utrecht leave? Fs1 [Set Question] P2: The next train to Utrecht leaves at 8:32. AuFB The next train to Utrecht = fs2 [Auto-Feedback] feedback dependence on fs1 Task The next train to Utrecht leaves I think at 8:32. = fs3 [Answer]

89. DiAML example P1: What time does the next train to Utrecht leave? = functional segment fs1 P2: The next train to Utrecht leaves I think at 8:32. AuFB The next train to Utrecht = fs2 [Auto-Feedback] feedback dependence on fs1 Task The next train to Utrecht leaves at 8:32. = fs3 [Answer] functional dependence on the Set Question in fs1

90. DiAML example 1. P1: What time does the next train to Utrecht leave? Task: fs1: What time does the next train to Utrecht leave? Set Question 2. P2: The next train to Utrecht leaves at 8:32. Task: fs2: The next train to Utrecht leaves I think at 8:32. Answer functional dependence relation to preceding Set Question AuFB: fs3: The next train to Utrecht leaves Auto-Positive feedback dependence relation to fs1

91. DiAML example, annotation P1: What time does the next train to Utrecht leave? fs1 [SetQuestion] P2: The next train to Utrecht leaves at 8:32. AuFB The next train to Utrecht fs2 [autoPositive] feedback dependence on fs1 Task The next train to Utrecht leaves I think at 8:32. fs3 [answer] functional dependence on preceding SetQuestion <dialogueAct xml:id="da1" sender="#p1" addressee="#p2" target="#fs1" communicativeFunction="setQuestion" dimension="task”/> <dialogueAct xml:id="da2" sender="#p2" addressee="#p1" target="#fs2" communicativeFunction="autoPositive” dimension="autoFeedback” feedbackDependence="#fs1"/> <dialogueAct xml:id="da3" sender="#p2" addressee="#p1" target="#fs2" communicativeFunction="answer” dimension="task” functionalDependence="#da1"/>

92. DiAML example 2 P1: What time does the next train to Utrecht leave? P2: The next train to Utrecht leaves I think at 8:32.

93. DiAML example 2 certainty qualifier P1: What time does the next train to Utrecht leave? = functional segment fs1 P2: The next train to Utrecht leaves I think at 8:32. AutoFB The next train to Utrecht = fs2 [Auto-Positive] feedback dependence on fs1 Task The next train to Utrecht leaves at 8:32. = fs3 [Answer, uncertain] functional dependence on the Set Question in fs1

94. DiAML example 2 annotation P1: What time does the next train to Utrecht leave? fs1 [Set Question] P2: The next train to Utrecht leaves I think at 8:32. AuFB The next train to Utrecht fs2 [Auto-Positive] feedback dependence on fs1 Task The next train to Utrecht leaves I think at 8:32. fs3 [Answer, uncertain] functional dependence on preceding Se tQuestion <dialogueAct xml:id="da1" sender="#p1" addressee="#p2" target="#fs1" communicativeFunction="setQuestion" dimension="task”/> <dialogueAct xml:id="da2" sender="#p2" addressee="#p1" target="#fs2" communicativeFunction="autoPositive” dimension="autoFeedback” feedbackDependence="#fs1"/> <dialogueAct xml:id="da3" sender="#p2" addressee="#p1" target="#fs2" communicativeFunction="answer” certainty=“uncertain” dimension=“task” functionalDependence="#da1"/>

95. DiAML example 3 1. P1: Do you know what time the next train to Utrecht leaves?

96. DiAML example 3 1. P1: Do you know what time the next train to Utrecht leaves? Difference with What time does the next train to Utrecht leave? [Set Question] (“direct question” versus “indirect question”)

97. DiAML example 3 1. P1: Do you know what time the next train to Utrecht leaves? Difference with What time does the next train to Utrecht leave? [Set Question] (“direct question” versus “indirect question”) Definition of Set Question; Communicative function of a dialogue act performed by a speaker S in order to know which elements of a given set have a certain property, specified by the semantic content of the dialogue act; S puts pressure on the addressee A to provide this information; S believes that at least one element of that set has that property; S assumes that A knows which elements of that set have that property.

98. DiAML example 3 1. P1: Do you know what time the next train to Utrecht leaves? Difference with What time does the next train to Utrecht leave? [Set Question] (“direct question” versus “indirect question”) Definition of Set Question; Communicative function of a dialogue act performed by a speaker S (a) in order to know which elements of a given set have a certain property, specified by the semantic content of the dialogue act; (b) S puts pressure on the addressee A to provide this information; (c) S believes that at least one element of that set has that property; (d) S assumes that A knows which elements of that set have that property. For the question ”Do you know…” condition (d) does not apply; in fact the speaker asks whether A has the requested information.

99. DiAML example 3 1. P1: Do you know what time the next train to Utrecht leaves? Difference with What time does the next train to Utrecht leave? [Set Question] (“direct question” versus “indirect question”) For the question ”Do you know…” condition (d) does not apply; in fact the speaker asks whether A has the requested information. “Do you know what time the next train to Utrecht leaves?” is semantically equivalent to: “Please tell me what time the next train to Utrecht leaves, if you know.” = conditional request: [Request, conditional]

100. DiAML example 3 1. P1: Do you know what time the next train to Utrecht leaves? is semantically equivalent to: “Please tell me what time the next train to Utrecht leaves, if you know.” = conditional request: [Request, conditional] DiAML annotation: <dialogueAct xml:id=“da1” target=“#fs1” sender=“#p1” addressee=“#p2” communicativeFunction=“request” conditionality=“conditional” dimension=“task”/>

101. DiAML example 4 P1: Do you know what time the next train to Utrecht leaves? P2: Let me see,… the next train to Utrecht leaves I think at 8:32.

102. DiAML example 4 P1: What time does the next train to Utrecht leave? fs1 [Set Question, conditional] P2: Let me see,… the next train to Utrecht leaves I think at 8:32. TimeM Let me see,… fs2 [Stalling] AuFB The next train to Utrecht fs3 [Auto-Positive] feedback dependence on fs1 Task The next train to Utrecht leaves I think at 8:32. fs4 [Answer] functional dependence on preceding Set Question

103. DiAML example 4 P1: What time does the next train to Utrecht leave? fs1 [Set Question, conditional] P2: Let me see,… the next train to Utrecht leaves I think at 8:32. TimeM Let me see,… fs2 [Stalling] TurnM Let me see,… fs2 [Take Turn] AuFB The next train to Utrecht fs3 [Auto-Positive] feedback dependence on fs1 Task The next train to Utrecht leaves I think at 8:32. fs4 [Answer] functional dependence on preceding Set Question

104. DiAML example 1. P1: What time does the next train to Utrecht leave? fs1 [Set Question, conditional] 2. P2: Let me see,… the next train to Utrecht leaves I think at 8:32. TimeM Let me see,… fs2 [Stalling] TurnM Let me see,… fs2 [Take Turn] AuFB The next train to Utrecht fs3 [Auto-Positive] feedback dependence on fs1 Task The next train to Utrecht leaves I think at 8:32. fs4 [Answer] functional dependence on preceding Set Question Annotation of turn 2: <dialogueAct xml:id="da2" sender="#p2" addressee="#p1" target="#fs2" communicativeFunction=“stalling” dimension=“timeManagement”/> <dialogueAct xml:id="da3" sender="#p2" addressee="#p1" target="#fs2" communicativeFunction=“turnTake” dimension=“turnManagement”/> <dialogueAct xml:id="da4" sender="#p2" addressee="#p1" target="#fs3" communicativeFunction="autoPositive” dimension="autoFeedback” feedbackDependence="#fs1"/> <dialogueAct xml:id="da5" sender="#p2" addressee="#p1" target="#fs4" communicativeFunction="answer” dimension=“task” functionalDependence="#da1"/>

105. 8. Conclusion, further information

106. Concluding remarks  ISO 24617-2 accepted in January 2011 as international standard.  Distinction of general-purpose versus dimension-specific function and 9 or 10 dimensions, in combination with qualifiers, functional dependence relations, feedback relations, and rhetorical relations gives ISO 24617-2 and DIT ++ the means to build annotations with a high precision; the higher levels in the communicative function hierarchies allow less fine-grained annotations.  DIT ++ Release 5 is a strictly downward compatible extension of ISO 24617-2, having the additional dimension of Contact Management and more fine- grained feedback functions.  The DiAML annotation language has a formal model-theoretic semantics semantics in terms of updates of a context model, using combinations of elementary update schemes.  ISO 24617-2 annotations can be made manually with satisfactory reliability, and can be automatically performed with high accuracy (Petukhova & Bunt, IWCS 2011).

107. Documentation Available at http://semantic-annotation.uvt.nl -ISO DIS 24617-2, July, 2010 (copyrighted); -Bunt, Harry, Harry, Jan Alexandersson, Jean Carletta, Jae-Woong Choe, Alex Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David Traum “Towards an ISO standard for dialogue act information”, Proceedings LREC 2010 - Petukhova, Volha and Harry Bunt, “The independence of dimensions in multimodal dialogue annotation”. In Proceedings NAACL-HLT 2009 - Ide, Nancy and Harry Bunt (2010) “Anatomy of semantic annotation schemes: Mappings to GrAF”. In Proceedings of the 4th Linguistic Annotation Workshop (LAW-IV), Uppsala. ISOcat data category registry: http://www.isocat.org

108. References Bunt, Harry (2010) “A methodology for designing semantic annotation languages.” In Proceedings of the 2nd International Conference on Global Interoperability for Language Resources (ICGL-2), Hong Kong, pp. 29-46. Bunt, Harry (2011) “Multifunctionality in dialogue.” Computer, Speech and Language 25, 225-245. Bunt, Harry (2011) “The semantics of dialogue acts.” In Proceedings IWCS 2011, the 9th International Conference on Computational Semantics. Oxford. Petukhova, Volha and Laurent Prévot and Harry Bunt (2011) “Multilevel discourse relations between dialogue units.” In Proceedings ISA-6, the 6th International Workshop on Interoperable Semantic Annotation. Oxford. Petukhova, Volha and Harry Bunt (2011) “Incremental dialogue understanding.” In Proceedings IWCS 2011, the 9th International Conference on Computational Semantics. Oxford.