1. Completions and continuations in dialogue: a preliminary account Massimo Poesio (Uni Essex) Hannes Rieser (Uni Bielefeld) CATALOG Barcelona, July 2004

2. Sentence cooperations: an example Inst: So, jetzt nimmst Du Well, now you grasp Cnst:eine Schraube a screw. Inst:eine orangene mit einem Schlitz. an orange one with a slit Cnst:Ja. Yes.

3. Sentence cooperations: an informal definition (Clark, 1996) SENTENCE COOPERATION: At least two dialogue participants contribute to a sentence production COMPLETION: sub-sentential structure continued by obligatory constituents CONTINUATION: material added to already `complete’ sentence

4. The significance of sentence cooperations Clear evidence that dialogue requires coordination at the sub-sentential level (see also Pickering and Garrod, in press) Provide insights into incrementality and compositionality issues A tool to investigate competing claims about coordination in dialogue Purely intentional models Pickering and Garrod’s IAM based on simpler alignment mechanisms.

5. Outline of the talk Sentence cooperations in the Bielefeld Toy Plane Corpus (BTPC) An introduction to PTT (Poesio and Traum 1997, 1998; Matheson et al, 2000; Poesio, to appear) Intro to model of we-intentions we use, following Bratman (1992), Tuomela (2000) and Grosz and Kraus (1996) A PTT implementation of an intentional analysis of completions (May have time to sketch an IAM analysis)

6. The Bielefeld Toy Plane Corpus

7. 22 video-filmed, speech recorded and transcribed dialogues two agents, Instructor and Constructor constructing a “Baufix” airplane different sight conditions: total screen, half-screen, face to face 3675 contributions 160 sentence cooperations (4.34 %) in most of them cooperation other-initiated (95%)

8. Sentence cooperations in the BTPC (Skuplik, 1999) 126 sentence cooperations from the BTPC 54 completions (43%) 72 continuations (57%) Production of the completion / continuation: 79% Cnst, 21% Inst 84% of compl. / contin. accepted by previous speaker (41% implicitly) Release-turn signalled in 31% of cases

9. A few other observations Completions become more frequent as dialogue procedes (routinization?)

10. The example, revisited Inst: So, jetzt nimmst Du Well, now you grasp Cnst:eine Schraube a screw. Inst:eine orangene mit einem Schlitz. an orange one with a slit Cnst:Ja. Yes. CNST COMPLETION (70%) WITH AN OBLIGATORY NP RESULTING IN A SENTENCE WHEN MERGED SIGNALED BY LENGHTENING OF “DU”, LEVEL TONE

11. The example A CNST COMPLETION (70%) By way of an OBLIGATORY NP (a screw ) (30%) making up A SENTENCE when merged with Inst’s contribution (50%) A REQUEST TO CHANGE SPEAKER signaled by prosodic means (lengthening of Du, level tone)

12. Outline of the talk The Bielefeld Toy Plane Corpus (BTPC) An introduction to PTT (Poesio, 1995; Poesio and Traum 1997, 1998; Matheson et al, 2000; Poesio, to appear) Intro to model of we-intentions we use, following Bratman (1992), Tuomela (2000) and Grosz and Kraus (1996) A PTT implementation of an intentional analysis of completions

13. PTT A theory of semantics and interpretation in dialogue originally motivated by work on the TRAINS project Key characteristics: Building on (Compositional) DRT (Muskens, 1996) Common ground as a record of the discourse situation (Barwise and Perry, 1983) An account of incremental semantic interpretation An account of GROUNDING So far, primarily concerned with aspects of dialogue driven by obligations

14. Common ground: beyond assertion A They have at their disposal enormous assets // and their policy B //look can I just come in on that// last year A //YES IN A MINUTE IF YOU MAY AND WHEN I’M FINISHED // then you’ll know B // yes I’M SO SORRY (Coulthard 1977)

15. Common ground: beyond assertion B: Go to Elmhurst, pass the courthouse and go to Elmhurst and then to Elmhurst, uh north. A: mm hum. B: Towards Riverton, till you come to that Avila Hall A: Oh yes B: Dju know where that//is? A: //uh huh A: Oh surely B: Avilla Hall on the corner of Bor//don A: //uh huh B: Well there, on Bordon you turn back to town, left. (George Psathas, "Direction-giving in Interaction," in Boden and Zimmerman, ed.)

16. From DRT to PTT a. A: There is an engine at Avon. b. B: It is hooked to a boxcar DRT: [ x,w,y,u,s,s’| engine(x), Avon(w), s: at(x,w), boxcar(y), s’:hooked-to(u,y), u=x]

17. Common ground and discourse situation in PTT [ce1,ce2,K1,K2| K1=[x,w,s| engine(x), Avon(w), s: at(x,w)], ce1: assert(A,B,K1) K2=[y,z,s’| boxcar(y), s’:hooked-to(z,y), z=x], ce2: assert(B,A,K2)]

18. Locutionary acts in the common ground “The fact that a speaker is speaking, saying the words he is saying in the way he is saying them, is a fact that is usually accessible to everyone present. Such observed facts can be expected to change the presumed common background knowledge of the speaker and his audience in the same way that any obviously observable change in the physical surroundings of the conversation will change the presumed common knowledge.” (Stalnaker, Assertion, p. 323)

19. The time-order of sentence processing GARDEN-PATH phenomena shows that parsing is INCREMENTAL (Bever, 1974; Frazier, 1987) Marslen-Wilson 1973, 1975: semantic information ALSO accessed immediately Swinney, 1979: lexical access incremental Just and Carpenter,1980: IMMEDIACY HYPOTHESIS (“Every word encountered should be processed to the deepest level possible before the eye moves on to the next word”) Eye-tracking work (Tanenhaus et al, 1995, tomorrow): really fine-grained incrementality

20. Alignment at all levels Pickering & Garrod

21. Clarification questions (Ginzburg and Cooper, Purver and Ginzburg) A: Did Bo leave? B: BO? A: Bo Smith. B: Yes, half an hour ago. Matthew: It wasn’t all that bad. At least the pool was clean. Lara: MR POOL? Matthew: The pool. Lara: Oh. (“What is the intended content of your utterance ‘Bo’?”) (“Did you utter the words ‘Mr. Pool’?”)

22. Micro conversational events (Poesio, 1995) boxcar  [u|u:utter(A,”boxcar”), Noun(u), sem(u)= x [|boxcar(x)], + SYN INFO (NEXT)] umm  [u,ce| u: utter(A,”umm”), ce: keep-turn(A), generate(u,ce)]

23. MCEs in the example dialogue [mce6| mce6:utter(Cnst,"Schraube"), Noun(mce6), sem(mce6)= v([ |screw(v)]]; [mce1,ce1| mce1:utter(Inst,“so"), Adv(mce1), ce1:take-turn(Inst), generate(mce1,ce1)]; [mce2,ce2| mce2:utter(Inst,“jetzt"), Adv(mce2), ce2:keep-turn(Inst), generate(mce2,ce2)]; [mce3| mce3:utter(Inst,"nimmst"), Verb(mce3), sem(mce3)= Q x(Q( x’[e| e: grasp(x, x’)]))]; [mce4| mce4:utter(Inst,"Du"), Pro(mce4), sem(mce4)= P.P (you)]; [mce5| mce5:utter(Cnst,"eine"), Det(mce5), sem(mce5)= P’ P([y| ]; P’(y); P(y))]

24. Syntactic interpretation with MCEs (Poesio, 1996) MCE1 CE1 mce1:utter(Inst,“so"), ce1:take-turn(Inst), generate(mce1,ce1)]; MCE2 CE2 U1:S MCE3:”nimmst”:V U2:NP U3:NP MCE3 MCE4:”Du”:Pro U4:NP MCE4

25. Syntactic Interpretation with MCEs, II MCE1 CE1 mce1:utter(Inst,“so"), ce1:take-turn(Inst), generate(mce1,ce1)]; MCE2 CE2MCE3 U1:S MCE3:”nimmst”:V U2:NP U3:NPMCE4:”Du”:Pro MCE4

26. Semantic interpretation and compositionality BINARY SEMANTIC COMPOSITION U3:  (  ) U1:  U2: 

27. Intentions and obligations OBLIGATIONS: [o | o:Obl Cnst ([|address(Cnst, ce1)])] INTENTIONS: [i | i:Int Inst&Cnst ([|join(Cnst, wing1,fuselage1)])] INTENTIONAL STRUCTURE: Grosz&Sidner-like sp(i1) = i2 dom(i1) = i2

28. Grounding As in proposals such as Clark and Schaefer (1989) and Traum (1994), establishment of common ground (‘G’) modeled in terms of CONTRIBUTIONS, or DISCOURSE UNITS, that may be ACKNOWLEDGED or REPAIRED

29. Discourse Units and Grounding Acts MCE1 CE1 mce1:utter(Inst,“so"), ce1:take-turn(Inst), generate(mce1,ce1)]; MCE2 CE2MCE3 … U1:S MCE3:”nimmst”:V U2:NP U3:NPMCE4:”Du”:Pro DU1 …. DU17 DU17 = ACK(DU17) CONT(DU17) REPAIR(DU17)

30. MCEs in the example dialogue (simplified) [mce1,ce1| mce1:utter(Inst,“so"), Adv(mce1), ce1:take-turn(Inst), generate(mce1,ce1)]; [mce1| mce1:utter(Inst,"nimmst"), Verb(mce1), sem(mce1)= Q x(Q( x’[e| e: grasp(x, x’)]))]; [mce2| mce2:utter(Inst,"Du"), Pro(mce2), sem(mce2)= P.P (you)]; [mce3| mce3:utter(Inst,"eine"), Det(mce3), sem(mce3)= P’ P([y| ]; P’(y); P(y))] [mce4| mce4: utter(Inst, “orangene”), Adj(mce4), sem(mce4) = P z([ |orange(z)]; P(z))]; [mce5| mce5:utter(Const,"Schraube"), Noun(mce5), sem(mce5)= v([ |screw(v)]];

31. The dynamics of discourse situations, I [AX-SP-SDRT] Let p, r be core speech acts; K1 and K2 DRSs. Then  ce1,ce2,A,B,C,D,K1,K2,f,g [ | ce1: p(A,B,K1), ce2: r(C,D,K2), sp(ce2)=ce1] (f,g)  (  i, j,k [ce1,ce2| ] (f,i)  K1(i,j)  K2(j,k)  [|sp(ce2)=ce1](k,g))

32. The dynamics of discourse situations, II [AX-DOM-MCE]  u, u’, u’’, f, g, [ | u’  u, u’’  u, sem(u)=   1..  m , sem(u’)=   ’1..  ’n , sem(u’’)=   ’’1..  ’’p  ](f,g)  (  i,j, x 1 …. x m,  ( x 1 )(x 2 )…. (x m )(i,j)  (  k,l,y 1 … y n, z 1.. z p, i  k  l  j   (y 1 )… (y n ) (i,j)   ( z 1 )…( z p ) (k,l)))

33. Anaphoric expressions and resource situations the  [u|u:utter(A,”the”), Det(u), sem(u)= K P P’ [ x | ]; [ | x =  y. K; P(x)]; P’(x) sem(the boxcar)  K P’ [ x | ]; [ | x =  y. K; [|boxcar(x)] ]; P’(x)

34. What prompts the completion? Two accounts Intentional account Need to explain why help Alignment account What representation is aligned? HR

35. Outline of the talk The Bielefeld Toy Plane Corpus (BTPC) An introduction to PTT (Poesio and Traum 1997, 1998; Matheson et al, 2000; Poesio, to appear) Intro to model of we-intentions we use, following Bratman (1992), Tuomela (2000) and Grosz and Kraus (1996) A PTT implementation of an intentional analysis of completions

36. Public and private (partial) plans Inst has a fully specified plan for building the toy-airplane (drawing or model) The public plan among Inst and Cnst is usually underspecified, but gets more refined throughout the construction dialogue Besides the shared partial plan, the agents have private plans which overlap to some extent with the shared plan. The difference between the public plan and the private plans leads to discrepancies and negotiations

37. The partial shared plan before the example Join wing and fuselage Assemble toy airplane Assemble fuselageAssemble wing get 5h bar get 3h bar get 7h bar1 get 7h bar2 join align wing& fuselage get bolt put through get bolt get nut

38. We-intentions for shared cooperative activity Tuomela’s (2000) modification of Bratman’s 1993 definition of SCA, adapted: Inst and Cnst WE-INTEND that Cnst join wing and fuselage is equivalent to: It is Inst’s and Cnst’s mutual knowledge that Inst intends that Cnst join wing&fuselage because Cnst intends that Cnst join wing&fuselage and Cnst intends that Cnst join wing&fuselage because Inst intends that Cnst join wing&fuselage

39. Tuomela’s definition of we-intention, formal Int Inst&Cnst (join(Cnst, W&F))  MK((Int Inst join(Cnst, W&F)) / r Int Cnst (join(Cnst, W&F))) and (Int Cnst join(Cnst, W&F)) / r Int Inst (join(Cnst, W&F)))) (where / r is the reason relation, which is factual)

40. Outline of the talk The Bielefeld Toy Plane Corpus (BTPC) An introduction to PTT (Poesio and Traum 1997, 1998; Matheson et al, 2000; Poesio, to appear) Intro to model of we-intentions we use, following Bratman (1992), Tuomela (2000) and Grosz and Kraus (1996) A PTT implementation of an intentional analysis of completions

41. The situation before example 1 MP

42. The partial shared plan before the example Join wing and fuselage Assemble toy airplane Assemble fuselageAssemble wing get 5h bar get 3h bar get 7h bar1 get 7h bar2 join align wing& fuselage get bolt put through get bolt get nut

43. Reaching the intention to perform a directive 2.: intertwined discourse / domain plan join(Cnst,Obj1,Obj2) > Ash&Morr b & c & d & e & f, (b) 1. direct(Inst, Cnst, grasp(Cnst, Bolt)), 2. grasp(Cnst, Bolt), 3. tell(Cnst, Inst, grasp(Cnst, Bolt )) (c) 1.direct(Inst, Cnst, grasp(Cnst, Nut)), 2. grasp(Cnst, Nut), 3. tell(Cnst, Inst, grasp(Cnst, Nut)) (d) …put through… (e) …fasten.. (f) ….feedback. 1.: (partial) we-intention to join: [i | i:Int Inst&Cnst ([|join(Cnst, wing1,fuselage1)])]

44. Deciding to perform a directive, II 5. Achieve partial Inst intention (& Cnst intention) [i2 | i2:Int Inst ([K1,ce1|K1 = [e, x|bolt(x), e:grasp(Cnst,x)] ce1:direct(Inst, Cnst, K1)])] 3. Distributivity of we-intention  4. Achieve partial we-intention to perform directive: [i1 | i1:Int Inst&Cnst ([K1,ce1|K1 = [e,x|bolt(x), e:grasp(Cnst,x)] ce1:direct(Inst, Cnst, K1)])]

45. Inst’s private plan Join wing and fuselage Assemble toy airplane Assemble fuselageAssemble wing get 5h bar get 3h bar get 7h bar1 get 7h bar2 join align wing& fuselage get bolt … orange-bolt- with-slit

46. Cnst’s private plan Join wing and fuselage Assemble toy airplane Assemble fuselageAssemble wing get 5h bar get 3h bar get 7h bar1 get 7h bar2 join align wing& fuselage get bolt ….. get bolt bolt

47. Planning the directive 7. develop plan to perform utterance that generates directive: [i4 | i4:Int Inst ([u1.1 | utterance(u1.1), sem(u1.1) = K2, generates(u1.1,ce2)])] 6. arrive at more specific intention (evidence: subsequent repair) [i3 | i3:Int Inst ([K2,ce2|K2 = [e, x|bolt(x),x=orange-slit-bolt, e:grasp(Const,x)] ce2:direct(Inst, Cnst, K2)])]

48. Micro-plan 8. plan to perform utterance in terms of MCEs [i5a | i5a:Int Inst ([u1.2 | u1.2: „so“:take-turn])]; 9. lengthening signals problem - Inst doesn’t necessarily know which bolts are unused  possibly does not know how to refer to bolt (NP type / content) [i5b | i5b:Int Inst ([u1.3 | u1.3:“jetzt“:keep-turn])]; [i5c | i5c:Int Inst ([u1.4.. u1.7| S(u1.1), u1.4:“nimmst“:V, u1.5:“Du“:NP, u1.7:VP, NP(u1.6), u1.5  u1.1, u1.7  u1.1, u1.4  u1.7, u1.6  u1.7]

49. Possible motivations for the completion Interpret lengthening as request to continue obl(Cnst, cont(DU)) Interpret lengthening as request for acknowledgment (also standard PTT) obl(Cnst, ack(DU)) Cooperativeness `Blurting out’

50. A cooperativeness analysis 10. Cnst acquires intention to turn the directive in a joint action (cfr. Tuomela’s “unrequired contributory actions”) [i6 | i6:Int Cnst ([K3,ce3|K3 = [e,x|bolt(x), e:grasp(Const,x)] ce3:direct(Inst&Cnst, Cnst, K3)])] Derivation of 10 not axiomatized by Tuomela, but we assume here is the result of an intention to help. 11. Cnst produces plan to perform action to generate directive; analogous to Inst’s, but content (partial) K3: [i7 | i7:Int Cnst ([u1.1a | utterance(u1.1a), sem(u1.1a) = K3, generates(u1.1a,ce3)])]

51. Further specification not possible

52. Micro-plan for Cnst’s completion 12. most of actions in plan already performed by Inst; Cnst plans missing action [i8 | i8:Int Cnst ([ K1.1d | u1.6:“eine Schraube“:NP, K1.1d=[x1|bolt(x1)] sem(u1.6) = K1.1d ])] (Same action would be planned to continue contribution and to acknowledge) 13. Instructor begins repair due to his private plan

53. An alternative analysis: the IAM model Successful dialogue involves the development of aligned representations at all levels Aligned representations the result of priming mechanisms at every level of linguistic representation Mental state reasoning an option but not basic

54. Alignment at all levels

55. An IAM-analysis of the BTPC example What leads Cnst to produce “eine Schraube”? What is a situation model in this domain? “the key dimensions encoded in situation models are SPACE, TIME, CAUSALITY, INTENTIONALITY, and REFERENCE to the MAIN INDIVIDUALS … “ (p. 7)

56. Two possible views of the situation model in BTPC domain The partial shared plan (cfr. earlier analysis) Cfr. mention of intentionality Would make the IAM model much closer to the model presented earlier The state of Cnst’s plane assembly

57. The ‘state of assembly route’ Problems with this type of dialogues: Situation models are clearly NOT aligned (they will only be so at the END of the conversation) Cannot assume implicit common ground Need to extend model to deal with directives

58. A proposal Based on the notion of priming Extend the notion of “routine” to non- linguistic actions During the dialogue, an AGGREGATE FORMATION ROUTINE gets established

59. Development of the AF routine JOIN(TAIL, FUSELAGE) JOIN(SIDE-RUDDER, FUSELAGE)

60. The routine PARAMETERS: 2 constituents to be joined through a port a fixing mechanism going through this port (currently underspecified) ROUTINE FOR AGGREGATE FORMATION: align material to be joined (requires port identif.) obtain fixing mechanism put fixing mechanism through port fasten fixing mechanism yields: aggregate

61. Generation of “eine Schraube” Based not on global shared plan, but on local instantiation of AF-routine, primed in context Only one parameter is still underspecified: the bolt Local execution of routine at the ‘obtain fixing mechanism’ point Syntactic structure to realize action presumably also available through routine

62. Open problems with this account Some motivation for completion needed More in general, how is helping done? (Normal cooperativity axioms are based on beliefs & intentions.) Also need to say something more about the choice of that particular realization Economy principle?

63. Preliminary conclusions PTT provides the technical tools to formalize a crucial feature of sentence cooperations: coordination at the micro conversational event level Mind-reading always difficult, but Tuomela’s theory of we-intention goes some way towards formalizing one of the possible motivations for completions, in terms of “help” A preliminary investigation of the alignment route also possible