2. Gu Dialogue Systems Lab 1 Issue-based Dialogue Management in GoDiS Staffan Larsson Dialogsystem HT 2004

3. Gu Dialogue Systems Lab 2 Overview 1.Goals 2.Basic issue-based dialogue management 3.Grounding and feedback 4.Adressing Unraised Issues 5.Action-oriented Dialogue 6.Multilinguality 7.Conclusions

4. Gu Dialogue Systems Lab 3 1. Goals Explore and implement issue-based dialogue management –starting from a theory of dialogue semantics (by Jonathan Ginzburg) –based on notions of DGB (Dialogue Gameboard) QUD (Questions Under Discussion) Update Protocols –adapt this theory to dialogue system (GoDiS) and implement it using TrindiKit –extend theory coverage, taking in relevant theories General theory of dialogue –minimize effort for adapting dialogue system to new domains

5. Gu Dialogue Systems Lab 4 Goals cont’d Incrementally extend theory and system to handle increasingly complex types of dialogue –clarifies relation between dialogue genres –promotes reuse of update rules Some dialogue phenomena covered –Multiple tasks, information sharing between tasks –Feedback and grounding –Accommodation, re-raising, clarification –Menu based action oriented dialogue –Multi-linguality & multiple domains

6. Gu Dialogue Systems Lab 5 information state –an abstract data structure (record, DRS, set, stack etc.) –accessed by modules using conditions and updates dialogue moves are associated with IS updates using IS update rules –rule name and class –preconditon list: conditions on TIS –effect list: updates on TIS dialogue move engine –module or group of modules responsible for –updating the IS based on observed moves –selecting moves to be performed Information State Approach

7. Gu Dialogue Systems Lab 6 input inter- pret TIS DATABASE LEXICON DOMAIN data- base control updateselect gene- rate output lexicon domain knowledge DME

8. Gu Dialogue Systems Lab 7 TrindiKit GoDiS GoDiS-I GoDiS-A Travel Agency Auto- route Xerox manual VCR manager IBDM home device manager IS approach genre- specific applicatio n-specific

9. Gu Dialogue Systems Lab 8 2. Issue-based dialogue management Inquiry-oriented dialogue (database search) basis: –Ginzburg’s Dialogue Gameboard (DGB) and –related DGB update protocols dialogue moves: ask, answer, greet, quit raising and addressing issues –incl. short answers. e.g.”yes”, ”no”, ”paris”, ”in april” dialogue plans sample domain: travel agency extension: –reraising issues –handling multiple issues

10. Gu Dialogue Systems Lab 9 Semantics Simple First Order Logic without quantifiers, but with questions Questions –Y/N-questions: ?P, P is a proposition –wh-questions: ?x.p(x) (p is a predicate) ? works much like like –alt-questions: {?P1, …, ?Pn} Content of short answers –individual markers: paris, april, … –yes, no

11. Gu Dialogue Systems Lab 10 Semantics, cont’d Q-A relations (adapted from Ginzburg) –resolves(A,Q): A resolves Q dest-city(paris) resolves ?x.dest-city(x) –relevant(A,Q): A is relevant to Q (about Q) not(dest-city(paris)) is relevant to ?x.dest-city(x), but does not resolve it

12. Gu Dialogue Systems Lab 11 Basic GoDiS information state record type QUD: Questions which have been raised but not resolved, and which are available for resolution of short answers PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) COM : set( Prop ) QUD : stackset( Question ) LU: SPEAKER: Speaker MOVES: OpenQueue( Move )

13. Gu Dialogue Systems Lab 12 Sample dialogue plan <findout(?x.dest-city(x)) findout(?x.depart-city(x)) findout(?x.transport(x)) findout(?x.dept-month(x)) findout(?x.dept-day(x)) raise( {?class(economy), ?class(business)} consultDB(?x.price(x)) >

14. Gu Dialogue Systems Lab 13 Answer integration RULE: integrateAnswer PRE: in( $ /SHARED/LU/MOVES, answer(A) ) fst( $ /SHARED/QUD, Q ) $ DOMAIN :: relevant( A, Q ) EFF: ! $ DOMAIN : combine( Q, A, P ) add( / SHARED/COM, P ) Before an answer can be integrated by the system, it must be matched to a question on QUD

15. Gu Dialogue Systems Lab 14 Basic dialogue with updates U: ”price information please”; raises price issue –if user asks Q, push Q on QUD and push respond(Q) on AGENDA –if respond(Q) on AGENDA and PLAN empty, find plan for Q and load to PLAN –if findout(Q) first on PLAN, ask Q S: ”where do you want to go?” U: ”Paris” –if LU/MOVES contains answer(A) and A relevant to Q, add P=Q[A] to SHARED.COM –if P in SHARED.COM and Q topmost on QUD and P resolves Q, pop QUD –if P in SHARED.COM and P resolves Q and findout(Q) on PLAN, pop PLAN

16. Gu Dialogue Systems Lab 15 Basics cont’d … S: ”Do you want economy class or business class?” U: ”economy class” –if consultDB(Q) on PLAN, consult database for answer to Q; store result in PRIVATE.BEL –if Q on QUD and P in PRIVATE.BEL s.t. P resolves Q, answer(P) S: ”The price is £123”

17. Gu Dialogue Systems Lab 16 Information sharing across plans GoDiS does not keep track of when propositions were added, or which plan was being executed so information sharing is determined by question sharing across plans plan for VISA question: findout(?x.dest-city(x)) findout(?x.citizenship(x)) –shares a questions with plan for ?x.price(x) so if visa-issue raised after price-issue, no need to ask for destination again

18. Gu Dialogue Systems Lab 17 Dealing with multiple open issues if user asks Q, push Q on QUD and load plan for dealing with Q if users asks Q’ while system is dealing with Q, throw out plan for Q but Q remains on QUD; load plan for Q’ when Q’ resolved, Q topmost on QUD will trigger reloading plan for dealing with Q –general rule: if SHARED.COM contains info resolving Q, don’t ask Q –so any resolved questions in plan will be thrown out

19. Gu Dialogue Systems Lab 18 Sample dialogue: multiple tasks & info sharing S> Welcome to the travel agency! U> price information S> (…) Lets see. How do you want to travel? U> by flight S> (…) What city do you want to go to? U> paris S> (…) What city do you want to go from? U> do I need a visa ? S> (…) Lets see. What country are you from? U> sweden S> Okay. Yes, you need a Visa. S> Returning to the issue of price. Lets see. What city do you want to go from?

20. Gu Dialogue Systems Lab 19 A problem with QUD A: Who won the world cup? B: Where? A: in Italy B: Are you sure it wasn’t in France? A: Yes B: Where did you say it was again? A: Italy B: Germany If QUD= and q1 is resolved, q2 is available for resolution of short answers –takes no account of how many turns since q2 was raised –but short answers a long distance away from the question are not as easily processed as an adjacent answer

21. Gu Dialogue Systems Lab 20 ISSUES and QUD We extend Ginzburg’s DGB by adding ISSUES of type Stack(Question) ISSUES contains all raised but unresolved questions –ISSUES determines relevance of user answers QUD used for resolving short answers –questions drop off QUD after N turns –a short answer to a question that’s on ISSUES but not QUD requires adjusting QUD by copying a question on ISSUES

22. Gu Dialogue Systems Lab 21 3. Grounding and feedback ICM –Interactive Communication Management Grounding –making sure that the participants are percieving, understanding, and accepting each other’s utterances –dealing with problematic situtations where e.g. an utterance is not percieved Feedback moves –(short) utterances which signal grounding status of previous utterance Sequencing moves –utterances which signal switching task (”returning to…”), reraising questions (”so…”) etc.

23. Gu Dialogue Systems Lab 22 Grounding moves We want a typology of feedback moves Some parameters: –Action levels in dialogue (Allwood, Clark) contact (con) perception (per) understanding –Word meanings (sem) –Contextual meaning (und) acceptance (acc) –Polarity Positive (pos) Negative (neg) Checking (chk / int)

24. Gu Dialogue Systems Lab 23 Some grounding moves in GoDiS Formal representation –icm:Level/Type{*Polarity}{:Content} Feedback moves –icm:und*neg – ”I don’t understand” –icm:und*pos:P – ”To Paris.” –icm:acc*neg:Q – ”Sorry, I can’t …” –icm:acc*pos – ”Okay” Feedback type selected depending on –Quality of recognised speech –Whether system can find a (relevant) interpretation –Whether system can accept what’s been said Sequencing moves –icm:reraise:Q – ”Returning to the issue of Q” –icm:loadplan – ”Let’s see…”

25. Gu Dialogue Systems Lab 24 Grounding on multiple levels S> Lets see. What can I do for you? U> (mumble mumble) S> Sorry, I didnt hear what you said. What can I do for you? U> siridus S> I heard you say siridus. Sorry, I dont understand. What can I do for you? U> channel one S> channel one. I dont quite understand. Do you want to add a program or change channel? (dialog3a_edit) Negative perception feeback Positive perception Negative understanding (sem) Positive understanding (word meaning) Negative understanding (und) Task clarification

26. Gu Dialogue Systems Lab 25 U> turn on the t v S> turn on the TV… Sorry, I cannot control the TV set. Do you want to add a program or change channel?” U> pardon S> Sorry, I cannot control the TV set. Do you want to add a program or change channel? U> change channel S> Okay. (dialog3b_edit) Rejection (negative acceptance feedback) Positive understanding User negative perception System repetition Positive acceptance

27. Gu Dialogue Systems Lab 26 Correcting the system S> Lets see. What can I do for you? U> add program S> Okay. Add a program… S> Lets see. What channel do you want? U> channel six S> okay, channel five. U> no S> not channel five. So, What channel do you want? U> channel six S> channel six?” (dialog4_edit) Positive feedback Negative answer to fb. Signals reraising (Sequencing ICM)

28. Gu Dialogue Systems Lab 27 4: Addressing Unraised Issues (QUD and ISSUES) General strategy: question accommodation issue accommodation QUD accommodation dependent issue accommodation issue reacommodation

29. Gu Dialogue Systems Lab 28 Typical human-human dialogue S(alesman), C(ustomer) S: hi C: flights to paris S: when do you want to travel? C: april, as cheap as possible...

30. Gu Dialogue Systems Lab 29 Accommodation Lewis (1979): If someone says something at t which requires X to be in the conversational scoreboard, and X is not in the scoreboard at t, then (under certain conditions) X will become part of the scoreboard at t Has been applied to referents and propositions, as parts of the conversational scoreboard / information state

31. Gu Dialogue Systems Lab 30 Question accommodation If questions are part of the information state, they too can be accommodated If the latest move was an answer, and there is an action in the plan to ask a matching question, then –put that question on ISSUES –(and QUD if it is a short answer) Requires that the number of possible matching questions is not too large –(or can be narrowed down by asking clarification question)

32. Gu Dialogue Systems Lab 31 issue accommodation PLANISSUES If –LM=answer(A) –no Q in ISSUES s.t. relevant(A,Q) then –find findout(Q) in PLAN s.t. relevant(A,Q) –push Q on ISSUES used when prevously unraised question (available in plan) is answered using a short or full answer

33. Gu Dialogue Systems Lab 32 QUD accommodation ISSUESQUD If –LM=answer(A) –A is a short answer –no Q in QUD s.t. relevant(A,Q) then –find Q in ISSUES s.t. relevant(A,Q) –push Q on QUD –raise Q in ISSUES (make Q topmost) used when –previously raised question has dropped off QUD, but is answered using a short answer –previously unraised question is answered using short answer [needs PLANISSUES accommodation]

34. Gu Dialogue Systems Lab 33 dependent issue accommodation DOMAIN  ISSUES (+PLAN) If –LM=answer(A) –no Q in ISSUES s.t. relevant(A,Q) –no findout(Q) in PLAN s.t. relevant(A,Q) then –find Plan for some Q’ in DOMAIN s.t. findout(Q) or raise(Q) in Plan and relevant(A, Q) –push Q’ on ISSUES –set PLAN to Plan used when previously unraised question, unavailable in PLAN, is answered using full or short answer (AKA ”task accomodation”)

35. Gu Dialogue Systems Lab 34 Sample dialogue: accommodation S: Welcome to the travel agency. U: From London to Paris in April –not relevant to any question that has been raised, or to any current plan –look in domain knowledge for a plan (for dealing with some question Q) with matching questions –load this plan, push Q on ISSUES –find in the plan the question(s) matching the user’s answer, and push them onto ISSUES –integrate answer (requres matching question on ISSUES) S: Alright, you want to know about price. (…) –proceed to next plan item S: How do you want to travel? –ISSUES=

36. Gu Dialogue Systems Lab 35 Task clarification (VCR application) S> What can I do for you? U> channel five S> channel five. I dont quite understand. Do you want to add a program or change channel? U> change channel S> Okay. change channel. S> Lets see. channel five? U> yes S> The channel was changed

37. Gu Dialogue Systems Lab 36 Issue reraising SHARED.COM  ISSUES If –LM=ask(Q) –there is a P in SHARED.COM s.t. relevant(P,Q) then –push Q on ISSUES –remove P from SHARED.COM used when previously resolved question is asked again reraising should be inicated; ”so,…”; reformulation may be needed

38. Gu Dialogue Systems Lab 37 Issue reaccommodation SHARED.COM  ISSUES If –LM=answer(A) –no Q in ISSUES s.t. relevant(A,Q) –P in SHARED.COM s.t. there is a Q s.t. relevant(A,Q) and relevant(P,Q) then –push Q on ISSUES –remove P from SHARED.COM Used when previously resolved question is answered again

39. Gu Dialogue Systems Lab 38 Revision by reaccommodation and reraising S> What can I do for you? U> add program channel five S> Okay. Do you want to add a program? U> yes S> Lets see. channel five? U> yes S> What date? U> channel six S> Okay. channel six… What date?

40. Gu Dialogue Systems Lab 39 5: Issues and goals in Action- Oriented Dialogue (AOD) each goal action associated with a dialogue plan add field –SHARED.ACTIONS : StackSet(Action) New moves: request(Action), report(Action, Status) ACTIONS has a similar role to ISSUES adapt accommodation strategies to AOD

41. Gu Dialogue Systems Lab 40 AOD information state PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stackset( Question ) ACTIONS : stackset( Action ) LU:… ISSUES: stackset(Question)

42. Gu Dialogue Systems Lab 41 Requests vs. answers requests adresses a general question –”what shall I do next?”, ”what can I do for you?” or similar (”prompt”) –semantics in AOD: ?x.action(x) dialogue –”What can I do for you?” raises?x.action(x) –”Search the phonebook” interpreted as request(search_phonebook) Rule: –If ?x.action(x) is topmost on ISSUES and L- M is request(a) –then pop ISSUES

43. Gu Dialogue Systems Lab 42 Action Oriented Dialogue subtypes ”Command dialogue” –user instructs system to perform actions device control; menu-based travel agency booking ”Instructional dialogue” –system instructs user to perform actions ”interactive manual” –system can explaing how to perform some action joint planning –user and system jointly agree to a plan –also involves negotiation

44. Gu Dialogue Systems Lab 43 input nuance inter- pret Information State LEXICON DOMAIN DEVICES lexicon vcrphone control updateselect gene- rate output nuance device phone domain vcrphone device vcr lexicon home domain home rel1 rel2 dim10 VCR device alert

45. Gu Dialogue Systems Lab 44 Converting menus to plans MenuPlan multi-choice list P 1, P 2,..., P n findout({?P 1,?P 2,...,?P n }) tick-box +/-Pfindout(?P) dialogue window p=___ findout(?x.p(x)) pop-up message M answer(M)

46. Gu Dialogue Systems Lab 45 Menu-based AOD S: Welcome to the telephone and VCR manager! U: Channel one today S: Right, I assume you want to add a program. Let’s see. Okay. Channel one. Today. What time do you want to start recording? S: What time is it? U: It’s one forty five S: Set the clock to two forty five U: Okay, set the clock. Two forty five, is that correct? S: Yes. U: Okay. The clock was set. Returning to timer recording. So, what time do you want to start recording? U: eight thirty to nine thirty S: Okay. Starting at eight thirty. Ending at nine thirty. The program has been added.

47. Gu Dialogue Systems Lab 46 Task and question accommodation in AOD S> Lets see. What can I do for you? U> channel five today S> Alright. add a program… Lets see. Okay. channel five… today… What time do you want to start recording? U> from six thirty to seven thirty S> Okay. from six thirty… to seven thirty… The program has been added. ” Signals accommodation (sequencing ICM)

48. Gu Dialogue Systems Lab 47 7. Multilinguality and multiple domains Language and domain resources can be switched on the fly Does not affect the rest of the system Thanks to modularity structure of system

49. Gu Dialogue Systems Lab 48 Multilinguality S> Lets see. What can I do for you? U> add program S> Okay. Do you want to add a program? U> yes S> Lets see. What channel do you want? U> svenska S> Okej. S> Gå tillbaks till lägg till ett program. Vilken kanal vill du spela in? U> kanal ett S> kanal ett, är det korrekt? S> Så, Vilken kanal vill du spela in? U> english S> Okay...

50. Gu Dialogue Systems Lab 49 Multiple application domains S> Welcome to the VCR manager! S> Lets see. What can I do for you? U> change domain S> Okay. change domain… Lets see. What domain do you want? U> telephone S> Okay. This is your automatic telephone operator… Lets see. Please specify a function. U> call S> Okay. make a phone call. S> Lets see. Please specify the destination of the call. U> luis S> Okay. Luis… Calling

51. Gu Dialogue Systems Lab 50 8. Conclusions: GoDiS & Issue- Based Dialogue Management general solutions to –dealing with multiple tasks –sharing information between tasks –grounding and feedback –user initiative (accommodation) –menu-based dialogue rapid prototyping of applications –dialogue plans switching language and domain online

52. Gu Dialogue Systems Lab 51 Current and future work Extend to more complex dialogue types –Negotiation (theory exists, not yet implemented) –Tutorial dialogue –… Explore use of QUD and ISSUES to assign proper focus intonation –Stina Ericsson, forthcoming PhD thesis Build VCR control application, available on the phone –Users can phone up and get TV programs recorded on a computer –Collect corpus to use as basis for further developments Integrate with type-theoretical situation semantics

53. Gu Dialogue Systems Lab 52 Future dialogue systems research in Göteborg TALK project –Talk and Look, Tools for ambient linguistic knowledge Multilingual dialogue systems –grammar synchronisation –relating abstract domain representations (e.g. for devices) to natural language expressions Extending to multimodal dialogue systems –language representations vs tables, graphics

54. Gu Dialogue Systems Lab 53 Automatic generation of dialogue systems from domain/device descriptions Improving speech recognition by looking at dialogue information states Multiple recognition grammars/statistical models