1. TrindiKit Staffan Larsson Göteborg University Sweden

2. TrindiKit architecture & concepts what’s in TrindiKit? comparison with other architectures this talk

3. What is TrindiKit? A toolkit for dialogue systems R&D Based on the Information State Update approach Aims to close the gap between theory and practice of dialogue systems –today: complex theories, simplistic systems –adapt theories to make use of them in systems

4. Theoretical advantages of TrindiKit theory-independent –allows implementation and comparison of competing theories –promotes exploration of middle ground between simplistic and very complex theories of dialogue intuitive formalisation and implementation of dialogue theories –the implementation is close to the theory

5. Practical advantages of TrindiKit promotes reuse and reconfigurability on multiple levels general solutions to general phenomena enables rapid prototyping of applications allows dealing with more complex dialogue phenomena not handled by current commercial systems

6. Key ideas thinking in terms of IS updates –update rules functions IS (+moves)  IS (+moves) generic domain-independent dialogue management –requires modularity use of global information state –all modules can access all information

7. module 1 module … Total Information State (TIS) Information state proper (IS) Module Interface Variables Resource Interface Variables resource 1 control module i module j module … module n resource … resource m DME

8. TrindiKit basic dialogue theory domain & language resources basic system application information state approach genre-specific theory additions genre-specific system

9. an abstract data structure (record, DRS, set, stack etc.) accessed by modules using conditions and updates Information State (IS)

10. module or group of modules responsible for –updating the IS based on observed moves –selecting moves to be performed dialogue moves are associated with IS updates using IS update rules update rules: rules for updating the TIS –rule name and class –preconditon list: conditions on TIS –effect list: updates to TIS Dialogue Move Engine (DME)

11. Modules (dialogue move engine, input, interpretation, generation, output etc.) –access the information state –no direct communication between modules only via module interface variables in TIS modules don’t have to know anything about other modules increases modularity, reusability, reconfigurability –may interact with user or external processes Resources (device interface, lexicons, domain knowledge etc.) Modules and resources

12. What does TrindiKit provide? High-level formalism and interpreter for implementing dialogue systems –promotes transparency, reusability, plug- and-play, etc. –allows implementation and comparison of dialogue theories –hides low-level software engineering issues Ready-made modules and resources –speech –interfaces to databases, devices, etc. –reasoning, planning

13. a library of datatype definitions (records, DRSs, sets, stacks etc.) –user extendible a language for writing information state update rules (improved version in 3.0) GUI: methods and tools for visualising the information state (coming in 3.0) debugging facilities (improved in 3.0) –typechecking –logs of communication modules-TIS –etc. TrindiKit contents (1)

14. A language for defining –update algorithms used by TrindiKit modules to coordinate update rule application –basic control structure, to coordinate modules A library of basic ready-made modules for input/output, interpretation, generation etc.; –e.g. Nuance speech recognition and synthesis A library of ready-made resources and resource interfaces, e.g. to hook up databases, domain knowledge, devices etc. TrindiKit contents (2)

15. Special modules and resources included with TrindiKit Speech recognition and synthesis modules –TrindiKit shells for off-the-shelf recognizers and synthesizers –e.g. ViaVoice, Nuance, Festival, MARY OAA interface resource –enables interaction with existing software (e.g. Longbow planner, in Lisp) Possible future modules –robust parser –planning and reasoning modules –multimodal input and output

16. Asynchronous TrindiKit Internal communication uses either –OAA (Open Agent Architecture) from SRI, or –AE (Agent Environment), a stripped-down version of OAA, implemented for TrindiKit enables asynchronous dialogue management –e.g.: system can listen and interpret, plan the dialogue, and talk at the same time

17. TrindiKit and OAA TrindiKit uses a stripped-down version AE (Agent Architecture) OAA can be used with TrindiKit in various ways –use OAA as base for TrindiKit instead of AE; TrindiKit modules and infostate are OAA agents –Whole TrindiKit system is an OAA agent –TrindiKit modules are OAA agents or AE agents

18. explicit information state datastructure –makes systems more transparent –enable e.g. context sensitive interpretation, distributed decision making, asynchronous interaction update rules –provide an intuitive way of formalising theories in a way which can be used by a system –represent domain-independent dialogue management strategies TrindiKit features

19. TrindiKit features cont’d resources –represent domain-specific knowledge –can be switched dynamically e.g. switching language on-line in GoDiS modular architecture promotes reuse –basic system -> genre-specific systems –genre-specific system -> applications

20. TrindiKit and VoiceXML VoiceXML –industry standard –form-based dialogue manager –web/telephony infrastructure –requires scripting dialogues in detail Theory-specific? –VoiceXML implements a specific theory of dialogue –TrindiKit allows implementation of several different theories of dialogue –More complex dialogue phenomena hard to deal with in VoiceXML

21. TrindiKit and VoiceXML, cont’d Combine VoiceXML with TrindiKit? –future research area –support connecting TrindiKit to VoiceXML infrastructure –use TrindiKit system as VoiceXML server, dynamically building VoiceXML scripts –convert VoiceXML specifications to e.g. GoDiS dialogue plans

22. technical features interfaces to OAA (but can also run without it) –allows connecting systems to external software system modules can run either serially or in parallell wrappers for off-the-shelf recognizers and synthesizers runs on UNIX, Windows, Linux currently uses SICStus Prolog –but considering moving to shareware Prolog –possibly reimplement in other language –or make it independent of programming language (compilers for several languages)

23. availability version 2.1 is available version 3.0a coming at end of 2002 –SIRIDUS deliverable D6.4 TrindiKit website –www.ling.gu.se/projects/trindi/trindikit SourceForge project –development versions available –developer community? licensed under GPL more info in –Larsson & Traum: NLE Special Issue on Best Practice in Dialogue Systems Design, 2000 –TrindiKit manual (available from website)

24. GoDiS – information state based on Questions Under Discussion (Larsson et al 2000) –currently being reimplemented for thesis MIDAS – DRS information state, first-order reasoning (Bos & Gabsdil, 2000) EDIS – information state based on PTT (Matheson et al 2000) –extended to handle tutorial dialogue by Moore, Zinn, Core et al SRI Autoroute – information state based on Conversational Game Theory (Lewin 2000); robust interpretation (Milward 2000) Systems developed using TrindiKit

25. Recent work D’Homme (EU 2001) –Dialogues in the Home Environment –GoDiS, SRI system Instruction Based Learning for mobile robots (U Edinburgh) –MIDAS Tutoring Dialogue (U Edinburgh) –BEETLE (based on EDIS) Student projects (Gothenburg) adapting GoDiS to various domains

26. TrindiKit in SIRIDUS added modules for connecting speech improved update rule language GUI (in 3.0) –monitoring dialogue –generate dialogue printouts incl. infostat improved debugging facilities (in 3.0) –tracing (other than Prolog trace) –typechecking extending coverage of individual systems to action-oriented and negotiative dialogue

27. Conclusions: TrindiKit & Information State Approach a toolkit for dialogue systems R&D freely available to researchers close the gap between theory and practive of dialogue systems theory-independent promotes reuse and reconfigurability on multiple levels

28. Flexible issue-based dialogue management in GoDiS Staffan Larsson Göteborg University Sweden

29. Goals explore and implement issue-based dialogue management –adapt an existing semi-computational theory of dialogue (Ginzburg’s KOS) to dialogue system (GoDiS) and implement –extend theory (incl. accommodation, action-oriented dialogue, negotiation, conditional responses) separating general and domain-dependent phenomena helps reconfigurability –general theory of dialogue, extended into subtheories for different dialogue types –minimize effort for adapting to new domains

30. TrindiKit GoDiS GoDiS-I GoDiS-A Travel Agency Auto- route Xerox manual VCR manager IBDM home device manager IS approach genre- specific applicatio n-specific

31. Basic issue-based dialogue management GoDiS-I: enquiry-oriented dialogue –typically, database search dialogue as raising and addressing issues dialogue plans to drive dialogue –each plan associated with a ”task question” deals with multiple simultaneous issues enables information sharing between plans

32. basic infostate PRIVATE : PLAN : OpenStack( Action ) AGENDA : OpenQueue( Action ) SHARED : BEL : Set( Prop ) COM : Set( Prop ) QUD : OpenStack( Question ) LU: SPEAKER: Speaker MOVES: OpenQueue( Move ) ISSUES : OpenStack( Question ) QUD:local, questions available for ellipsis resolution ISSUES: global, questions which have been raised but not yet resolved

33. Semantics Proposition: n-ary predicate-argument structure –e.g. dest-city(paris) Question –Y/N-questions: ?P, P is a proposition –wh-questions: ?x.P(x) –alt-questions: {?P1, …, ?Pn} ShortAns –individual markers: paris, april, … –yes, no Q-A relations (adapted from Ginzburg) –resolves(A,Q): A resolves Q dest-city(paris) resolves ?x.dest-city(x) –relevant(A,Q): A is a relevant answer to Q not(dest-city(paris)) is relevant to ?x.dest-city(x), but does not resolve it

34. sample dialogue plan task-q: ?x.price(x) <findout(?x.how(x)) findout(?x.dest-city(x)) findout(?x.dept-city(x)) findout(?x.dept-month(x)) raise(?x.dept-day(x)) findout(?return) If return then < findout(?x.ret_month) findout(?x.ret_day) > raise(?x.class(x)) consultDB(?x.price(x))>

35. Multiple issues and information sharing user can switch freely between any number of tasks –e.g. set the clock while programming the VCR information collected while doing task A can be used in task B –e.g. where the user wants to travel; can be used for travel reservation, visa enquiries, hotel reservation etc. this is a consequence of keeping a global information state, rather than task-local states (as in e.g. VoiceXML)

36. Dealing with multiple issues if user asks Q, push Q on ISSUES and load plan for dealing with Q if users asks Q’ while system is dealing with Q, throw out plan for Q but Q remains on ISSUES when Q’ resolved, Q topmost on ISSUES will trigger reloading plan for dealing with Q –general rule: if SHARED.COM contains info resolving Q, don’t ask Q –so any resolved questions in plan will be thrown out

37. Information sharing across plans GoDiS does not keep track of which plan was being executed when propositions were added so information sharing is determined by question sharing across plans plan for VISA question: ?need_visa <findout(?x.dest-city(x)) findout(?x.dept-city(x)) findout(?x.citizenship(x)) consultDB(?need_visa) > –shared 2 questions with plan for ?x.price(x)

38. Sample dialogue: multiple tasks & info sharing S> Welcome to the travel agency! U> price information S> (…) Lets see. How do you want to travel? U> by flight S> (…) What city do you want to go to? U> paris S> (…) What city do you want to go from? U> do I need a visa ? S> (…) Lets see. What country are you from? U> sweden S> Okay. Yes, you need a Visa. S> Returning to the issue of price. Lets see. What city do you want to go from?

39. Grounding and feedback grounding –making sure that the participants are percieving, understanding, and accepting each other’s utterances –dealing with problematic situtations where e.g. an utterance is not percieved feedback –(short) utterances which signal grounding status of previous utterance –in addition, utterances which signal switching task (”returning to…”), reraising questions (”so…”) etc.

40. Grounding and feedback feedback types (Allwood, Clark) –action level: contact, perception, understanding, acceptance –polarity: positive, negative update strategies –optimistic non-cautious cautious –pessimistic feedback and grounding for a dialogue system

41. Feedback: action levels and associated metaissues A uttered U to B; A and B are now faced with a number of issues contact: do A and B have contact? perception: –A: does B percieve U (correctly)? –B: what did B say? / Did B say V? understanding: –A: does B understand U (correctly) –B: what did B mean? / Did B mean C? acceptance –A: does B accept U –B: should I accept U?

42. Some ICM dialogue moves Interactive Communication Management –feedback –turntaking –sequencing: reflects dialogue structure icm:Level{*Polarity}{:Content} –icm:und*neg – ”I don’t understand” –icm:und*pos:P – ”To Paris.” –icm:acc*neg:Q – ”Sorry, I can’t answer Q” –icm:acc*pos – ”Okay” –icm:reraise:Q – ”Returning to the issue Q” –icm:loadplan – ”Let’s see…”

43. Question Accommodation to deal with –user giving more/less/other information than requested –simple information revision –reraising issues basic idea: –move questions to QUD or ISSUES to adapt to user utterances –e.g. short answers where question can be found in the context –generate clarification question if necessary more powerful than corresponding mechanisms in VoiceXML

44. Sample dialogue: accommodation S: Welcome to the travel agency. U: From London to Paris in April –not relevant to any question that has been raised, or to any current task –look in domain knowledge for a plan (for dealing with some question Q) with matching questions –load this plan, push Q on ISSUES –find in the plan the question(s) matching the user’s answer –integrate answer (requres matching question on ISSUES) S: Alright, you want to know about price. (…) –proceed to next plan item S: How do you want to travel? –ISSUES=

45. Action-oriented dialogue based on menus GoDiS-A: adapted for the genre of action-oriented dialogue each plan now associated with an action or a question semi-automatic conversion of menus to dialogue plans sample domain: menu-based dialogue for VCR

46. Converting menus to plans MenuPlan construct multi-choice list: findout({?action(a1), ?action(a2),...,?action(a n )}) tick-box: +/-P findout(?P) dialogue window: parameter=___ findout(?x.parameter(x)) pop-up message confirming action a as done confirm(a)

47. VCR menu structure fragment change play status –play, stop etc. change channel –channel: _ timer recording –add program channel:_ date:_ start-time:_ end_time:_ display added program –delete program display existing programs delete program:_ settings –clock, etc.

48. Plan for VCR>timer recording>add program findout(?x.program_position_to_store(x)), findout(?x.date_to_store(x)), findout(?x.start_time_to_store(x)), findout(?x.stop_time_to_store(x)), dev_do(vcr, 'AddProgram') (confirmations generated by update rule whenever some action is done, so not needed in plan)

49. Sample menu dialogue (with language change!) S: Welcome to the VCR manager. Let’s see. What can I do for you? U: Add a program today, channel five –request(add_program) -> plan is loaded –issue accommodation to integrate ”channel five” … S: What time do you want to start recording? U: Five fifteen …

50. Conclusions: GoDiS & Issue- Based Dialogue Management general solutions to –dealing with multiple tasks –sharing information between tasks –grounding and feedback –user initiative (accommodation) –menu-based dialogue –negotiative dialogue (not yet implemented) rapid prototyping of applications –dialogue plans switching language online

51. end

52. Menu navigation step-by-step S: Welcome to the VCR manager! What can I do for you? S: Do you want to change play status, change channel, go to timer recording, go to settings, or make a query? U: timer recording S: OK, timer recording. Do you want to add a program, or delete a program? U: add a program S: OK, add a program. What channel do you want? U: one S: What date? U: today S: What time do you want to start recording? U: eight thirty S: What time do you want to stop recording? U: nine thirty [UPnP] AddProgram(1,today,830,930) S: I will record channel 1 today from 08:30 to 09:30.

53. Several steps at once + responding to unasked questions (question acc.) S: Welcome to the VCR manager! U: go to the vcr and go to timer recording and add a program S: OK, add a program. What channel do you want? U: channel one today from eight thirty (accommodate ”what date?”) (accommodate ”When do you want to start?”) S: What time do you want to stop recording? U: nine thirty [UPnP] AddProgram(1,today,830,930) S: I will record channel 1 today from 08:30 to 09:30.

54. Jumping to a submenu S: Welcome to the VCR manager! U: add a program S: OK, add a program. What channel do you want? U: channel one today from eight thirty to nine thirty [UPnP] AddProgram(1,today,830,930) S: I will record channel 1 today from 08:30 to 09:30.

55. Answer unasked questions, unspecified task (task and question accommodation) S: Welcome to the VCR manager! U: channel one today (accommodate task: add program) (accommodate ”What channel?”) (accommodate ”What date?”) S: OK, add a program. What time do you want to start recording? U: eight thirty to nine thirty (accommodate ”When do you want to stop?”) [UPnP] AddProgram(1,today,830,930) S: I will record channel 1 today from 08:30 to 09:30.

56. System clarification question (task clarification and accommodation) S: Welcome to the VCR manager! U: eight thirty (answer matches several plans > generate clarification) S: What do you mean by that? Do you want to add a program, or set the clock? U: set the clock (load plan for setting clock) (accommodate ”What time?”) [UPnP] SetClock(830) S: OK, clock. The clock was set.

57. Further switching language on-line switching task on-line ”Issues Under Negotiation” –proposals add alternative solutions (answers) to some problem (question) –Larsson 2002 SigDial paper –(not yet part of demo)

58. more about flexible dialogue in SIRIDUS Knowledge-based Reference Resolution for Dialogue Management in a Home Domain Environment (Quesada & Amores) –at 12:00 today Cooperation and Collaboration in Natural Command Language Dialogues (Amores & Quesada) –at 16:30 today Enhancing collaboration with conditional responses in information-seeking dialogues (Kruijff-Korbayova, Karagjosova & Larsson) –at 17:00 today

59. Other architectures Agent communication protocols –Open Agent Architecture –KQML Dialogue system toolkits –DARPA Communicator –SOAR, NL-SOAR Systems (implementing specific dialogue theory) –Verbmobil –VoiceXML have been compared to TrindiKit (see deliverables or ask)

60. relevant properties support for information state approach –which datastructures/datatypes are supported? support for managing control flow –not limited to pipelining asynchronous processing modularity potential for scalability

61. OAA ”Facilitator” (hub) manages communication between ”agents” (spokes) facilitator can maintain global information state –what datastructures are available? asynchronous processing modular similar to scriptless version of DARPA Communicator

62. KQML Knowledge Querying and Manipulation Language communication protocol between agents

63. DARPA Communicator Hub-and-spoke architecture modularity, flexible control, asynchronous processing hub functions –message routing between servers (spokes) –state maintenance –control flow; script decides which server to call next

64. DARPA communicator cont’d limited support for information states –”tokens”: frames with name, index, and list of feature-value pairs –tokens processed by scripts set of rules determining when to call a server + which arguments (features) to pass Hub scripts too limited to do real dialogue management –most actual systems have separate dialogue manager –hub mostly used for data routing

65. SOAR toolkit for modeling cognitive agents similar to TrindiKit in some respects keeps single information state –however, only one datastructure whereas TrindiKit has extendible range of (possibly complex) datatypes update rules –some nice ways of triggering rules which can be considered for TrindiKit supports ”chunking” of rules –however, may not be too useful in practice

66. Verbmobil not dialogue system per se –rather, dialogue translation system several local information states, not one global –”partitioned blackboard” –which datastructures? extendible set? limited control support