1. TrindiKit

2. TrindiKit architecture & concepts what’s in TrindiKit? comparison with other architectures this talk

3. What is TrindiKit? a toolkit for – building and experimenting with dialogue move engines and systems, – based on the information state approach originally developed in TRINDI, considerably extended and improved in SIRIDUS

4. Key ideas thinking in terms of IS updates –update rules functions IS (+moves)  IS (+moves) generic domain-independent dialogue management –requires modularity use of global information state –defined in terms of abstract datatypes –all modules can access all information

5. module 1 module … Total Information State (TIS) Information state proper (IS) Module Interface Variables Resource Interface Variables resource 1 control module i module j module … module n resource … resource m DME

6. TrindiKit basic dialogue theory domain & language resources basic system application information state approach genre-specific theory additions genre-specific system

7. an abstract data structure (record, DRS, set, stack etc.) accessed by modules using conditions and updates the Total Information State (TIS) includes –Information State proper (IS) –Module Interface variables –Resource Interface variables Information State (IS)

8. module or group of modules responsible for –updating the IS based on observed moves –selecting moves to be performed dialogue moves are associated with IS updates using IS update rules –there are also update rules no directly associated with any move (e.g. for reasoning and planning) update rules: rules for updating the TIS –rule name and class –preconditon list: conditions on TIS –effect list: updates to TIS update rules are coordinated by update algorithms Dialogue Move Engine (DME)

9. Modules (dialogue move engine, input, interpretation, generation, output etc.) –access the information state –no direct communication between modules only via module interface variables in TIS modules don’t have to know anything about other modules increases modularity, reusability, reconfigurability –may interact with user or external processes Resources (device interface, lexicons, domain knowledge etc.) –hooked up to the information state (TIS) –accessed by modules –defined as object of some type (e.g. ”lexicon”) Modules and resources

10. What does TrindiKit provide? High-level formalism and interpreter for implementing dialogue systems –promotes transparency, reusability, plug- and-play, etc. –allows implementation and comparison of dialogue theories –hides low-level software engineering issues Ready-made modules and resources –speech –interfaces to databases, devices, etc. –reasoning, planning

11. a library of datatype definitions (records, DRSs, sets, stacks etc.) –user extendible a language for writing information state update rules (improved version in 3.0) GUI: methods and tools for visualising the information state (coming in 3.0) debugging facilities –typechecking –logs of communication modules-TIS –etc. TrindiKit contents (1)

12. A language for defining update algorithms used by TrindiKit modules to coordinate update rule application A language for defining basic control structure, to coordinate modules A library of basic ready-made modules for input/output, interpretation, generation etc.; A library of ready-made resources and resource interfaces, e.g. to hook up databases, domain knowledge, devices etc. TrindiKit contents (2)

13. Special modules and resources included with TrindiKit Speech recognition and synthesis modules –TrindiKit shells for off-the-shelf recognizers and synthesizers –currently ViaVoice, Nuance, Festival OAA interface resource –enables interaction with existing software (e.g. Longbow planner, in Lisp) Possible future modules –robust parser –planning and reasoning modules –multimodal input and output

14. Asynchronous TrindiKit Internal communication uses either –OAA (Open Agent Architecture) from SRI, or –AE (Agent Environment), a stripped-down version of OAA, implemented for TrindiKit enables asynchronous dialogue management –e.g.: system can listen and interpret, plan the dialogue, and talk at the same time

15. TrindiKit and OAA TrindiKit uses a stripped-down version AE (Agent Architecture) OAA can be used with TrindiKit in various ways –use OAA as base for TrindiKit instead of AE; TrindiKit modules and infostate are OAA agents –Whole TrindiKit system is an OAA agent –TrindiKit modules are OAA agents or AE agents

16. explicit information state datastructure –makes systems more transparent –enable e.g. context sensitive interpretation, distributed decision making, asynchronous interaction update rules –provide an intuitive way of formalising theories in a way which can be used by a system –represent domain-independent dialogue management strategies resources –represent domain-specific knowledge –can be switched dynamically e.g. switching language on-line in GoDiS modular architecture promotes reuse –basic system -> genre-specific systems –genre-specific system -> applications TrindiKit features

17. technical features interfaces to OAA (but can also run without it) –see e.g. demo by Zinn et al. system modules can run either serially or in parallell wrappers for off-the-shelf recognizers and synthesizers runs on UNIX, Windows, Linux currently uses SICStus Prolog –but considering moving to YAP prolog (freely available Sicstus clone) –possibly reimplement in other language

18. availability version 2.1 is available version 3.0a coming at end of 2002 –SIRIDUS deliverable D6.4 TrindiKit website –www.ling.gu.se/projects/trindi/trindikit SourceForge project –development versions available –developer community? licensed under GPL more info in –Larsson & Traum: NLE Special Issue on Best Practice in Dialogue Systems Design, 2000 –TrindiKit manual (available from website)

19. GoDiS – information state based on Questions Under Discussion (Larsson et al 2000) –currently being reimplemented for thesis MIDAS – DRS information state, first-order reasoning (Bos & Gabsdil, 2000) EDIS – information state based on PTT (Matheson et al 2000) –extended to handle tutorial dialogue by Moore, Zinn, Core et al SRI Autoroute – information state based on Conversational Game Theory (Lewin 2000); robust interpretation (Milward 2000) Systems developed using TrindiKit

20. Recent work D’Homme (EU 2001) –Dialogues in the Home Environment –GoDiS, SRI system Instruction Based Learning for mobile robots (U Edinburgh) –MIDAS Tutoring Dialogue (U Edinburgh) –BEETLE (based on EDIS) Student projects (Gothenburg) adapting GoDiS to various domains

21. TrindiKit in SIRIDUS added modules for connecting speech improved update rule language GUI (in 3.0) –monitoring dialogue –generate dialogue printouts incl. infostat improved debugging facilities (in 3.0) –tracing (other than Prolog trace) –typechecking extending coverage of individual systems to action-oriented and negotiative dialogue

22. Flexible issue-based dialogue management

23. Goals explore and implement issue-based dialogue management –adapt Ginzburg’s KOS to dialogue system (GoDiS) and implement –extend theory (incl. accommodation, action-oriented dialogue, negotiation, conditional responses) separating general and domain-dependent phenomena helps reconfigurability –general theory of dialogue, extended into subtheories for different dialogue types –minimize effort for adapting to new domains

24. TrindiKit GoDiS GoDiS-I GoDiS-A Travel Agency Auto- route Xerox manual VCR manager IBDM home device manager IS approach genre- specific applicatio n-specific

25. Basic issue-based dialogue management enquiry-oriented dialogue (database search) starting point: –Ginzburg’s DGB and –related DGB update protocols moves: ask, answer, (greet, quit) raising and addressing issues –incl. short answers dialogue plans handling multiple simultaneous issues information sharing between plans sample domain: travel agency

26. basic infostate PRIVATE : PLAN : OpenStack( Action ) AGENDA : OpenQueue( Action ) SHARED : BEL : Set( Prop ) COM : Set( Prop ) QUD : OpenStack( Question ) LU: SPEAKER: Speaker MOVES: OpenQueue( Move ) ISSUES : OpenStack( Question ) QUD:local, questions available for ellipsis resolution ISSUES: global, questions which have been raised but not yet resolved

27. Semantics Proposition: n-ary predicate-argument structure –e.g. dest-city(paris) Question –Y/N-questions: ?P, P is a proposition –wh-questions: ?x.P(x) –alt-questions: {?P1, …, ?Pn} ShortAns –individual markers: paris, april, … –yes, no Q-A relations (adapted from Ginzburg) –resolves(A,Q): A resolves Q dest-city(paris) resolves ?x.dest-city(x) –relevant(A,Q): A is a relevant answer to Q not(dest-city(paris)) is relevant to ?x.dest-city(x), but does not resolve it

28. sample dialogue plan task: ?x.price(x) <findout(?x.how(x)) findout(?x.dest-city(x)) findout(?x.dept-city(x)) findout(?x.dept-month(x)) raise(?x.dept-day(x)) findout(?return) If return then < findout(?x.ret_month) findout(?x.ret_day) > raise(?x.class(x)) consultDB(?x.price(x))>

29. Dealing with multiple issues if user asks Q, push Q on ISSUES and load plan for dealing with Q if users asks Q’ while system is dealing with Q, throw out plan for Q but Q remains on ISSUES when Q’ resolved, Q topmost on ISSUES will trigger reloading plan for dealing with Q –general rule: if SHARED.COM contains info resolving Q, don’t ask Q –so any resolved questions in plan will be thrown out

30. Information sharing across plans GoDiS does not keep track of which plan was being executed when propositions were added so information sharing is determined by question sharing across plans plan for VISA question: ?need_visa <findout(?x.dest-city(x)) findout(?x.dept-city(x)) findout(?x.citizenship(x)) consultDB(?need_visa) > –shared 2 questions with plan for ?x.price(x)

31. Sample dialogue: multiple tasks & info sharing S> Welcome to the travel agency! U> price information S> (…) Lets see. How do you want to travel? U> by flight S> (…) What city do you want to go to? U> paris S> (…) What city do you want to go from? U> do I need a visa ? S> (…) Lets see. What country are you from? U> sweden S> Okay. Yes, you need a Visa. S> Returning to the issue of price. Lets see. What city do you want to go from?

32. Grounding and feedback ( Interactive Communication Management) feedback types (Allwood, Clark) –action level: contact, perception, understanding, acceptance –polarity: positive, negative update strategies –optimistic non-cautious cautious –pessimistic feedback and grounding for a dialogue system

33. Feedback: action levels and associated metaissues A uttered U to B; A and B are now faced with a number of issues contact: do A and B have contact? perception: –A: does B percieve U (correctly)? –B: what did B say? / Did B say V? understanding: –A: does B understand U (correctly) –B: what did B mean? / Did B mean C? acceptance –A: does B accept U –B: should I accept U?

34. Some ICM dialogue moves Interactive Communication Management –feedback –turntaking –sequencing: reflects dialogue structure icm:Level{*Polarity}{:Content} –icm:und*neg – ”I don’t understand” –icm:und*pos:P – ”To Paris.” –icm:acc*neg:Q – ”Sorry, I can’t answer Q” –icm:acc*pos – ”Okay” –icm:reraise:Q – ”Returning to the issue Q” –icm:loadplan – ”Let’s see…”

35. Question Accommodation to deal with –user initiative (other than asking questions) –giving more/less/other information than requested –simple information revision –reraising issues basic idea: –move questions to QUD or ISSUES to adapt to user utterances

36. Sample dialogue: accommodation S: Welcome to the travel agency. U: From London to Paris in April –not relevant to anything on ISSUES (since ISSUES is empty) –look in domain knowledge for a plan with matching questions –the plan for dealing with ?x.price(x) is found; this question depends on –dependent issue accommodation: push on ISSUES ?x.price(x) –load plan for ?x.price(x) –issue accommodation: push on ISSUES ?x.dest-city(x) –integrate answer (requres matching question on ISSUES) –same with ?x.dept-city(x), ?x.dep_month(x) S: Alright, you want to know about price. (…) –proceed to next plan item S: How do you want to travel? –ISSUES= (NOTE: sys only deals with price & visa issues)

37. Action-oriented dialogue based on menus GoDiS-A: for action-oriented dialogue each plan now associated with an action or a question –add SHARED.ACTIONS: Stack(Action) –ACTIONS has a similar role to ISSUES New moves: –request(Action) –confirm(Action) [or report(Action, Status)] adapt accommodation strategies to AOD sample domain: menu-based dialogue for VCR (demo during lunch break)

38. Converting menus to plans MenuPlan construct multi-choice list: findout({?action(a1), ?action(a2),...,?action(a n )}) tick-box: +/-P findout(?P) dialogue window: parameter=___ findout(?x.parameter(x)) pop-up message confirming action a as done confirm(a)

39. VCR menu structure fragment change play status –play, stop etc. change channel –channel: _ timer recording –add program channel:_ date:_ start-time:_ end_time:_ display added program –delete program display existing programs delete program:_ settings –clock, etc.

40. Plan for VCR>timer recording>add program findout(?x.program_position_to_store(x)), findout(?x.date_to_store(x)), findout(?x.start_time_to_store(x)), findout(?x.stop_time_to_store(x)), dev_do(vcr, 'AddProgram') (confirmations generated by update rule whenever some action is done, so not needed in plan)

41. Sample menu dialogue (with language change!) S: Welcome to the VCR manager. Let’s see. What can I do for you? U: Add a program today, channel five –request(add_program) -> plan is loaded –issue accommodation to integrate ”channel five” S: What time do you want to start recording? U: Five fifteen …

42. Menu navigation step-by-step S: Welcome to the VCR manager! What can I do for you? S: Do you want to change play status, change channel, go to timer recording, go to settings, or make a query? U: timer recording S: OK, timer recording. Do you want to add a program, or delete a program? U: add a program S: OK, add a program. What channel do you want? U: one S: What date? U: today S: What time do you want to start recording? U: eight thirty S: What time do you want to stop recording? U: nine thirty [UPnP] AddProgram(1,today,830,930) S: I will record channel 1 today from 08:30 to 09:30.

43. Several steps at once + responding to unasked questions (question acc.) S: Welcome to the VCR manager! U: go to the vcr and go to timer recording and add a program S: OK, add a program. What channel do you want? U: channel one today from eight thirty (accommodate ”what date?”) (accommodate ”When do you want to start?”) S: What time do you want to stop recording? U: nine thirty [UPnP] AddProgram(1,today,830,930) S: I will record channel 1 today from 08:30 to 09:30.

44. Jumping to a submenu S: Welcome to the VCR manager! U: add a program S: OK, add a program. What channel do you want? U: channel one today from eight thirty to nine thirty [UPnP] AddProgram(1,today,830,930) S: I will record channel 1 today from 08:30 to 09:30.

45. Answer unasked questions, unspecified task (task and question accommodation) S: Welcome to the VCR manager! U: channel one today (accommodate task: add program) (accommodate ”What channel?”) (accommodate ”What date?”) S: OK, add a program. What time do you want to start recording? U: eight thirty to nine thirty (accommodate ”When do you want to stop?”) [UPnP] AddProgram(1,today,830,930) S: I will record channel 1 today from 08:30 to 09:30.

46. System clarification question (task clarification and accommodation) S: Welcome to the VCR manager! U: eight thirty (answer matches several plans > generate clarification) S: What do you mean by that? Do you want to add a program, or set the clock? U: set the clock (load plan for setting clock) (accommodate ”What time?”) [UPnP] SetClock(830) S: OK, clock. The clock was set.

47. Further switching language on-line switching task on-line ”Issues Under Negotiation” –proposals add alternative solutions (answers) to some problem (question) –Larsson 2002 SigDial paper –(not yet part of demo)

48. more about flexible dialogue in SIRIDUS Knowledge-based Reference Resolution for Dialogue Management in a Home Domain Environment (Quesada & Amores) –at 12:00 today Cooperation and Collaboration in Natural Command Language Dialogues (Amores & Quesada) –at 16:30 today Enhancing collaboration with conditional responses in information-seeking dialogues (Kruijff-Korbayova, Karagjosova & Larsson) –at 17:00 today

49. Other architectures Agent communication protocols –Open Agent Architecture –KQML Dialogue system toolkits –DARPA Communicator –SOAR, NL-SOAR Systems (implementing specific dialogue theory) –Verbmobil –VoiceXML have been compared to TrindiKit (see deliverables or ask)

50. relevant properties support for information state approach –which datastructures/datatypes are supported? support for managing control flow –not limited to pipelining asynchronous processing modularity potential for scalability

51. OAA ”Facilitator” (hub) manages communication between ”agents” (spokes) facilitator can maintain global information state –what datastructures are available? asynchronous processing modular similar to scriptless version of DARPA Communicator

52. KQML Knowledge Querying and Manipulation Language communication protocol between agents

53. DARPA Communicator Hub-and-spoke architecture modularity, flexible control, asynchronous processing hub functions –message routing between servers (spokes) –state maintenance –control flow; script decides which server to call next

54. DARPA communicator cont’d limited support for information states –”tokens”: frames with name, index, and list of feature-value pairs –tokens processed by scripts set of rules determining when to call a server + which arguments (features) to pass Hub scripts too limited to do real dialogue management –most actual systems have separate dialogue manager –hub mostly used for data routing

55. SOAR toolkit for modeling cognitive agents similar to TrindiKit in some respects keeps single information state –however, only one datastructure whereas TrindiKit has extendible range of (possibly complex) datatypes update rules –some nice ways of triggering rules which can be considered for TrindiKit supports ”chunking” of rules –however, may not be too useful in practice

56. VoiceXML Frame-based dialogue manager not really an architecture, more like a domain-independent dialogue system Requires scripting dialogues in detail Combine with GoDiS? –use GoDiS as VoiceXML server, dynamically building VoiceXML scripts –use VoiceXML specifications converted to GoDiS

57. Verbmobil not dialogue system per se –rather, dialogue translation system several local information states, not one global –”partitioned blackboard” –which datastructures? extendible set? limited control support