2. Siridus an information state approach to flexible spoken dialogue systems David Milward Staffan Larsson david.milward@linguamatics.com sl@ling.gu.se

3. Overview Siridus project Information State Approach Main areas of work Some selected topics –Trindikit –Flexible issue based dialogue management –Robust interpretation Conclusions

4. Siridus Project EU Framework 5 Partners –Universities of Gothenburg (technical coordinator) Seville Saarland (administrative coordinator) –Telefonica –Linguamatics –(SRI) Duration –Jan 2000 - Dec 2002

5. The IS Approach considering dialogue in terms of transitions between states –structured “information states" –can be used to model mental states of the dialogue participants advantages –theoretical objects/data structures for dialogue analysis –perspective for comparison of systems and theories

6. IS transitions IS for System IS for System – what are the best structures to represent the state of the dialogue? –stacks? sets of feature structures? –private beliefs + shared commitments? – is information in the IS relevant to: –recognition/interpretation –generation/synthesis – what is explicitly coded, what is emergent? –moves/games emerge as an aggregate of updates Interpret U Generate U’ IS for System

7. IS inspired systems IS seems well suited for building systems between e.g. VoiceXML and full BDI Examples: Delfos, Godis Information structure including –lists of feature structures (Delfos) –stack of QUD (Godis)

8. Main areas of work in Siridus Dialogue types Generating prosody using IS Enhancing speech recognition through IS Demonstrators Toolkit for researchers (Trindikit 2/3) Flexible dialogue Robust interpretation and reconfigurability of dialogue systems

9. Dialogue Types Natural command languages –Quesada, Amores: Cooperation and collaboration in Natural Command Language Dialogues Conditional responses –Kruijff-Korbayova, Karagjosova, Larsson: Enhancing collaboration with conditional responses in information seeking dialogues Clarifications –Cooper & Ginzburg: Using Dependent Record Types in Clarification Ellipsis Negotiative dialogue

10. Limited Enquiry Negotiation Dialogue (LEND) information seeking dialogues often assume a fixed set of parameters users must fill out in human-human corpora, participants often negotiate which parameters to use they are batting proposals to and fro users need not supply any particular subset of the parameters of an action similarly, the system needn’t be indifferent about the best set of parameters to use – –it may greatly prefer to get a departure-time if it can decide on the next question according to maximal utility

11. Main areas of work in Siridus Dialogue types Generating prosody using IS Enhancing speech recognition through IS Demonstrators Toolkit for researchers (Trindikit 2/3) Flexible dialogue Robust interpretation and reconfigurability of dialogue systems

12. Generating Prosody using IS Do we need it? Sample dialogue –S: How can I help you? –U: I'd like a flight from London to Paris –S: When do you want to travel? –S: WHEN do you want to travel? Default intonation in Text-To-Speech systems: –When do you WANT to travel? (Festival, ViaVoice) –When DO you want to travel? (Articulator) –When do you want TO travel? (AT&T)

13. Can we fix the intonation for particular sentences/phrases? No! Intonation is context dependent U: A train from London to Edinburgh, the first S: Sorry, do you want to travel on the first of SEPTEMBER or do you want to travel first CLASS? U: First of September how much is it? S: SECOND class costs FIFTY pounds. FIRST class costs ONE HUNDRED pounds

14. Information State and Prosody Motivate prosody using abstract notion of information structure partitioning –c.f. Steedman, Vallduvi, Prague school etc. Integration of these concepts into an information state update approach –Theme/rheme - questions under discussion (QUD) QUD can be implicit so more general than QA pairs –Focus/background comparison of current utterance vs –shared commitments from dialogue history –alternatives available in the domain aim to avoid over generation of focus

15. Main areas of work in Siridus Dialogue types Generating prosody using IS Enhancing speech recognition through IS Demonstrators Toolkit for researchers (Trindikit 2/3) Flexible dialogue Robust interpretation and reconfigurability of dialogue systems

16. Enhancing speech recognition and synthesis using IS For finite state or form based dialogue can associate particular grammars with particular prompts But what if you generate your prompts automatically? –Set of LMs to cover classes of generated prompts e.g. Prompts expecting a np reply –Mix of grammar based and statistical LMs Statistical LM used for back off or embed grammars in a statistical LM –Single statistical LM for the whole domain + rescoring of n-best/lattice according to syntax/semantics/context

17. Lattice/n-best approach Potential sources of information: –previous move/dialogue history Where do you want to go.... Boston... –syntactic/semantic coherence of fragments switch on the door –coherence of fragments relative to each other leave..... at 6pm –reference resolution turn off all heaters vs. turn off hall heater –state of the world turn on|off the light in the hall Combine/contrast sources of evidence to decide –most likely utterance –when to clarify

18. Main areas of work in Siridus Dialogue types Generating prosody using IS Enhancing speech recognition through IS Demonstrators Toolkit for researchers (Trindikit 2/3) Flexible dialogue Robust interpretation and reconfigurability of dialogue systems

19. TrindiKit GoDiS GoDiS-IGoDiS-A Travel Agency VCR manager IBDM / KOS ISU approach Delfos basic system Delfos framework DELFOS-NCL Home manager ISU approach ATOS Home manager

20. Demonstrators: Automated telephone operator Natural language telephone-based access to company telephone directory/PABX –Dial by name –Email address –Multi-party conference –Call transfers KQML based message passing Delfos dialogue management User trials at Telefonica I+D

21. Demonstrators: Home device control Command and control of home devices (following on from the D’Homme Project) –Is the lounge TV on –Switch the TV to Channel 4 –Turn all the lights off in the bedrooms –I’d like to record a programme –Night mode Reconfigurability issues “plug and play” of devices: what happens when move devices around, add new devices? Advanced reference resolution –Quesada and Amores: Knowledge based reference resolution for Dialogue Management in a Home Domain Environment

22. Home Simulation

23. Por favor, enciende la luz. USDD Real devices

24. Main areas of work in Siridus Dialogue types Generating prosody using IS Enhancing speech recognition through IS Demonstrators Toolkit for researchers (Trindikit 2/3) Flexible dialogue Robust interpretation and reconfigurability of dialogue systems

25. Main areas of work in Siridus Dialogue types Generating prosody using IS Enhancing speech recognition through IS Demonstrators Toolkit for researchers (Trindikit 2/3) Flexible dialogue Robust interpretation and reconfigurability of dialogue systems

26. Interpretation in flexible dialogue systems System initiative useful in constraining what users are likely to say User initiated utterances are more variable and less easy to predict Compiling a precise grammar into the recogniser is less likely to give good results Want to be able to extract partial results and avoid total failure

27. Task map from IS and recognition lattice/n-best list to a (partial) interpretation using IS information after interpretation may be too late S: what kind of route would you prefer? U: the quick route pleased need to use information in IS to help –choose between fragments, –find sensible combinations of fragments but can we deal with fragments at all?

28. Self contained fragments I want to go from London... to Birmingham origin = London destination = Birmingham No need to put fragments together Can extract pieces of semantics independently Combination of meaning is via implicit conjunction i.e origin = London & destination = Birmingham

29. Dependent fragments I want to leave Birmingham.... at 3 pm departure-time = 3pm Can’t decide between arrival/departure time without considering other fragments Can use some keyword spotters - phrase grammars not enough

30. Context vs. Syntax S: When do you want to leave? U:... 4pm... U: I’m not sure when I’d like to leave, but I’d like to arrive by 4pm Need to be able to override use of keywords + context using syntactic information

31. Integrating syntactic constraints Phrase spotters go to a particular depth of analysis even if a full parse is possible When syntactic information is available should use it When not should do as well as keyword/phrase spotting Approach: –distribute the semantic representation –get inside fragments to access particular components e.g. “leave” –structural constraints available where necessary Allows –rules similar to keyword/phrase spotting –more specialised rules involving syntactic constraints

32. Indexed representations (1) Can split up any recursive representation into a set of indexed constraints e.g. John wants to leave Boston wants(john, leave(john, boston) e1: wants e2: john e3: leave e4: john e5: boston e0: e1(e2,e6) e6: e3(e4,e5)

33. Indexed Representations (2) Individual constraints provide content and structure Uniform representation for utterance fragments or fully connected sentence –fragments miss certain constraints, or identity between indices Provides a description of the original semantic representation semantic chart or lattice is an indexed representation where readings are packed –same index -> alternative reading –c.f. edges with ‘index’: 0-1-np

34. Achieving compositionality What happens when the output semantics needs to be more complex?, –I want to go to Glasgow on Tuesday and Edinburgh on Wednesday –Turn on all the lights except in the kitchen Not enough to analyse each fragment independently Need to put semantics together

35. Compositional concept based semantics What can provide clues for putting things together if syntax is inadequate? regular patterns of hesitations and repairs function argument structure general ontological knowledge specific real world knowledge

36. Example Consider –the light in the back bedroom –the back bedroom light –light.... back... bedroom if there is only one light in each bedroom, no need for clarification general ontological knowledge might suggest a device in a room specific knowledge about the house allows choice of “back bedroom” vs “back light”

37. Approach Connect concepts according to a model of the domain Ontological relationships: –devices can be in locations –lights are devices –rooms are locations Specific information: –b1 is a bedroom –b1 orientation is back –l1 is a light –l1 in b1

38. Conclusions Semantic chart provides a convenient knowledge representation for manipulating fragment semantics Ontological/domain knowledge used to combine fragments –allows recursive combination –not just filling in a fixed number of slots in a template Allows mapping from IS + word lattice to interpretation without requiring a full parse Reconciling traditional syntax and compositional semantics with robust approaches

39. IS Approach: Conclusions a framework in which we can do interesting theoretical work –improving prosody/recognition using IS –exploration of different dialogue genres –better comparison of theories a research tool for dialogue developers (TrindiKit) systems which are –modular –reconfigurable –between FS/form filling and BDI/planning

40. Siridus www.ling.gu.se/projekt/siridus