© Löckelt, Becker, Pfleger, Alexandersson; DFKI Edilog 2002 Workshop Jan Alexandersson (Tilman Becker, Markus Löckelt, Norbert Pfleger) German Research Center for Artificial Intelligence – DFKI GmbH Stuhlsatzenhausweg 3, Geb Saarbrücken phone: (+44) Overlay – a non-monotonic operation for discourse processing

© Alexandersson; DFKI Nancy What is this Talk about Dialogue systems must deal with utterances relating elliptically to previous dialogue User: „half past five“ System: „at what time would you like to start recording?“ User: „I would like to start recording at half past five“... or utterances only partially related to the previous discourse System: [Showing a list of films] „Here is a list of films“ User: „what films are on TV tonight?“ User: ``Thats a boring program. I‘d rather go to the movies´´ User: ``Thats a boring program. I‘d rather go to the movies tonight´´

© Alexandersson; DFKI Nancy What is this Talk about? ``...an expensive large portrait of the queen by Wainright hanging in the library´´  ``and one of the princess over the mantel´´ There are similar elliptical phenomena like one-anaphora  ``and an expensive large portrait of the princess hanging over the mantel´´

© Alexandersson; DFKI Nancy Overview SmartKom Domain Model Overlay Discourse Modelling Action Planning Processing Partial Utterances Conclusion

© Alexandersson; DFKI Nancy What is the challenge? General tasks for a dialogue manager –Enrichment: Given incomplete information, enrich it from other sources in order to proceed. –Validation: Validate hypotheses against the current discourse state

© Alexandersson; DFKI Nancy Our Approach One general operation – Overlay –Domain representation: (Typed) Feature Structures –Non-monotonic: Always succeeds –Computes a score: Reflecting the structural consistency of the two arguments of overlay

© Alexandersson; DFKI Nancy SmartKom: Task-Oriented Dialogue System MM Dialogue Back- Bone Home: Consumer Electronics EPG Public: Cinema, Phone, Fax, Mail, Biometrics Mobile: Car and Pedestrian Navigation Application Layer SmartKom-Mobile SmartKom-Public SmartKom-Home/Office

© Alexandersson; DFKI Nancy Module Overview on the SmartKom Control GUI

© Alexandersson; DFKI Nancy The True Story

© Alexandersson; DFKI Nancy The DFKI Dialogue Back-Bone Communication pools Main data flow Context information Analysers External Services Modality Fusion Discourse Modelling DiM Action Planning AP Presentation Manager Generators Speech Gesture Speech Graphics Gesture

© Alexandersson; DFKI Nancy theater: MovieTheater movie: Movie reservationNumber: PositiveInteger Domain Model Used for communication in the back-bone Frame-based ontology; representation as Typed Feature Structures in M3L (XML) name: String director: Person cast: PersonList yearOfProduction: PositiveInteger… address: Address seats: SeatStructure… CinemaReservation Application objects composed of subobjects Slots: Feature paths meaningful for the dialogue (entities that can be talked about / referenced to); e.g. movie:director:lastName in a CinemaReservation object Slots can recursively contain other slots firstName: String lastName: String…

© Alexandersson; DFKI Nancy Discourse Modeling Lattice of intention hypothesis sequences from Modality Fusion Enrichment and Validation –compare and enrich with a selected number of discourse states: fill in consistent information compute a score –for each hypothesis - background pair: Overlay (covering, background) Overlay gives a score representing how well the covering fits the background. Scores of overlay, speech interpretation etc. give overall score One augmented hypothesis sequence with best score is passed on to Action Planning Covering: Background: Intention lattice Selected augmented hypothesis sequence

© Alexandersson; DFKI Nancy Example for Overlay with TFS User: What films are on TV tonight? System: [presents list of films] User: That‘s a boring program, I‘d rather go to the movies. How do we inherit “tonight” ?

© Alexandersson; DFKI Nancy The Domain Model A named entertainment at some time A named TV program at some time on some channel A named Movie at some time at some cinema

© Alexandersson; DFKI Nancy Unification Simulation Films on TV tonightGo to the movies Fail – type clash

© Alexandersson; DFKI Nancy Overlay Simulation Go to the moviesFilms on TV tonight Assimilation Background Covering

© Alexandersson; DFKI Nancy ``Formal´´ Definition Overlay Let –co be covering –bg be background Step 1: –Assimilate(co,bg) T bg co Step 2: –Overlay(co,assimilate(co,bg)) If co and bg are frames: recursion If co is empty: use bg If bg is empty: use co If conflict: use co

© Alexandersson; DFKI Nancy Domain Models with Multiple Inheritance Assimilate(co,bg) –Compute the set of minimal upper bounds (MUB) –Specialize the MUBs –Unify the specialized MUBs T cobg Overlay remains untouched MUB

© Alexandersson; DFKI Nancy Processing Partial User Contributions Our Dialogues contain a lot of partial contributions –``Three´´ –``Three o‘clock´´ –``At three o‘clock´´ –``At three [  ] o‘clock´´ –``Later´´ –``Information [  ]´´ –... Requirement –Advanced discourse modelling and action planning –Bidirection communication DiM - AP

© Alexandersson; DFKI Nancy Discourse Modelling - Three-Tiered Context Representation DO 1 DO 2 LO 1 DO 10 DO 3 DO 9 Modality layer Discourse layer System: Here [pointing gesture] I show you a list of films running in Heidelberg. show heidelberg list LO 2 LO 3... Domain layer DomainObject 1 ticketfirst DO 11 DO 12 reserve LO 4 LO 5 LO 6 DomainObject 2 GO 1...  User: Reserve a ticket for the first!

© Alexandersson; DFKI Nancy Action Planner – Regression Planning AP processes plan operators –Uniform description of mixed initiative with user or applications based on Games and Moves. Example: User: request – System: response System: request – Application: response Processing steps 1.Constructs a plan from plan operators –A Plan specifies a partial order of games 2.Interprets the plan –Execute the games Expectations are published during execution of the plan –Guide the interpretation of input

© Alexandersson; DFKI Nancy Action Planning - Regression planning provides(Y) needs(D) condition(D=x) Goal provides() needs(X,Y) provides(X) needs(A,B) provides(C) needs() provides(B) needs(D) provides(A) needs(C) provides(F,G) needs(H) provides(D) needs(E) For Example: Slot D=y Current State provides(?) needs(?) successor states

© Alexandersson; DFKI Nancy AP - Example Plan: VCR Goal: VCR_record VCR_checkMediumVCR_getChannelVCR_getStartTimeVCR_getEndTime VCR_rewindMedium

© Alexandersson; DFKI Nancy Action Planning Initiation of sub-dialogues to gather necessary information Order proposed by the system, but the user may not follow the trail Several cases of user responses to requests: –Providing a matching answer, –Supplying information not (yet) asked for, –Changing already established information –Acting uncooperatively or changing goals

© Alexandersson; DFKI Nancy Expectations When AP initiates a sub-dialogue, it expects a certain answer An Expectation structure is published to help the analysis modules –Expected Slots: corresponding to the request –Possible Slots: plausible in context –Filled Slots: already provided –The currently active Application Allows for a fine-grained distinction between types of possible utterances Discourse Modelling can employ different rankings for each category

© Alexandersson; DFKI Nancy Integration of Partial Partial Utterances are analyzed as subobjects –validation and enrichment include the integration into the preceding context (to achieve a coherent discourse) –we consider three types of partial utterances: 1.The system has the initiative and the user responds elliptically to a system request 2.An elliptical user utterance does not correspond to a request but can be interpreted in context 3.Misinterpretation or non-cooperative user behaviour

© Alexandersson; DFKI Nancy Integration of Partial - Possible Situations User Initiative System Initiative Expected slotN A expected, unify, plan continues Possible slot plausible, unify, replanning plausible, unify, replanning Filled slot plausible, overlay, replanning plausible, overlay, replanning Otherimplausible – recover strategies User contribution DiM processing AP processing

© Alexandersson; DFKI Nancy DiM - Processing User Responses User provides expected slots –Bridging: a new application object is created (of same type as the current application object) for each subobject in the hypotheses its type is compared to the expected slots –matching subobjects are integrated into the new application object –remaining subobjects are interpreted as plausible partial utterances (next slide) –Overlay: The extended application object is overlayed over the application object currently in focus

© Alexandersson; DFKI Nancy Processing Plausible Partial Utterances User provides possible slots –processed like expected slots (but get a score penalty) User changes filled slots –search for a discourse object DO of the same type in the discourse context –create application object AO1 of the same type as the application object AO2 within which DO was mentioned –overlay AO1 over AO2 –gets a score penalty based on position in discourse history If integration fails, all subobjects are passed on unchanged (integration fails)

© Alexandersson; DFKI Nancy Example: Expected and Possible Slots (1) User: I want to record a film. (2) System: When should I start recording? (3) User: 1:30pm on channel two. Analysis of (1) contains the setting of a goal VCR_record (empty application object of type VCR introduced) AP triggers presentation (2) and publishes an expectation containing: –VCR as the type of the current application object –VCR_startTime as the (only) expected slot –some possible slots (e.g., VCR_endTime and VCR_channel ) –and maybe some filled slots (e.g. VCR_mediumPresent ) Analysis of (3) contains two subobjects –the expected slot is filled by “1:30pm” –but also one possible slot is filled by “channel two”

© Alexandersson; DFKI Nancy Example: Filled Slots (1) User: What is currently running on TV? (2) System: The following [Smartakus points at a list of programmes] programmes are currently running. (3) User: And tonight?

© Alexandersson; DFKI Nancy Example: Filled Slots show programcurrently Modality layer Discourse layer Domain layer DO 1 DO 2 LO 1 LO 2 DO 3 LO 3 tonight DO 4 LO 4 DO 3

© Alexandersson; DFKI Nancy Conclusion Uniform, flexible and robust mechanism for interpreting possible partial utterances Pragmatical classification of partial utterances from the point of view of the action planner Modality independent Approach integrates seamlessly with other aspects of discourse processing, e.g. anaphora resolution Implemented and running in, e.g., a large multimodal dialogue system

© Alexandersson; DFKI Nancy Papers Jan Alexandersson and Tilman Becker. Overlay as the Basic Operation for Discourse Processing in a Multimodal Dialogue System. In: Proceedings of the IJCAI Workshop ``Knowledge and Reasoning in Practical Dialogue Systems,´´ Seattle, Norbert Pfleger, Jan Alexandersson, and Tilman Becker. Scoring Functions for Overlay and their Application in Discourse Processing. In Proceedings of ``KONVENS 2002,´´ Saarbrücken, Germany, Markus Loeckelt, Tilman Becker, Norbert Pfleger and Jan Alexandersson Making Sense of Partial. In: Bos, Foster & Matheson (eds): ``Proceedings of the sixth workshop on the semantics and pragmatics of dialogue (EDILOG 2002),´´ Edinburgh, UK, Pages Jan Alexandersson and Tilman Becker. The Formal Foundations Underlying Overlay. Submitted to IWCS5

© Alexandersson; DFKI Nancy Thank you very much for your attention! Merci infiniment de votre attention! Babelfish.altavista.com