1. Dialogue Managers in two projects: Comic and Amitie Roberta Catizone University of Sheffield

2. COMIC system features Multimodal (speech and pen input) Mixed initiative, but aimed to keep control Bathroom application in two phases Inputting room measurements and features Browsing and Choosing tilesets for your bathroom 3-D view of you bathroom

3. COMIC system Phase 1 Very goal driven necessary for the user to input: fours walls + dimensions door placement window placement Phase 2 Browsing so only weakly goal driven User should select preferred tileset

4. COMIC architecture Input from Fusion(XML) Output to Fission (XML) DAM decides what the system will do next

5. COMIC Dialogue Manager General purpose control structure Domain specific structures - DAFS Modality independent External resources Dialogue History Ontology

6. Dialogue manager The core mechanism for the DAM is a simple pop- push stack, onto which structures are loaded and run. The structures are Augmented Transition Networks (ATNs) which have Turing power. A control structure analyses the input and chooses an appropriate ATN to load – or continues to run the current ATN. This control structure has to deal with problems like topic change.

7. COMIC DM Input from the Natural Language Understanding Module (DA, Event Type and Object type specified) Output to Fission module with DA + Event type + Object Type) *There may be multiple output strings.

8. COMIC DM DM domain knowledge built by hand using a GUI DAF editor DAFs built manually using common sense knowledge based on how people would use the system (no data collection for pretuning) Not built to discuss topics outside of the COMIC system.

9. Dialogue Action Forms (DAFs) Augmented transition networks (nodes + arcs) Arcs are made up of a test and an action (java programs) System executes the DAFs as specified waiting for input where appropriate.

10. DAF example

11. DAF example for showing and describing a tileset

12. DAF Gui editor

13. COMIC DAFs DAFs prestacked, but there is a mechanism for overriding if necessary. DAF identification and confirmation based on indexing terms (crude triple) Dialogue act (request, inform etc) Event type (show, input-dimension) Object type (wall, door, tileset, color etc) Read/write access to a set of context registers

14. COMIC Control structure Runs DAFs on the stack in order Has the capability to work around a stack entry that needs rescheduling. Phase 4 DAF Phase3 DAF Phase2 DAF window DAF Phase1 DAF bye DAF clarify ATN window DAF bye DAF door DAF

15. COMIC system example System Output: Greeting and phase1 introduction. Please input the first wall. Inform greeting and phase 1 introduction (sent to Fission) Request to draw a wall (sent to Fission) Expectation of a wall (sent to ASR, pen and Fusion) User input: user draws a line on the touchscreen DAM input: DialogueAct Response EventType Add_Bathroom_Part ObjectType Wall startY 100 startX 100 endY 100 endX 400 inkid 0 streamid1 1

16. COMIC system example System Output: Please input the length of this wall Beautification of the wall (sent to visoft application) Request length of wall (sent to Fission) Expectation of a wall length (sent to ASR, pen and Fusion) User input: something unintelligible DAM input: reason ASR_TOO_UNCERTAIN DialogueAct AnalysisError

17. COMIC system example System output: Please repeat Request repeat (sent to Fission) Expectation of a wall length (sent to ASR, pen and Fusion) User input: inputs the length of the wall DAM input: DialogueAct Response ObjectType Wall EventType Modify_Bathroom_Part length Size sizeMeasure m sizeValue 3...

18. Amitie system Telephone banking call center Galaxy Communicator architecture

19. Amitie system features Frame-filling approach Uses data-driven strategy Tasks Verifying the customer’s identity Identifying the customer’s desired transaction Executing the transaction Balance enquiry Report of lost or stolen card Debit card payments Change of customer’s address

20. Amitie Dialogue Manager Frame filling process - data driven. Order independent As frames are filled, need for dialogue decreases Maintains task history and attribute history

21. Amitie DM architecture Task Info Dialogue Act Classifier Frame Agent Task ID Frame Agent Verify-Caller Frame Agent Input from NLU Via HUB (token string, language id, Named entities Task Execution Frame Agents Response Decision Customer Database Dialogue history External files Domain-specific Via HUB DB server

22. Amitie Dialogue Manager Task History Records the topology of tasks Tasks that have been executed successfully or unsuccessfully Task currently under control

23. Amitie Dialogue Manager Attribute History Records the list of attributes needed by the current task Records attributes requested by the system Records attributes provided by the user

24. Amitie system If the system fails to recognize or gather all the necessary data from the user, re- prompts are used, but not more than once. The user is passed to the customer service dept in the case of failure.

25. Amitie system task identification To choose the task the customer wants to perform given An utterance A list of possible tasks Collected data from over 500 call center conversations. Annotations dialogic Stylistic Semantic Discovered 14 distinct tasks, 4 chosen to be implemented.

26. Amitie classification in Task ID frame Agent Adapted a vector-based similarity method Uses term-vector approach common in information retrieval Domain-independent & automatically trained Terms are stemmed content words found in task-specific utterances Documents are vectors of weighted term frequencies derived from the corpus.

27. Amitie training process purpose Creates document vectors to be used in task identification Queries are compared with documents to determine the most likely task. Classification accuracy - 84.7% (based on confidence scores)

28. Amitie Dialogue Act Classifier Purpose is to identify a caller’s utterance as one or more domain-independent dialogue acts: Accept, Reject, Non-understanding, Opening, Closing, Backchannel, Expression)

29. Amitie Dialogue Act Classifier Trained the classifier on corpus of transcribed, calls and used vector-based classification techniques. Differs from the task identifier training in 2 ways 1) an utterance may have multiple correct classifications 2) a different stoplist is necessary Performs well if utterance is short and falls into one of the selected categories (86%)

30. COMIC and Amitie DM comparison Both use a form-filling approach for gathering necessary domain specific data Both use general DAs that are domain independent Amitie classifies the user utterance (dialogue act,task id) based on similarity to dialogue segments in training set. COMIC identifies the semantic content of a user utterance (dialogue act, event type and object type) using the system expectation plus the output of the NLU module which assigns semantic classes based on sets of 1) pre-defined basic dialogue acts (questions vs statements), 2) verbs (show, describe) and 3) nouns (wall, door, tileset)