TARGETED HELP FOR SPOKEN DIALOGUE SYSTEM SREEDHAR ELLISETTY

MAIN ISSUES » What is TARGETED HELP ? » How it provides Feedback ? » How Target help Module Works? » How SLM Recognizer makes system Robust? » Experimental Evidence » How some ITSs providing Feedback? TARGETED HELP FOR SPOKEN DIALOGUE SYSTEM : INTELLIGENT FEEDBACK IMPROVES NAIVE USERS’ PERFORMANCE

INTRODUCTION - 1 » Targeted Help: Gives immediate Intelligent Feedback to Naïve users’ about their Out-of-Coverage spoken Utterances. » Goal: Producing Intelligent Feedback to the Naive users’ and guiding users Towards In-Coverage Utterances. » Key words - Expert Users - Naive Users - In-Coverage Utterance - Out-of-Coverage Utterance - Grammar based Recognizer - SLM Recognizer

INTRODUCTION - 2 » Expert users know the limitations and capabilities of the system » Grammar-based recognizers tuned to a domain can handle well, the in-coverage Utterances. » Naive users don’t know the Grammar of the System and they produces Out-of-Coverage Utterances. » Grammar based Recognizers rejects and produces messages like “Sorry, I didn't Understand ” » Statistical Language Model (SLM) Recognizer handles Out-of-Coverage Utterance.

INTRODUCTION - 3 » SLM Recognizer produces a recognition hypothesis, which is used by the Targeted Help agent to give the users feedback » Feedback message consists of Diagnostics of the user Utterance and In-Coverage examples. » In-Coverage example uses In-coverage words in the users’ Utterances and the Same Dialogue move » Encourages users’ to align the users Utterance with the Language model of the System » Makes users expert in quick time, improves performance, Reduces time.

Grammar – Based Recognizer » Tuned to a domain performs well for all In-Coverage utterances. » Example: Grammar of FAIRY TALE READER » Grammar can be written quickly Drawbacks: - Puts constraints to the user on the usage of the system - Cant’ handle Out-of-Coverage utterance - Takes lot of time for simple sentences. » Defining a Grammar is to predict future user utterance as good as possible by observing the real users in the desired application » Problem in Defining appropriate Grammar - It should cover as much as possible - It shouldn’t contain unnecessary words, which leads to recognition errors > It should satisfy these issues: Precise, Coverage.

SLM RECOGNIZER » Also called Secondary, Fall-back, Category-based statistical Language model Recognizer. » Handles all the Out-of-Coverage Utterances well » Produces a recognition hypothesis, which is used by the targeted help agent to Give Feedback » Problems with SLM : - Needs to be trained to perform efficiently - Collecting a large enough corpus to create an SLM is time consuming and expensive - A separate parser must be implemented to extract semantic content. Advantages - Don’t put constraints to the users’

System Description » Targeted Help was developed and tested as part of the WITAS dialogue system » WITAS: A command and control and mixed-initiative dialogue system for interacting with a simulated robotic helicopter or UAV » Interaction in WITAS involves joint-activities between an autonomous system and human operator. » Interactions with such a system are not scriptable in advance, and rely on mixed task and dialogue initiatives in conversation. » So, the system is a good test-bed for Targeted Help

THE TARGETED HELP MODULE » Is a separate component that can be added to an existing dialogue system with minimal changes. » Module design makes it quite portable » Goal is to handle Utterance that cant’ be processed by the Main Dialogue system » Aligns the users’ input with the coverage of the system as much as possible PARTS IN THE MODULE: - The secondary Recognizer - The Targeted Help Activator - The Targeted Help Agent

MAIN DIALOGUE SYSTEM TARGETED HELP MODULE ParserDialogue Manager Speech Synthesizer Primary speech recognizer Secondary speech recognizer Targeted help activator Targeted help agent Speech in Speech out Normal response path Targeted help response path Targeted help response path (if secondary SR result parses) ARCHITECTURE OF DIALOGUE SYSTEM WITH TARGETED HELP MODULE

THE TARGETED HELP ACTIVATOR » The Activator’s behavior is as follows for the four possible combinations of Recognizer outcomes - 1.Both Recognizers get a recognition hypothesis 2.Main recognizer gets a recognition hypothesis and secondary recognizer rejects 3. Main recognizer rejects, secondary recognizer gets a recognition hypothesis and it can be Parsed (rare) 4. Main recognizer rejects, secondary recognizer gets a recognition hypothesis and it cant’ be Parsed 5. Both recognizers reject.

MAIN DIALOGUE SYSTEM TARGETED HELP MODULE ParserDialogue Manager Speech Synthesizer Primary speech recognizer Secondary speech recognizer Targeted help activator Targeted help agent Speech in Speech out Normal response path Targeted help response path Targeted help response path (if secondary SR result parses) BOTH RECOGNISERS GET A RECOGNITION HYPOTHESIS

MAIN DIALOGUE SYSTEM TARGETED HELP MODULE ParserDialogue Manager Speech Synthesizer Primary speech recognizer Secondary speech recognizer Targeted help activator Targeted help agent Speech in Speech out Normal response path Targeted help response path Targeted help response path (if secondary SR result parses) MAIN RECOGNISER ACCEPTS, SECONDARY RECOGNISER REJECTS

MAIN DIALOGUE SYSTEM TARGETED HELP MODULE ParserDialogue Manager Speech Synthesizer Primary speech recognizer Secondary speech recognizer Targeted help activator Targeted help agent Speech in Speech out Normal response path Targeted help response path Targeted help response path (if secondary SR result parses) MAIN RECOGNISER REJECTS, SECONDARY RECOGNISER HYPOTHESIS CAN BE PARSED

MAIN DIALOGUE SYSTEM TARGETED HELP MODULE ParserDialogue Manager Speech Synthesizer Primary speech recognizer Secondary speech recognizer Targeted help activator Targeted help agent Speech in Speech out Normal response path Targeted help response path Targeted help response path (if secondary SR result parses) MAIN RECOGNISER REJECTS, SECONDARY RECOGNISER HYPOTHESIS CANNOT BE PARSED

FEEDBACK MESSAGE » Target Help Agent constructs a message based on SLM recognizer hypothesis What the System Heard: - A Report of the Backup SLM recognition hypothesis What the Problem was - Problem with the users’ Utterance What you might say instead - A Similar In-Coverage example. » Feedback Message consists: » Spots the In-Coverage words in the SLM recognizer hypothesis and Dialogue- move type.

PROBLEM DIAGNOSTIC » Diagnostics are of three major types which accounts for major failed utterances 1. End pointing Errors: Eg. User says: Search for the red car System heard: For the red car Category: Push-to-talk button too late 2. Unknown Vocabulary : Eg: Fly over to the Hospital. “The system doesn't understand the word over” vocabulary which is not the Main recognizer”. 3. Sub categorization Mistakes: Saying “Zoom in on the Red car” but system knows only intransitive “Zoom in”. Zooming in on a particular object is not part of the functionality of the system

In-Coverage Example Helps Naive users learn the coverage of the system. Takes in-coverage words produced by the user. Helps the user to imitate in-coverage pattern. Leads to linguistic alignment between the user and the system. It always matches the utterance dialogue-move type. Eg: Wh-Question, Yes/No-Question, Command,etc. Users are likely to produce atleast few in-coverage words. Eg : User : Fly over to the Hospital. In-Coverage Example : Fly to the Hospital.

Constructing In-Coverage Example In what sense they should be similar? It takes into account information from a wide variety of sources, including discourse history, user model and pedagogical strategy. Constraints on not only what words should be included in the example, but also what syntactic structures, semantic representations, and word order. Definition of generation task is: Given a grammar G, and a set of target words T, find a word string W=W1,…,Wn, W ε L(G) such that T {W1,…….,Wn}. Algorithms for generating In-Coverage Example: 1. Iterative-Deepening (ID) Algorithm. 2. A* Algorithm 3. A combined A*-ID Algorithm. 4. Anytime Algorithm.

Example session with targeted help version of the Home Control System User: Hello? System : Hello, How can I help ? User: Um, Can I have some lights on ? System: Sorry, I didn’t quite catch that. To turn a device on or off, you could try something like “turn on the kitchen light”. User: Turn on the kitchen light. System : The kitchen light is now on. User: What other sort of things are there ? System : I didn’t quite catch that. To find out what devices there are you could try something like “ what devices are there”, or “What devices are there in the lounge”. User: What devices are there in the lounge ? System: There are three things in the lounge; the computer, the VCR and the television.

DESIGN OF EXPERIMENTS To assess the effectiveness of the targeted help of this system. Compared the performance of 2 groups of 20 users, One that received help and one that didn’t. Provided minimal written instruction on how to use the system. Task is to direct a helicopter within a city to complete various tasks. Given task is ended when one of the following is met. - The task was accurately completed. - The user thinks he completed the task, though it wasn’t. - The user gave up. The system produced feedback to group in help group. They concluded from the experiment that, users achieved -> HIHER TASK COMPLETION. -> REDUCED TIME.

EXPERIMENTAL RESULTS Found clear evidence that targeted help improves performance in this environment, by showing statistical analysis of the experiment. Users who received help were less likely to give up than those who didn’t received help. - During first task 11% Vs 45% Users who received help took less time to complete tasks than those who didn’t, 290.4s Vs 440.6s. Users who didn’t receive help took 67.0% longer to complete than those who received. Finally, 89% users in the help group and 40% of the users in the No Help group accurately completed the task. Conclusion: It is possible to construct effective Targeted help messages even from fairly low quality secondary recognizers. Such an approach can improve the speed of the training for Naive users and may result in lasting improvements in the quality of their understanding.

My Suggestions It is always better to provide instructions to users on how to use the system and some in-coverage examples. If possible system should give mock session to the user and evaluate the users level of expertise. This leads to better interaction between the user and the system. Giving the system a way to adjust its help strategy to the level of expertise of the user is to be accomplished. We should use both recognizers simultaneously and select the one which gives higher confidence value. Adding multimodal Input/Output and Animated Agents along with this approach in ITSs is very useful.

feedback of geometry tutor The cognitive tutor component determines what feedback to give to the student, based on the classification of the explanation. NLU component uses a knowledge-based approach to recognize sentences as correct or partially correct explanations. It uses a Statistical Text Classifier (STC) when the knowledge-based method fails which acts as a Backup. This is based on the Naive Bayes classification method. System falls back on the STC to determine whether the student’s explanation is focusing on the right geometry rule. The tutor say “You appear to be focusing on the right geometry rule. However, the tutor doesn’t understand your explanation, Could you please state it in a different way”. Thus, the STC enables the tutor to provide more informative feedback in response to unexpected input.

feedback of geometry tutor: Example Explanation of the Triangle sum theorem Student : they are 180. EQUAL -180 Tutor : Could you be more specific? Who are “they”? And what tells you that they are 180. Student : the angles are 180. ANGLES-180 Tutor : You are heading in the right direction, but you need to state this rule more precisely. First where does the 180 come from? Are all angles 180? Student : the angles in a triangle are 180. ANGLES-OF TRIANGLE-180 Tutor : You are awfully close. But is each angle measure equal to 180? Precisely what is 180? Student : the sum of angles in a triangle are 180. TRIANGLE-SUM Tutor : Correct.

SUMMARY Seen experimental evidence that providing Naive users an immediate help message improves their performance and achieves, - HIHER TASK COMPLETION. - REDUCED TIME Adding SLM Recognizer to the Dialogue system makes system robust. Targeted help makes Naïve users expertise in the usage of the system in a quick time. In-Coverage example leads to greater linguistic alignment between the user and the system. This approach is easy to add to an existing system with little modifications. Seen how this approach is useful in some ITSs.

references 1.Generating Canonical examples using candidate words. Dowding, G.Aist, B.A. Hockey, and E.O. Bratt G.Gorell, I.Lewin, and M.Rayner Adding intelligent help to mixed-initiative spoken dialogue systems. 3.B.A. Hockey, G.Aist, J.Dowding, and J.Hieronymus. 2002a. Targeted help and dialogue about plans. 4.B.A. Hockey, G.Aist, J.Dowding, J.Hieronymus, and O. Lemon. 2002b. Targeted Help: Embedded training and methods for evaluation. 5.B.A. Hockey, G.Aist, J.Dowding, J.Hieronymus, O. Lemon, E. Campana, L. Hiatt, A. Gruenstein. Targeted Help for spoken Dialogue Systems: Intelligent feedback improves Naïve users performance. 6.V. Aleven, O. Popescu, and K.R. Koedinger. Towards Tutorial Dialog to support Self Explanation : Adding Natural Language Understanding to a Cognitive Tutor.

THANK YOU ! ANY QUESTIONS?