1. Predicting and Adapting to Poor Speech Recognition in a Spoken Dialogue System Diane J. Litman AT&T Labs -- Research http://www.research.att.com/~diane Shimei Pan Columbia University http://www.cs.columbia.edu/~pan

2. Overview Problem –spoken dialogue system performance can vary widely for different users, and even for the same user during different dialogues Hypothesis –system performance can be improved by dynamically detecting and responding to problematic dialogue situations Results –task success increases in a system which predicts and adapts to poor speech recognition during the course of a dialogue

3. A Successful Dialogue without Adaptation S: Hi, this is AT&T Amtrak schedule system. This is Toot. How may I help you? U: I want a train from Baltimore to New York City on Sunday at 8:30 in the evening. S: Do you want me to find the trains from Baltimore to New York City on Sunday approximately at 8:30 in the evening now? U: Yes. S: I am going to get the train schedules for you. It will take some time depending on the internet traffic. Please wait patiently....

4. A Successful Dialogue after Adaptation S: Hi, this is AT&T Amtrak schedule system. This is Toot. How may I help you? U: I need to find a train going to New York City. S: Which city do you want to leave from? U: Baltimore. S: Do you want me to find the trains from Baltimore to DC on Thursday at anytime now? U: No. S: How may I help you? U: I need to leave on Sunday. **TOOT adapts to more conservative dialogue behaviors** S: I heard you say go to DC between 1 night and 8. Do you want me to find the trains from Baltimore to DC on Thursday between 1 night and 8 now? U: No.

5. S: You can say yes or no. U: No. S: How may I help you? U: I need to go to New York City from Baltimore on Sunday at 8:30 pm. S: I heard you say go to Chicago on Tuesday around 8:45 am. Do you want me to find the trains from Baltimore to Chicago on Tuesday around 8:45 am now? U: No. **TOOT adapts to even more conservative dialogue behaviors** S: Which city do you want to go to? U: New York City. S: Do you want to go to New York City? U: Yes. S: Which city do you want to leave from? U: Baltimore...

6. Toot: An Adaptive Spoken Dialogue System A phone-based system for accessing train schedules Implemented using internal platform (Kamm et al ‘97) –phone interface –automatic speech recognition (ASR) –text to speech (TTS) –dialogue manager –application manager Different versions depending on dialogue strategy and adaptability parameters –initiative strategies (system, mixed, user) –confirmation strategies (explicit, implicit, none) –adaptability condition (adaptive, non-adaptive)

7. Learning to Detect Problematic Dialogues (Litman et al. ‘99) Use machine learning to automatically derive rules for detecting poor speech recognition at the dialogue level –speech recognition is most predictive of performance –can improve recognition by changing dialogue strategies 2 classifications –if (% of misrecognized utterances > threshold) then BAD –else GOOD 23 (automatically computable) features –acoustic confidence, dialogue efficiency, dialogue quality, lexical, experimental parameters

8. Instantiation for Toot Corpus –120 Toot dialogues from previous experiments (Litman, and Pan ‘99) –45 GOOD dialogues (e.g., first Toot example) –75 BAD dialogues (e.g., second Toot example) Machine learning program –Ripper (Cohen ‘96) Best learned ruleset uses a single acoustic feature –if (predicted_misrecog%_using_confScore_-4 > 3%) then BAD –else GOOD –80% cross-validated accuracy rate

9. Example S1: This is Toot. How may I help you? U1: I need to find a train going to New York City. ASR1: string=DC I don’t care on ThursdayconfScore= -5.3 S2: Which city do you want to leave from? U2: Baltimore. ASR2: string=BaltimoreconfScore= -1.7 Since predicted_misrecog%_using_confScore_-4 = 50%, which is > 3%, dialogue is classified as BAD

10. Predicting and Adapting to Problems Online Algorithm and tuneable parameters: Main … for each user utterance if ((turns since CurStrat assigned) >= AdaptFreq) PredictUsing(Ruleset); … PredictUsing(Ruleset) for each rule R in Ruleset if (CheckPre(R) == “TRUE”) if (RightHandSide(R) == “BAD”) AdaptConservative(CurStrat);

11. Experimental Evaluation Adaptive vs. Non-Adaptive Toot –initial dialogue strategy = UserNo (user initiative, no confirmation) –adaptation frequency = 4 user turns –ruleset = rules learned using Ripper –AdaptConservative = switch to MixedImplicit, then switch to SystemExplicit 6 subjects (naïve users) per Toot 4 tasks per subject 48 total dialogues (recordings, logs, and user surveys) Evaluation measures: dialogue efficiency, dialogue quality, task success, and usability

12. Adaptability Results Adaptive Toot outperforms Non-Adaptive Toot –higher task success –higher user expertise and overall satisfaction –more accurate ASR –shorter dialogues Adaptive Toot only adapts when appropriate –task success is.66/.60 when adaptation is/isn’t triggered

13. Related Work Dialogue level –Litman and Pan ‘99, Litman et al.’99, Walker et al. ‘00 Utterance level –Smith ‘98, Levow ‘98, Litman et al. ‘00, Chu-Caroll and Nickerson ‘00 Multiple dialogue level –Levin & Pieraccini ‘97, Satinder et al. ‘00 (see talk tomorrow at 2!)

14. Summary A fixed dialogue strategy will not be ideal for different users, or even for the same user in different circumstances An adaptive system can improve performance –adaptive Toot adapts to more conservative dialogue strategies, based on its predictions of ASR problems –task success increases from 23% to 65%

15. Sample Task Scenario Try to find a train going to New York City from Baltimore on Sunday at 8:30 pm. If you cannot find an exact match, find the one with the closest departure time. Please write down the exact departure time of the train you found as well as the total travel time.