1. Tuning Your Application: The Job’s Not Done at Deployment Monday, February 3, 2003 Developing Applications: It’s Not Just the Dialog

2. Presentation Overview 1.Potential “Non-Recognizer” Errors 2.System Evaluation Metrics 3.Tuning/Testing Tools Speech applications face perception problems that revolve around numerous “non-recognizer” errors.

3. Types of Potential Errors  Grammar Errors  Prompt Errors  User Errors  Pronunciation Errors

4. Types of Potential Errors: Grammar Errors Program Error – Application fails to perform as designed (perhaps a bug or incorrect linkage) Prediction Error – Grammar set does not accurately reflect all the words a caller might use

5. Types of Potential Errors: Prompt Errors Vague Prompts – Prompt doesn’t provide enough information to help the caller complete their goal Redundant Prompts – Prompt continually repeats information that can easily be inferred Lengthy Prompts – Prompt is too long so that a customer loses track of their options Misleading Prompts – Prompt presents choices where the language may lead a call to an incorrect/inappropriate response

6. Types of Potential Errors: Prompt/Grammar Coordination Problems - Prompts and grammar sets do not match Pronunciation Variation – Unanticipated pronunciations cause mismatches

7. Types of Potential Errors: Issues on the Caller’s Side Loud Background Noises Speech directed at a person in the background instead of the system Bad phone or connection Unintentional speech (like exclamations) or speech-like noise (coughs, breath)

8. System Evaluation Metrics 1.Non-Programmatic Evaluations Customer Satisfaction Word Error Rate (WER) Dialog System Metrics 2.Programmatic Evaluations PARADISE M. A. Walker, D. Litman, C. A. Kamm, and A. Abella. PARADISE: A general framework for evaluating spoken dialogue agents. In Proceedings of the 35th Annual Meeting of the Association of ComputationalLinguistics, ACL/EACL 97, pages 271-280, 1997.

9. Non-Programmatic Evaluation Customer Satisfaction Relies on customer surveys Poses problem with accuracy and missing/inaccurate information Low Response Rate Typically, the most neglected evaluation, yet the most important

10. Non-Programmatic Evaluation Word Error Rate (WER)/Word Accuracy (WA) WER measures the percentage of words incorrectly recognized WA measures the percentage of words correctly recognized, regardless of insertions, deletions, etc. WER and WA do not measure CPU load, response times, task completion rates, etc., all critical measures for a dialog system

11. Non-Programmatic Evaluation Dialog System Metrics These are not related to customer satisfaction or WER which generally require someone to count frequency Some typical questions: How often do callers hang-up in the middle of the call flow? How often/soon do callers “pound out”? How often does a caller use a word not in the grammar? How many times does the system ask for confirmation from the caller or the caller corrects the system?

12. Programmatic Evaluation Research Approach: PARADISE PARADISE is a complex, research approach to perform the same analysis as the non- programmatic approach. It uses a combination of customer satisfaction surveys along with detailed statistical models to produce the measure of the dialog system performance.

13. Programmatic Evaluation Things You Can Do In PARADISE Measure performance correlations between customer satisfaction and different dialog strategies Measure performance correlations between parts of the system and all of the system Measure performance correlations between very simple and very complex dialog systems/tasks

14. Programmatic vs Non-Programmatic Solutions PARADISE requires a fairly sophisticated level of programming support, as well as extensive knowledge of statistical analysis The Non-Programmatic Metrics are relatively straightforward to calculate and require very little programming

15. Tuning/Testing Tools Does the speech recognition company you either currently use or plan on using, provide them? Will the tools be detailed enough to allow someone within your company to evaluate potential errors and general caller satisfaction? If these tools are not available, your company will need to rely upon caller feedback and the diligence of your application provider?

16. Thank You!