1. 1 To Example or Not To Example, That is the Natural Language Question Frederick Parkinson, PhD, Project Manager, User Interface Designer Deborah Rapsinski, Senior User Interface Designer

2. 2 Introduction This paper presents best practices around the prompting of natural language recognition states –Structure of prompt –Use of examples

3. 3 What we will talk about Why we need best practices around Natural Language (NL)? What makes designing for NL challenging? What makes for a good NL response? The Nuance best practice for NL Evidence to support this strategy

4. 4 Why we need best practices for NL NL deployments are becoming more common NL deployments incur additional cost, so the demand for quick ROI is more acute Best practices allow adopters to achieve ROI goals more quickly by allowing designers to leverage “lessons learned” from previous deployments

5. 5 Designing for NL is challenging Directed Dialog versus Natural Language Why NL is appealing The Natural Language Paradox –Natural Language states are unnatural –While callers are encouraged to respond naturally, the fact that they are interacting with a computer makes the interaction unnatural

6. 6 How Natural Language systems work Two tasks: –recognition of words in the utterance –interpreting the meaning of those words How Natural Language systems are developed –trained against corpora of tens of thousands of utterances –training data is tagged by hand by humans To work well, the recognizer has to pick out a few salient words that map to specific caller intentions

7. 7 What makes a good Natural Language response A few meaningful words –four to ten words long –one or two meaningful words Good examples of responses –“I need to locate a pharmacy,” “I want to order a refill,” “my power is out,” “close account” Bad examples of responses –“I need help,” “inquiry,” “I need to check about amounts of several drugs that I already have when it's time to order new ones I don't want to order more than I can use”

8. 8 The Nuance best practice Provide the caller with examples –pro: addresses the “deer in the headlights” some callers face –con: humans don’t offer examples Examples provide just enough structure to help callers formulate their responses “In a few words, please tell me why you’re calling today. You can say things like: ‘I need my account balance,’ or ‘what’s the status of my order?’ Okay, go ahead.”

9. 9 What makes a good example? Shows callers how to respond1 statement, 1 question Succinct and Unambiguous Fewest number of turns to destination Makes use of caller profileIf past due: “I’d like to make payment arrangements.”

10. 10 Rotate examples to teach callers of what can be handled –Offer 2 examples at the initial prompt Reduce to 1 for re-prompting or looping through NL state Balance caller goals with business goals –Use examples that will, e.g. encourage callers to pay their bill or inform them of new product offerings Best Practice, Continued

11. 11 Evidence Usability Testing –Inferred success through completed tasks –Post-test interviews Experiments –No example, 2 statements, 2 questions, 1 statement and 1 question… Deployment Data –Fluency, delays –Number of words –Value of responses

12. 12 Evidence: Number of Words Used No Example Example Used

13. 13 Evidence: Fluency No Example Example Used

14. 14 Conclusion We’ve presented the Nuance best practice for the structure of the Natural Language question –Include examples –Examples include one statement, one question –Examples balance callers’ goals and business goals We’ve presented evidence in support of this best practice that shows that callers prefer hearing examples and that examples improve the quality of callers’ responses to the Natural Language question