1. 1 High Resolution Statistical Natural Language Understanding: Tools, Processes, and Issues. Roberto Pieraccini SpeechCycle roberto@speechcycle.com

2. 2 Directed Dialog vs. Open Prompt Please choose one of the following: account balance, fund transfer, payments, mortgage rates. Account balance Please tell me what you are calling about. I want to buy a house and I would like to know how much it would cost to borrow money from the bank. DIRECTED DIALOG OPEN PROMPT Context-Free Grammars Statistical Spoken Language Understanding

3. 3 Context-free grammars account balance fund transfer payments mortgage rates account balance fund transfer payments mortgage rates BALANCE TRANSFER PAYMENT MORTGAGE In grammar utterances ANYTHING ELSE NO-MATCH Out of grammar handcrafted SRGS ”account balance” “fund transfer payments ”mortgage rates” CFG N-best semantic categories

4. 4 Statistical Spoken Language Understanding (SSLU) BALANCE TRANSFER PAYMENT MORTGAGE All possible natural language expressions N-best semantic categories SSLU Statistical Language Model (SLM) Statistical Semantic Model (SSM) Provides a probabilistic constraint to the speech recognition engine Classifies a word string into a number of predefined categories N-best word strings OTHER Anything Else Bi-gram language model Statistical classifier

5. 5 Building SSLUs SSLU Training Statistical Language Model (SLM) Statistical Semantic Model (SSM) Annotated Transcriptions I need to know how much money I have  BALANCE I need to move funds from checking to savings  TRANSFER How much would it cost to borrow money to by a house  MORTGAGE I need to pay my utility bills  PAYMENT I dialed the wrong number  OTHER …

6. 6 The accuracy point of view expected responses unexpected responses expected responses unexpected responses DIRECTED DIALOG OPEN PROMPTS WITH SSLU Increase grammar coverage, Tighten prompt More training data High accuracy obtained by limiting unexpected responses and by controlling vocabulary and word confusability Unlimited input and uncontrolled vocabulary results in lower accuracy than directed dialog. When unexpected responses and user’s vocabulary can be controlled, directed dialog typically provides higher accuracy.

7. 7 Why SSLU? Number of options too large for directed dialog. –Please choose one of the following: clothing, automotive, hardware, appliances, …, gardening, …, bedding, … Options make little sense to users –Do you have a hardware, software, or configuration problem? User may chose the wrong option –Hmm…hardware? Unexpected responses and user’s vocabulary hard to control –I need to buy a car CD player that plays MP3s In all these situations, open prompts with SSLU can outperform directed dialog.

8. 8 Low and High Resolution SLUs APPLICATION: Call routing –10s of broad semantic categories APPLICATION: Technical Support –100s of semantic categories –Different degrees of specificity –Detailed confirmation User model differs from underlying model User don’t know the underlying model Low resolution High resolution

9. 9 TV Symptoms On Demand Pay-per-view Ordering No Picture Error PIN Other Error I have a problem with my TV service I could not order a show My movie on demand does not work I ordered a pay-per-view event but all I see is an error code on the display. Hierarchical SSLU

10. 10 TV Symptoms On Demand Pay-per-view Ordering No Picture Error PIN Other Error Hierarchical SSLU I could not order a show I have a problem with my TV service My movie on demand does not work I ordered a pay-per-view event but all I see is an error code on the display. I understand you have a problem with ordering. Is it on demand or pay-per-view?

11. 11 In order to build good SSLU it is important to establish a repeatable process –Transcription management –Creation of annotation guide –Measure annotation consistency –Revise annotation guide –Create VUI

12. 12 Development Cycle Transcription Normalization Linguistic Annotation SSLU Training Symptom and Annotation Guide SSLU Test Review Annotation Guide Develop disambiguation VUI Remove artifacts, acronyms, misspellings Annotate according to what the user says Measure annotation consistency Merge or split categories for better SLU performance Tens of thousands of utterances are needed for creating high performance SLUs

13. 13 SEMANTIC TRUTH SLU RESULTS Confusion Matrix

14. 14 Performance Analysis Utterances10,332100.00% In domain9,32290.22% Correct in-domain7,59181.43% Out of domain1,0109.78% Correct rejection out-of- domain24924.65% SSLU accuracy analysis

15. 15 Confirmation Analysis As a result of the confirmation prompt, users can –Accept a correct hypothesis –Accept a wrong hypothesis –Deny a correct hypothesis –Deny a wrong hypothesis –Do not confirm at all Accepted correct53559.8% Accepted wrong11813.2% Denied correct222.5% Denied wrong637.0% Unconfirmed576.4% No result10011.2% TOTAL895100.0%

16. 16 Experimental VUI The effect of the prompt

17. 17 Conclusions Understanding the choice between SSLU and directed dialog High-resolution SSLU for applications with hundreds of semantic categories. SSLU development process Data, data, data – assessment of performance is key to success.