1. Mining for What’s Missing: How to Find What’s Not in the Speech Application’s Vocabulary AMY NEUSTEIN, Ph.D. LINGUISTIC TECNOLOGY SYSTEMS lingtec@banet.net SpeechTEK 2004

2. First Problem: Critical business intelligence data is lost in a sea of recorded calls when callers use words outside of the application’s vocabulary

3. Second Problem: Early warning signs of caller frustration are hard to detect when callers do not use expected “keywords” from the application’s vocabulary to express frustration

4. Third Problem: To build a Statistical Language Model to accommodate all the ways users might express themselves would require a very large data corpus that is costly to assemble; and still there would be no guarantee that an accurate word match would be found.

5. THE SOLUTION: SEQUENCE PACKAGE ANALYSIS A new natural language intelligence method that has been successfully peer reviewed; and cited by other researchers as a data mining method for call center quality monitoring.

6. METHODOLOGY SPA draws mainly from the field of conversation analysis: the study of the orderly properties of interactive dialog that revolve around the turn-taking process; and other sequentially based features that are part of that process such as spacing between turns and overlap of turns

7. How Does Sequence Package Analysis (SPA) Work? SPA parses NL dialog to locate a series of related turns, discretely packaged as a sequence of conversational interaction. SPA locates generic sequence packages, rather than isolated key words, because speakers are more likely to vary in their choice of words than in their basic conversational sequence patterns.

8. SPA provides a “filter” for the front end of a speech recognizer, using generic templates that can be deployed in many different applications and languages. A SPA “add on” layer can be used with conventional vector-based n-gram language models, which hold spaces and determine “global weighting” of specific lexical items. WHERE DOES SPA FIT ON THE SPEECH RECOGNIZER?

9. MINING HELP-LINE CALLS Using SPA to caption the text of a help - line call to capture signs of caller frustration SPA mining tools are based on the detection of conversational sequence patterns rather than solely on word spotting (“get me a supervisor!”) or changes in prosody (e.g., increased pitch) While speakers can vary widely in their choice of words or in stress patterns, conversational sequence patterns are more consistent across a wide spectrum of callers

10. Australian Help-Line Desk Caller: “I’ve installed Office 97 and…I was a bit stupid. I went into uninstall and um pulled off a whole stack of items off the uninstall and it was a very silly thing to do so now when I start up my computer I get a screen um which say um a black- a black and white screen which says never delete this item. It’s a message screen and every time I start up it comes up……[deleted text]……... Caller: “I’m wondering if I reinstall will I wipe out [my documents]” Agent: “Okay, well look I could certainly have a technician look at the problem for you; we do charge for are you aware of that?” Caller: “I’m just asking a question - I’m just wondering whether or not I should uninstall Microsoft Word?”

11. Using SPA to Find CONVERSATIONAL SEQUENCE PATTERNS in this Dialog Sample Step One: Locate the pre-question phrases of reports of troubles and requests for assistance: “I’m wondering if” “I’m just asking a question” “I’m just wondering whether or not” Step Two: Quantify the number of times and the proximity of such pre-question phrases. Step Three: Determine if they escalate or, in the alternative, diminish?

12. ANALYSIS The caller to the Australian help-line began her complaint as a long winded narrative, but with the noticeable absence of a request for help. The caller later produced pre-question phrases when she made her request for help However, these phrases began to escalate (by being combined with one another) just at the point where she began to show signs of frustration: “I’m just asking a question - I’m just wondering whether or not I should uninstall Microsoft Word?”

13. CODA Conventional data mining programs would have“missed” these signs of caller frustration in that they try to locate keywords and phrases: “get me a supervisor” “I’m frustrated because I’m really not getting answers to my questions.” SPA offers as an add on layer to mining programs in order to locate what is missing