PG 1

PG 2 Improving Speech Applications with Usability Surveys How does Nortel measure the ‘Usability Pulse’ of Self Service? Judith Sherwood Sales Engineer Nortel Self Service Solutions

PG 3 Many Usability Test Methods Data from Each Participant Number of Participants Live Pilot Employee Test Calls Follow-Up (call-back) Surveys Focus Groups Usability Survey

PG 4 What is a Usability Survey? Usability testing is an evaluation of a customer touch-point from a user perspective Typically conducted using small focus groups (12-20 subjects) in a controlled studio environment

PG 5 What is a Usability Survey? Methodology: Usability Survey using 200 to panelists to reveal problems affecting fewer callers o Chosen from a large pool of over 80,000 panelists with known demographics

PG 6 Traditional Usability Testing Methods have sample size limitations If a problem affects only 5% of the users: o 10-call sample has a 40% chance of finding it o 100-call sample has a 99% chance of finding it If a problem affects only 1% of the users: o 100-call sample has a 63% chance of finding it o 500-call sample has a 99% chance of finding it Size Matters !

PG 7 Methodology  Panelist places call, completes a task, and then fills out a questionnaire on an internet website  Entire call conversation is recorded for analysis  Survey ties the individual caller experience and questionnaire response to the call recording Call-in Platform  Target Service  Post- Questionnaire Panelist Pool                                Design Tasks & Questionnaire Analysis & Recommen- dations Report

PG 8 Methodology  More efficient, less expensive, faster execution, broader feedback  Panelists recruitment and call-in campaign is outsourced  Analysis provides recommendations for service improvement  Listen to problem calls to suggest ways to fix Call-in Platform  Target Service  Post- Questionnaire Panelist Pool                                Design Tasks & Questionnaire Analysis & Recommen- dations Report

PG 9 Percentile-based letter grading system to compare against other speech applications Raw Scores are based on: o Caller Satisfaction (% very satisfied - % dissatisfied - % very dissatisfied) o Task Completion (% who finish task in one call) o Consistency (variability in call length) Usability Survey Grading Process

PG 10 Lessons Learned: A Case Study “Acme”: A regional Managed-Care Health Insurance Company. Customer Service available for Members and Healthcare Providers Available Tasks in Self-Service: o Members: Check Co-Pay amount and Physician Name Order a replacement ID card o Providers: Check Claim Status Verify Member Status and Co-Pay Initial Usability Survey, then Tuning, followed by 2 nd Survey

PG 11 Initial Results Satisfaction Grade = D Satisfaction Score = %VS – %D – %VD Satisfaction Score = +12 – 29 – 12 = – 29% Call Completion Grade = D Call Completion Score = % accomplished objective in one-call only Call Completion Score = (100–17) x (0.83) = 69%

PG 12 After Tuning Call Completion Grade = C Call Completion Score = (100–7)x(0.86) = 80% Satisfaction Grade = A Satisfaction Score = +50 – 8 – 2 = +40%

PG 13 UI Overall Improvement After Tuning How easy was it for you to accomplish your objective in this call? How satisfied were you with your overall experience? Initial ResultsAfter Tuning 80 % Call Completion 69% Call Completion + 40 % Satis- faction Score – 29% Satis- faction Score

PG 14 Look for the Red Flags of Voice Recognition o Low satisfaction and call-completion scores o Low voice-recognition rating scores o Check Complaints in free responses o Observe if “can’t hear” problems for low-score panelists Work with Developer and IT support o Is speech level strong enough in the IVR? Check switch gain levels o Then check speech detector parameters o Then check recognition confidence thresholds and grammars Best Practice 1: Catch Recognition Problems Quickly

PG 15 Voice Recognition Issues How well or poorly did the system recognize your responses when you spoke the answer to questions ? Fix: Increased digital gain from host switch Initial ResultsAfter Tuning How quickly or slowly did the system respond to your spoken answers?

PG 16 Look for the Red Flags of Prompt Confusion o Low satisfaction and call-completion scores o Low ‘What-to-Speak’ scores o Listen for caller hesitations for low-score panelists Work with Dialog Designer o Let callers know ahead of time that they can speak o Reword prompts; Callers appreciate clear choices o Give Touch-Tone options when reprompting o Coach your voice actor Best Practice 2: Spot Prompt Clarity Confusions

PG 17 Voice-Prompt Issues Was it clear what you needed to select or say at each step of the call? How appropriate or inappropriate was the speaking style and voice of this service? Initial ResultsAfter Tuning

PG 18 Voice-Prompt Issues Fix: Clarify prompt choice wording o Initial: “ Are you a member, provider, or a non- member looking for information? ” o After Tuning: “ First, tell me who you are: a member, a provider, or a non-member. ” Fix: It ’ s not just what you say, but how you say it. o Coach your voice talent for proper inflections.

PG 19 Look for the Red Flags of Call-Flow Frustration o Low satisfaction and call-completion scores o Low Enough-Choices scores o Check complaints about inflexible systems o Listen for caller frustrations for low-score panelists Work with Dialog Designer o Look for easy ways to complete repetitive tasks o Provide an easy exit strategy o Leverage the spoken language instinct Best Practice 3: Spot Call Flow Frustrations

PG 20 Call-Flow Issues When you were given menu choices, were you given too many, just enough, or too few? After Tuning Initial Results

PG 21 Call-Flow Issues Fix: To clarify options, Provide Anchor, Split Main Menu choices, and Add Grammar Synonyms o Initial : o “ Alright, I ’ m going to tell you the things I can help you with. When you hear the right one, just say it … Verify member status and office co-pays. Get status of a claim. Get member ID cards. Change PCP. Or, order forms and literature. o After Tuning : o “ Alright, Main Menu. Please say one of the following options at any time … Verify member status. Check PCP co-pay. Get claim status. Order ID cards. Change PCP. Order forms. Or, order literature …”

PG 22 Look for the Red Flags of Task Differences o Low satisfaction and call-completion scores for Specific Tasks o Check complaints about complicated tasks o Listen for caller confusion in low-scoring tasks Work with Dialog Designer o Look for easy ways to complete repetitive tasks o Make sure task instructions are clear o Provide an easy operator exit strategy Best Practice 4: Spot Task Differences

PG 23 Task Differences Relative Task Satisfaction can change after tuning o Can’t Hear problem can swamp other UI task issues o Claim Status Sat. low for business reasons (no shortcut for multiple claims)

PG 24 Task Differences Fixes o Improve recognition to shift all scores up and reveal other UI issues o Future: Give clearer exit menu options o Future: For Claim Status, offer stream-lined repeat-claim function

PG 25 The Usability Survey method is very effective for tuning applications prior to pilot production o Collect hundreds of calls and analyze results efficiently Recognition/Prompt/Call-Flow issues are revealed quickly in Usability Survey o Listen to calls for which various ratings are low Task differences show the effect of Task Consistency, Complexity, and Business Rules on Usability Quality o Longer tasks require more call-flow efficiency and friendly hand-holding. o Transfer to Agent for business reasons can sometimes lower satisfaction if callers must first spend a long time in Self-Serve, or if transfer takes place with no explanation. Conclusions

PG 26 Thank You!