2. A Scientific Approach to Improving VUI Design Peter U. Leppik, CEO Rick Rappe, VP of Business Development Vocal Laboratories Inc.

3. Agenda 1.Principles of Gourmet Customer Service 2.Data Collection Toolbox 3.The Project Life-Cycle 4.Conclusion/Q&A

4. Principles of Gourmet Customer Service Nearly every effort to improve service makes at least one of these mistakes: 1. No data is collected from actual callers. 2. The data collected does not relate to business goals. 3. There is not enough data collected to be statistically significant. 4. There is no context or understanding of the data (for example: what is a “good” satisfactions score?). 5. No action is taken based on the data collected.

5. The Scientific Approach 1.Set Goals o Use relevant business goals, not technical metrics 2.Gather data o Multiple kinds of tests may be needed to get a complete picture 3.Take action 4.Validate o Repeat the tests from Step 2. 5.Commit: Go back to Step 1.

6. Data Collection Toolbox There is no one way to collect data about a VUI or call center. o A VUI designer who only uses one test method is like a carpenter who only owns a hammer.

8. The Toolbox: General Tools Test Goals Statistical Considerations o Margin of Error o Probability of Finding Problems Prototypes o Functional Prototype (aka “Rapid Prototype”) o Wizard-of-Oz Prototype

9. The Toolbox: Idea Generating Tools Heuristics o “Best Practices” o Expert Opinions, “Audits,” “Assessments” Agent Feedback Call Recording and Monitoring

11. The Toolbox: Data Generating Tools Controlled Testing o Lab Tests o Rapid Assessments o Large Controlled Tests Follow-Up Surveys End-of-Call Surveys o Not Recommended due to extreme sample bias

13. The Toolbox: More Data Generating Tools Friends & Family & Employee Tests o Not Recommended due to unrepresentative sample Automated Load Testing Traversal Testing Call Logs and Stats

14. The Project Life-Cycle Every project goes through certain phases. o Analysis o Design o Code o Test o Release o Operational Every phase has different testing requirements

15. Analysis Phase Lay out the project scope, goals, and budget Ask the big questions o Who are these callers, and how do they live? o Why are they using the phone and not (e.g.) the web? o What is the competition doing? Look at the big picture and big ideas: o Generate ideas from consultants, vendors, agents, and customers o Develop baselines & benchmarks

16. Analysis Phase Tools Controlled Testing o Benchmarking, competitive analysis, baseline measurements o Need to understand your callers before you start Agent Feedback Follow-up Surveys Call Recordings Heuristics o “Best practices,” consultants’ opinions, etc.

17. Design Phase This is when you plan the details of the VUI Use a Prototype  Test  Prototype  Test iterative design cycle The keys to success are speed and flexibility o Test several designs, sometimes in the same day o Don’t blow the entire testing budget on The Usability Test

18. Design Phase Tools Prototypes o WOZ or Functional Prototype Controlled testing o Rapid Assessment or Lab Test o Lets you use an iterative design process o Nothing ends arguments faster than solid data.

19. Code Phase Implement the design Most testing is for debugging o Unit testing and some functional testing Relatively little VUI testing takes place here o Implementation may require some VUI changes; these should be tested

20. Test Phase Last stage before rolling out the application to live callers This is where the most thorough testing takes place

21. Test Phase Tools Traversal test o Ensures that the implementation matches the design Large controlled tests o Designed to catch even small usability glitches Automated load testing o Done last because changes in the first two test phases can affect load handling

23. The Test Phase can also be the Acceptance Phase Business goals are what are important, not the exact design o They are what should be in the contract. o Tests which measure business goals should be the acceptance criteria. Many of these tests can be used in the design phase.

24. Release Phase This is the process of rolling out the application By this time, you should be confident that the system works o Confident doesn’t mean cocky This is the time for a phased roll-out o Some call it a “pilot test” o This is a deployment strategy, not a testing strategy

26. Operational Phase The work doesn’t end when the system is live! o Application Drift o External changes Company Caller Lifestyle Competition This looks a lot like the analysis phase for the next project….

27. Operational Phase Tools Benchmarking (Controlled Testing) Competitive analysis (Controlled Testing) Follow-up surveys Agent feedback Monitoring call stats and logs Listening to call recordings

28. Lessons Learned 1.Good service does not have to be more expensive than bad service 2.Customers are smarter than machines 3.Every system can stand some improvement 4.The best VUI designs come from paying attention to the right things at the right time

29. Q & A