1. Find out what they like, and how they like it, And let ‘em have it just that way. Fats Waller Usability Testing 101 Gina Pearson, Web Manager January 29, 2003

2. Today’s Goals  Primary purpose is to learn how ERS is integrating usability testing into its Web design & development process  Review why user-centered design and testing (usability engineering) are important  Explain the usability testing process at ERS

3. Today’s Goals  You will learn how to:  Assemble your test team  Recruit & select test participants  Formulate a test plan  Create testing tasks & scenarios  Facilitate a test session  Test without a lab  Record, compile & analyze results  Communicate results effectively

4. Why Is Usability Important?

5. Ballot Usability What should have been addressed? –Font size –Lighting –Text vs. background color –Instructions –Layout –Act of voting (punching ballot card)

6. Ballot Usability  Costs  $20,000 to run usability tests addressing  Layout  Multiple votes  Dimpled ballots  About $1 billion spent by both candidates to try to get elected

7. Failure of Web Sites  Users could find information only 42% of time (15 large commercial sites - Spool study)  Sixty-two percent of web shoppers gave up looking for an item (Zona study)  Only 51% of sites complied with simple Web usability principles (Forrester study of 20 major sites)

8. User-Centered Design  Philosophical & Practical Approach  User-centric - not developer, designer nor organization-centric  Who are the users?  What information do they want?  Based on research and data not opinions  Testable & verifiable

9. The Traditional Development Process

10. User-Centered Design  A methodical approach to producing a web site or any user interface. It is a practical and systematic way to deliver a product that works for users.  Test early & often  Build a little, test a little, fix a little  At ERS, we are incorporating usability testing & other user assessment techniques as a regular part of our web design & development process

11. User-Centered Design  Types of user assessment techniques  Interviews  Ethnographic observation / field research  Focus groups  User personas  Expert (heuristic) reviews  Usability testing  Customer satisfaction surveys  Customer email trend analysis  Search engine query analysis  What else?

12. Myths About Usability Testing  Usability testing:  Is time consuming, expensive & otherwise resource intensive  Typically involves hiring a specialized consulting company  Requires a formal testing lab  Must be videotaped  Always involves writing a detailed analysis and report of results

13. Diagnostic User Testing Find out what is and is not working well on the site. In a diagnostic usability test, you usually want to answer questions like these:  Do users complete a task successfully?  If so, how fast do they do each task? Is that fast enough to satisfy them?  What paths do they take in trying? Do those paths seem efficient enough to them?  Where do they stumble?— What problems do they have?— Where do they get confused?  What words or paths are they looking for that are not now on the site?

14. Methods  Assemble test team  Administrator  Facilitator  Observers / recorders  Video recordings operator (optional)

15. Methods  Define goals & scope of what you are testing  Recruit & compensate participants  Develop testing tasks & script  Set up testing environment  Draft participation agreement  Rehearse & refine test protocol  Conduct tests  Analyze & report out results

16. Methods  Test Environment

17. Methods  How many participants are needed for a usability test?  It depends!  Typical range is from 5 -12 users in each test  Our approach: 3 users per test but conduct several rounds of testing throughout an application or feature’s development  If you have different potential user groups (for example, small business owners, unemployed, blue-collar workers), try to include representatives of all these groups

18. Methods  Participant recruitment & compensation  Typically time and labor intensive - consider outsourcing (average $200 per participant)  Go through professional networks & contacts for volunteers  Conduct testing at conferences & other events with volunteer representatives from your target audiences  Always give volunteers a small token of your appreciation (t-shirt, mug, etc)

19. Methods  Task construction: where do ideas for tasks come from?  Concerns about information architecture, design, terminology, functionality, etc that have surfaced among the product development team  Trend analysis of emails from site customers  Search engine query analysis  Trend analysis of call center inquiries  What else?

20. Methods  Constructing usability testing tasks  Find or scavenger hunt tasks  Verb-based tasks  Scenario-based tasks

21. Methods  Conducting the Test (about 2 hours)  Pre-test briefing for observers  Greet participant & administer participation agreement  Introduce recorder & observers  Explain testing process to participant  Administer pre-test interview & activities (if any)  Facilitate testing tasks  Administer post-test interview questions  Involve observers in general Q&A session at the end  After participant leaves, debrief recorder & observers

22. Methods  Developing the testing script & facilitating the test  3 parts of a usability test:  Pre-test interview  Testing tasks (“think out loud” methodology)  Post-test interview  See another sample script at:  http://www.sensible.com/Downloads/scri pt.doc http://www.sensible.com/Downloads/scri pt.doc

23. Methods  Characteristics of a Good Facilitator  Quick learner  Ability to establish instant rapport  Empathic “people person”  Good listening skills  Patient  Flexible & adaptable  Long & focused attention span  Impartial: not wedded to a particular design

24. Methods  Analyzing & reporting the test results using the KJ method  Assemble your testing team immediately after the last test for a brainstorming session.  Ask each participant to write down each major observed problem on an index card or sticky note.  Put all index cards or sticky notes on a large wall or board, no discussions!  Read each others cards silently, add additional problems.

25. Methods  Analyzing & reporting the test results using the KJ method  Sort problems by area. Eliminate duplicates if total agreement.  Name each group of problem areas.  Vote for most important problems, each participant has ten votes.  Assign one team member the task of writing up the results of brainstorming session.  Disseminate results of brainstorming session to development team via email

26. For Further Information  Contact:  Gina Pearson, Web Manager 694-5119  Brenda Powell, Web Analytics Manager 694-5134

27. For Further Information Don't Make Me Think! A Common Sense Approach to Web Usability Steve Krug The book every manager should read! Handbook of Usability Testing: How to Plan, Design & Conduct Effective Tests By Jeffrey Rubin Practical, step-by-step guidelines, requires no engineering or human factors training. A Practical Guide to Usability Testing By Joseph Dumas & Janice Redish Premier text in the field.

28. For Further Information Jakob Nielsen (www.useit.com) Why You Only Need to Test With 5 Users http://www.useit.com/alertbox/20000319.html Success Rate: The Simplest Usability Metric http://www.useit.com/alertbox/20010218.html Usability Metrics http://www.useit.com/alertbox/20010121.html

29. For Further Information Jakob Nielsen (www.useit.com) Cost of User Testing a Web Site http://www.useit.com/alertbox/980503.html When to Outsource the Recruiting of Test Users http://www.useit.com/alertbox/outsource_recruitin g.html

30. For Further Information Jared Spool (www.uie.com) Using Paper Prototyping to Manage Risks http://world.std.com/~uieweb/paper.htm Usability Labs: Our Take http://world.std.com/~uieweb/labs.htm Eight Is More Than Enough http://world.std.com/%7Euieweb/eight.htm

31. For Further Information Interaction Architect 13 Common Objections Against User Requirements Analysis & Why You Should Not Believe Them http://www.interactionarchitect.com/articles/arti cle20000609b.htm Observation Methods & Tips for Usability Testing http://www.interactionarchitect.com/knowledge/art icle19991212shd.htm

32. For Further Information DialogDesign (www.dialogdesign.dk) Ethics in Human Computer Interaction http://www.dialogdesign.dk/chi2001.html Usability Test of www.TowerRecords.com http://www.dialogdesign.dk/utestreports.html#sam ple

33. For Further Information Usability.gov  Why Should I Do Iterative Usability Testing?  How Can I Encourage People in My Organization to Conduct Usability Engineering and Testing?  Do I Need a Lab to Do Usability Testing?  How Many Participants Are Needed for a Usability Test?  How Much Does It Cost to Do Usability Testing?

34. For Further Information Training Opportunities  National Cancer Institute Usability Seminar Series & Education Training Program, http://usability.gov/events/seminars.html  Society for Technical Communication WDC Chapter, www.stcwdc.org  User Interface Engineering  Human Factors International  User Experience Conferences (Nielsen Norman Group)