1. Gina Pearson, Web Manager
Find out what they like, and how they like it, And let ‘em have it just that way Fats Waller
Usability Testing 101
Gina Pearson, Web Manager
January 29, 2003

2. Today’s Goals
Primary purpose is to learn how ERS is integrating usability testing into its Web design & development process
Review why user-centered design and testing (usability engineering) are important
Explain the usability testing process at ERS

3. Today’s Goals You will learn how to: Assemble your test team
Recruit & select test participants
Formulate a test plan
Create testing tasks & scenarios
Facilitate a test session
Test without a lab
Record, compile & analyze results
Communicate results effectively

4. Why Is Usability Important?

5. Ballot Usability What should have been addressed? Font size Lighting
Text vs. background color
Instructions
Layout
Act of voting (punching ballot card)

6. Ballot Usability Costs $20,000 to run usability tests addressing
Layout
Multiple votes
Dimpled ballots
About $1 billion spent by both candidates to try to get elected

7. Failure of Web Sites
Users could find information only 42% of time (15 large commercial sites - Spool study)
Sixty-two percent of web shoppers gave up looking for an item (Zona study)
Only 51% of sites complied with simple Web usability principles (Forrester study of 20 major sites)

8. User-Centered Design Philosophical & Practical Approach
User-centric - not developer, designer nor organization-centric
Who are the users?
What information do they want?
Based on research and data not opinions
Testable & verifiable

9. The Traditional Development Process

10. User-Centered Design
A methodical approach to producing a web site or any user interface. It is a practical and systematic way to deliver a product that works for users.
Test early & often
Build a little, test a little, fix a little
At ERS, we are incorporating usability testing & other user assessment techniques as a regular part of our web design & development process

11. User-Centered Design Types of user assessment techniques Interviews
Ethnographic observation / field research
Focus groups
User personas
Expert (heuristic) reviews
Usability testing
Customer satisfaction surveys
Customer trend analysis
Search engine query analysis
What else?

12. Myths About Usability Testing
Is time consuming, expensive & otherwise resource intensive
Typically involves hiring a specialized consulting company
Requires a formal testing lab
Must be videotaped
Always involves writing a detailed analysis and report of results

13. Diagnostic User Testing
Find out what is and is not working well on the site. In a diagnostic usability test, you usually want to answer questions like these:
Do users complete a task successfully?
If so, how fast do they do each task? Is that fast enough to satisfy them?
What paths do they take in trying? Do those paths seem efficient enough to them?
Where do they stumble?— What problems do they have?— Where do they get confused?
What words or paths are they looking for that are not now on the site?

14. Methods Assemble test team Administrator Facilitator
Observers / recorders
Video recordings operator (optional)

15. Methods Define goals & scope of what you are testing
Recruit & compensate participants
Develop testing tasks & script
Set up testing environment
Draft participation agreement
Rehearse & refine test protocol
Conduct tests
Analyze & report out results

16. Methods
Test Environment

17. Methods How many participants are needed for a usability test?
It depends!
Typical range is from users in each test
Our approach: 3 users per test but conduct several rounds of testing throughout an application or feature’s development
If you have different potential user groups (for example, small business owners, unemployed, blue-collar workers), try to include representatives of all these groups

18. Methods Participant recruitment & compensation
Typically time and labor intensive - consider outsourcing (average $200 per participant)
Go through professional networks & contacts for volunteers
Conduct testing at conferences & other events with volunteer representatives from your target audiences
Always give volunteers a small token of your appreciation (t-shirt, mug, etc)

19. Methods Task construction: where do ideas for tasks come from?
Concerns about information architecture, design, terminology, functionality, etc that have surfaced among the product development team
Trend analysis of s from site customers
Search engine query analysis
Trend analysis of call center inquiries
What else?

20. Methods Constructing usability testing tasks
Find or scavenger hunt tasks
Verb-based tasks
Scenario-based tasks

21. Methods Conducting the Test (about 2 hours)
Pre-test briefing for observers
Greet participant & administer participation agreement
Introduce recorder & observers
Explain testing process to participant
Administer pre-test interview & activities (if any)
Facilitate testing tasks
Administer post-test interview questions
Involve observers in general Q&A session at the end
After participant leaves, debrief recorder & observers

22. Methods Developing the testing script & facilitating the test
3 parts of a usability test:
Pre-test interview
Testing tasks (“think out loud” methodology)
Post-test interview
See another sample script at:

23. Methods Characteristics of a Good Facilitator Quick learner
Ability to establish instant rapport
Empathic “people person”
Good listening skills
Patient
Flexible & adaptable
Long & focused attention span
Impartial: not wedded to a particular design

24. Methods Analyzing & reporting the test results using the KJ method
Assemble your testing team immediately after the last test for a brainstorming session.
Ask each participant to write down each major observed problem on an index card or sticky note.
Put all index cards or sticky notes on a large wall or board, no discussions!
Read each others cards silently, add additional problems.

25. Methods Analyzing & reporting the test results using the KJ method
Sort problems by area. Eliminate duplicates if total agreement.
Name each group of problem areas.
Vote for most important problems, each participant has ten votes.
Assign one team member the task of writing up the results of brainstorming session.
Disseminate results of brainstorming session to development team via

26. For Further Information
Contact:
Gina Pearson, Web Manager
Brenda Powell, Web Analytics Manager

27. For Further Information
Don't Make Me Think! A Common Sense Approach to Web Usability
Steve Krug
The book every manager should read!
Handbook of Usability Testing: How to Plan, Design & Conduct Effective Tests
By Jeffrey Rubin
Practical, step-by-step guidelines, requires no engineering or human factors training.
A Practical Guide to Usability Testing
By Joseph Dumas & Janice Redish
Premier text in the field.

28. For Further Information
Jakob Nielsen (
Why You Only Need to Test With 5 Users
Success Rate: The Simplest Usability Metric
Usability Metrics

29. For Further Information
Jakob Nielsen (
Cost of User Testing a Web Site
When to Outsource the Recruiting of Test Users

30. For Further Information
Jared Spool (
Using Paper Prototyping to Manage Risks
Usability Labs: Our Take
Eight Is More Than Enough

31. For Further Information
Interaction Architect
13 Common Objections Against User Requirements Analysis & Why You Should Not Believe Them
Observation Methods & Tips for Usability Testing

32. For Further Information
DialogDesign (
Ethics in Human Computer Interaction
Usability Test of

33. For Further Information
Usability.gov
Why Should I Do Iterative Usability Testing?
How Can I Encourage People in My Organization to Conduct Usability Engineering and Testing?
Do I Need a Lab to Do Usability Testing?
How Many Participants Are Needed for a Usability Test?
How Much Does It Cost to Do Usability Testing?

34. For Further Information
Training Opportunities
National Cancer Institute
Usability Seminar Series & Education Training Program,
Society for Technical Communication WDC Chapter,
User Interface Engineering
Human Factors International
User Experience Conferences (Nielsen Norman Group)