1. Usability Testing
I. Omaima Al-Matrafi

2. Why Should You Care About Usability?
Have you ever…
gotten lost in a Web site?
left a site without finding the information you wanted?
waited too long for a page to download?
gone to a site you can’t view or read?
visited a site with outdated information?
Do you want people to visit and return to your site?

3. What is usability?
“A function of particular users performing particular tasks in a particular environment” (Smith et al. 68)
The “people who use the product can do so quickly and easily to accomplish their own tasks” (Dumas and Redish 4)

4. ISO Definition (9241-11) for Usability
“...the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”

5. Definitions (cont.) Ease of learning Efficiency of use
Usability is a combination of factors that affect the user's experience with the product or system, including:
Ease of learning
How fast can a user who has never seen the user interface before learn it sufficiently well to accomplish basic tasks?
Efficiency of use
Once an experienced user has learned to use the system, how fast can he or she accomplish tasks?

6. Definitions (cont.) Memorability Error frequency and severity
If a user has used the system before, can he or she remember enough to use it effectively the next time or does the user have to start over again learning everything?
Error frequency and severity
How often do users make errors while using the system, how serious are these errors, and how do users recover from these errors?
Subjective satisfaction
How much does the user like using the system?
Confusing, frustration and satisfaction

7. Usability Is not functionality Is not beta testing
Functionality is how the system works
Is not beta testing
Beta testing is unfocused and at end of process
Is not market research
Market research is about customer demands

8. Types of Usability Problems
Product doesn’t match job or task
Poor organization/layout
Unexpected occurrence of events
Product not self-evident
Requires recall rather than recognition
Inconsistent screens, messages, terminology
Design is inefficient
Cluttered or unattractive design
No feedback or poor feedback about status or errors
No exit or undo
Help or documentation is not helpful

9. Usability Problem Example: Inconsistent

10. Usability Problem Example: Poor Organization/Layout

11. Usability Problem Example: Product Not Self-evident

12. What is Usability Testing?
An empirical study of a product’s usability by observing actual users do real tasks with the product
The primary goal is to improve the usability of the product
5 characteristic:
Real users
Real tasks
Specific usability goals/concerns
Observing and recording the testing
Data analysis

13. Planning: Define the Purpose
Look at goals of your organization.
Which goals is the Web suited to meet?
Prioritize the goals for your site.
Have one primary goal.
Optionally have secondary goals.
Example goals:
Provide a positive image of NASA / your organization
Provide information/function that the user needs

14. Five Steps for test process
Step 1: User Analysis & Profiles
Step 2: Decide what to Test
Step 3: Preparing for the Testing
Step 4: Conducting the test
Step 5: Analyzing the Data

15. Step 1: User Analysis & Profiles Knowing Your Users
What are your users’ characteristics?
Age, computer literacy, domain knowledge, access methods, browsers, work environments, handicaps, etc.
Collect this information through
Surveys / Questionnaires
Visits to their environment
A typical test includes 6-12 participants

16. Step 2: Decide what to Test
Choose an overall purpose
Example: How useable is our new website?
Determine objectives or what you are testing for. Examples:
Does our search engine provide usable results in the first 5 links returned?
Are search results clear to the users?
Choose type of test:
Performance: Can they do it?
Understandability: Can they understand it?
Read-and-locate: Can they find it?

17. Step 2: Decide what to Test con.
Select tasks:
Consider tasks with a high chance of user failure (complex tasks, one-of-a-kind tasks, highly abstract or technical tasks)
Consider tasks with a high cost of user failure (tasks that require support, like help or support calls, to complete; tasks where data could be damaged or lost)
Consider:
First impressions (look and feel)
First tasks
Tasks most performed
Critical tasks
Specific problem areas
New task for the product

18. Step 2: Decide what to Test con.
Select performance objectives (should be individualized for each task)
Time: How long to complete tasks, to find things, to performance procedures
Error/Success: user errors, attempts to do/find something, numbers of times section re-read, if the task was completed successfully
Five attributes of usability (Nielsen, 1994):
Learnability: system is easy to learn so users can get started quickly
Efficiency: system should be easy to use, resulting in high productivity
Memorability: system should be easy to remember
Errors: system should have low error rate and allow error recovery
Satisfaction: system should be pleasant to use

19. The goal and the concerns of the test Example:
General goal: users can use this program to read the documents easily and quickly.
Quantitative usability goals: users will be able to find the menu choice they want in less than 30 second with no more than two wrong choices and Know the function of each icon from the first time they see it.
General concern for this test: Can users find the correct menu choices and icons as they need them (ease of learn)
Specific concern of this test:
1-unrestanding how can open a PDF file to read it.
2- Accessing particular information easy and quickly by searching the document.
3- Increase the ability of enhancing the vision by zooming the page.
4- The ability of copying a specific part from this document.
5- The ability of making a snapshot for any part of document

20. Planning: User Tasks Why will users come to your site?
What’s in it for them?
Best way to find out is to talk to them!
Create use scenarios to drive your design.
Example user tasks:
Find information about a particular product
Learn about newest projects
Find out about future presentations
Download copy of a document
Contact organization employees
Provide feedback

21. Planning: Example Scenario
Scenarios try to capture a moment in time for the user. A sample scenario might read:
John is working at XCompto make their systems more usable. He remembers that someone at MyComp is involved with usability engineering and wants to contact that person, either by phone or , for more information on doing usability engineering at a government agency.

22. Step 3: Preparing for the Testing
Choose order of tasks: start easy, go sequential, or be random
Create written test materials:
Task list for users
Written welcome speech/ Intro to be read to user
Consent forms
Pre-task and post task questionnaires & interview questions
Other materials
Recruit participants & determine “payment”
Define team member’s roles:
Facilitator/Briefer (necessary): Often only team member to interact with users
Observation recorder/note taker (necessary)
Camera operator (optional)
Help desk operator (optional)
Test administer (optional)

23. Step 3: Preparing for the Testing Cont.
Create written test plan
Practice: conduct walkthroughs of the testing and if possible pilot test (the pilot test users could even be a team member)
Prepare test environment (day of test)

24. Pre-test questionnaire
Ensure that participant meets user profile
Gather additional detail about the participants’ knowledge and experience

25. Post-task questionnaire
Asks questions about a task as soon as it is completed
Captures feedback when it is fresh
Can help measure change in attitude toward the task
Keep it short

26. Description of the test
Tasks and their scenarios
The path for each task
Measuring usability
Typical Criteria for measuring usability
Consent Form
Methods: Details how you will run the test
Pre-Test ( user profile ), Post-Task questionnaires, Post test questioner, and Observation.
Task resources and Procedure:
time, HW, SW and procedure
Preparing the test
Test Team Members: define what each person will do
Measure
Excellent
Acceptable
Unacceptable
Task1: open file
Time for task
<15 sec
15-20 sec
More than 20 sec
S = Selection error 1
More than 1
C= confusing
1 or more

27. Step 4: Conducting the test
Greet & Brief participant:
Read/say welcome
Emphasize that you are not testing them, but the product and that they should act as natural as possible
Explain think-aloud protocol (if using)
Emphasize how user tells you she has completed a task
Stress that the testing is “anonymous”
Be unbiased (especially the Facilitator/Briefer)
Intervene carefully (avoid as much as possible)
Observe and record data
Debrief user

28. Step 5: Analyzing the Data
Collate data into findings:
Choose an approach:
Top-down approach: predetermine categories of findings (like navigation, design, terminology) and go through data looking for “hits”
Bottom-up approach: put each observation on a sticky note/note card, sort into categories and label categories
Determine time and errors/success
Examine findings for each user, user profile, and task
Use analysis techniques such as statistics (even averages help)

29. Your opinion about the program Choices Very bad 1 2 3 4 Very good 5
No. answers
Percentage
0.40
0.20
Screen arrangement
Not logical
logically
0.60
Icons design
ambiguity
understandable
.40
Easy of finishing task
Very easy
Very difficult
Learn of the program
Your opinion on the task
Choices
Very easy 1 2 3 4
Very difficult 5
No. answers
Percentage
.20
Did you finish the task
Yes No
.80
Do you face any problem during executing the task
.40
.60
What is the kind of difficulties
Difficult in navigating between menus
Ambiguity icons
The path to complete the task is long
others .0 .5

30. Step 5: Analyzing the Data con.
Analyze data:
Determine cause of problems
Determine scope/severity of problems
Make recommendations/changes
Report Findings

31. Usability Related Web Sites
Goddard Usability Engineering Center
Usability resources by National Cancer Institute
User Interface Engineering (Jared Spool)
The Usability Professionals’ Association
The Alertbox: Current Issues in Web Usability (Jakob Nielsen)
Book: A Practical Guide to Usability Testing (Dumas & Redish)