1. Usability Testing

2. Usability Testing What is Usability? What is Usability Testing?
Design the Test
Prepare for the Test
Perform the Test
Process the Data

3. What is Usability?
Usability is the measure of the quality of a user’s experience when interacting with a product or system.
Usability is a quality attribute that assesses how easy user interfaces are to use.
The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

4. What is Usability?
Ease of learning—How fast can a user learn to accomplish basic tasks?
Efficiency of use—How fast can an experienced user accomplish tasks?
Memorability—Can a user remember enough to use it effectively the next time?
Error frequency and severity—How often do users make errors, how serious are these errors, and how do users recover from these errors?
Subjective satisfaction—How much does the user like using the system?

5. What is Usability? Quality of experience
Excellence of user experience
Nielsen’s ease of use
Error Frequency/User Satisfaction
Quantitative analysis
Qualitative analysis

6. What is Usability Testing?
A usability test is a structured process used to explore the interaction between an objective participant and a proposed design
Paper and pencil (paper prototype)
Heuristic evaluation
Cognitive walkthrough
Usability Lab (functional prototype)
Formal testing with subjects

7. What is Usability Testing?
Attributes common to most usability tests:
The goal is to improve a product.
Participants are real users.
The participants do real tasks.
Participants are formally observed.
The data are analyzed.
Recommendations for improvement are made.

8. What is Usability Testing?
A usability test has three basic components:
Participants—Actual users who are asked to perform realistic and representative tasks using a proposed design
Design—May be a fully functioning prototype or a simple paper prototype
Tester—There might be only one tester or there might be a testing team

9. What is Usability Testing?
Constraints on Usability Testing
Time
Design
Prepare
Administer (an hour to an hour and a half)
Analyze the results
Finance
Equipment and software
Laboratory time
Recording media
Participant compensation
Refreshments

10. What is Usability Testing?
Constraints on Usability Testing
Personnel—Formal usability tests require at least four people.
Laboratory—To perform a formal usability test, a dedicated laboratory is required.

11. What is Usability Testing?
Advantages
Minimize help desk calls
Increase product loyalty
Provide benchmarks for future products
Limitations
Artificial context
Not definitive of product acceptance
Skewed sample of users
Not always efficient

12. What is Usability? – Phases of a Usability Test
Design the Test
Prepare for the Test
Perform the Test
Process the Data

13. Design the Test Why: Define the Purpose
What: Define Your Concerns and Goals
What: Define the Tasks
What: Create the Scenarios
What: Define the Measurements
How: Define the Test Method

14. Design the Test
Where: Determine the Location of the Tests

15. Design the Test
Who: Select Participants, Testers, and Observers

16. Design the Test Participants should be real users
You do not always need to test a great many users
The people you recruit should have the following basic characteristics:
Availability
Responsiveness
Objectivity

17. Design the Test Tester roles include the following:
Administrator
Moderator
Data logger
Technician
Prototype expert
Potential observers include the following:
Other design team members not involved in the test
Clients
Programmers responsible for the final product

18. Prepare for the Test When: Create a Test Schedule Project level
Test preparation level
Test execution level
Task execution level

19. Prepare for the Test Writing Scripts Running a Pilot Test
Greeting the Participant
Preliminary Interview
Providing Instructions
Monitoring the Test
Debriefing the Participant
Running a Pilot Test
Be organized
Be presentable

20. Perform the Test Test Phases Pre-Test During the Test
Greet the participant.
Have the participant sign the informed consent form.
Have the participant fill out any pre-test questionnaire.
Proceed with scripts.
During the Test
Maintain a log or observation check list for each task.
Create a problem list to capture anything that is not covered by the check list.
Notate problems and jot down any hypotheses that occur to you about the problems.

21. Perform the Test Test Phases Post-Test Debrief the participant.
post-test questionnaire
verbal interview
Thank the participant and provide compensation.
Collect, summarize, and organize test data.
Reset the room for the next participant.

22. Process the Data Activities Performed on the Day of the Test
Collecting Data
Summarizing Data
Organizing the Material
Follow-Up Activities
Categorizing
Analyzing
Quantitative Data
Qualitative Data

23. Process the Data Documenting Identify problems Prioritize problems
Severity
Frequency
Errors of omission
Prioritize problems
Theorize reasons
Theorize solutions
Identify successes
Identify areas of uncertainty

24. Heuristic Evaluation

25. Heuristic Evaluation Introduction to Heuristic Evaluation
Phases of Heuristic Evaluation
How to perform the Heuristic Evaluation
Heuristics
Examples
Severity Rating
Debriefing
Summary

26. Heuristic Evaluation Developed by Jakob Nielsen
Helps find usability problems in a UI design
Small set (3-5) of evaluators examine UI
independently check for compliance with usability principles (“heuristics”)
different evaluators will find different problems
evaluators only communicate afterwards
findings are then aggregated
Can be performed on working UI or on sketches

27. Why Multiple Evaluators?
Every evaluator doesn’t find every problem
Good evaluators find both easy & hard ones

28. Heuristic Evaluation Evaluators goes through UI several times
inspects various dialogue elements
compares with list of usability principles
consider any additional principles or results that come to mind
Usability principles
Nielsen’s “heuristics”
supplementary list of category-specific heuristics

29. Phases of Heuristic Evaluation
1) Pre-evaluation training
give evaluators needed domain knowledge and information on the scenarios
2) Evaluation
individuals evaluate and then aggregate results
3) Severity rating
determine how severe each problem is (priority)
4) Debriefing
discuss the outcome with design team

30. How to Perform Evaluation
At least two passes for each evaluator
first to get feel for flow and scope of system
second to focus on specific elements
If system is walk-up-and-use or evaluators are domain experts, then no assistance needed
otherwise one might supply evaluators with scenarios
Each evaluator produces list of problems
explain why with reference to heuristic or other information
be specific and list each problem separately

31. Heuristics H1-1: Simple and natural dialog
H1-2: Speak the users’ language
H1-3: Minimize users’ memory load
H1-4: Consistency
H1-5: Feedback
H1-6: Clearly marked exits
H1-7: Shortcuts
H1-8: Precise and constructive error messages
H1-9: Prevent errors
H1-10: Help and documentation

32. Heuristics H2-1: Visibility of system status
keep users informed about what is going on
example: pay attention to response time
0.1 sec: no special indicators needed
1.0 sec: user tends to lose track of data
10 sec: max. duration if user to stay focused on 1 action
for longer delays, use percent-done progress bars

33. Heuristics H2-2: Match between system and real world
speak the users’ language
follow real world conventions
Mac desktop
Dragging disk to trash
should delete it, not eject it

34. Heuristics H2-3: User control and freedom
“exits” for mistaken choices, undo, redo
don’t force down fixed paths
Wizards
must respond to Q1 before going to next
for infrequent tasks
modem config.
not for common tasks
Good for beginners
have 2 versions
WinZip

35. Heuristics
H2-4: Consistency & standards

36. Heuristics H2-5: Error prevention H2-6: Recognition rather than recall
make objects, actions, options, and directions visible or easily retrievable
MS Web Pub. Wiz.
Before dialing
asks for id & password
When connecting
asks again for id & pw

37. Heuristics H2-7: Flexibility and efficiency of use
accelerators for experts (e.g., gestures, kb shortcuts)
allow users to tailor frequent actions (e.g., macros)
H2-8: Aesthetic and minimalist design
no irrelevant information in dialogues

38. Heuristics
H2-9: Help users recognize, diagnose, and recover from errors
error messages in plain language
precisely indicate the problem
constructively suggest a solution

39. Heuristics H2-10: Help and documentation easy to search
focused on the user’s task
list concrete steps to carry out
not too large

40. Examples Can’t copy info from one window to another
violates “Minimize the users’ memory load” (H1-3)
fix: allow copying
Typography uses mix of upper/lower case formats and fonts
violates “Consistency and standards” (H2-4)
slows users down
probably wouldn’t be found by user testing
fix: pick a single format for entire interface

41. Severity Rating Used to allocate resources to fix problems
Estimates of need for more usability efforts
Combination of
frequency
impact
persistence (one time or repeating)
Should be calculated after all evaluations are in
Should be done independently by all judges

42. Severity Ratings (cont.)
0 - don’t agree that this is a usability problem
1 - cosmetic problem
2 - minor usability problem
3 - major usability problem; important to fix
4 - usability catastrophe; imperative to fix

43. Severity Ratings Example
[H1-4 Consistency] [Severity 3][Fix 0]
The interface used the string "Save" on the first screen for saving the user's file, but used the string "Write file" on the second screen. Users may be confused by this different terminology for the same function.

44. Debriefing
Conduct with evaluators, observers, and development team members
Discuss general characteristics of UI
Suggest potential improvements to address major usability problems
Add ratings on how hard things are to fix
Make it a brainstorming session
little criticism until end of session

45. Summary Heuristic evaluation is a discount method
Have evaluators go through the UI twice
Ask them to see if it complies with heuristics
note where it doesn’t and say why
Combine the findings from 3 to 5 evaluators
Have evaluators independently rate severity
Discuss problems with design team
Alternate with user testing