1. CEN3722 Human Computer Interaction Usability Evaluation
Dr. Ron Eaglin

2. Objectives Define the term usability Define the term evaluation
Describe the three goals of evaluation
Describe why usability evaluation is important in the design of interactive systems

3. Objectives
Describe the following usability principles, and provide examples (more specific principles) of each:
Learnability
Flexibility
Robustness
Describe the term usability specification, and demonstrate the ability to generate example usability specifications

4. Objectives Describe the following usability evaluation methods:
Heuristic evaluation
Cognitive walkthrough
User and task observation
Severity ratings
Scenarios
Simplified thinking aloud

5. Usability - Definition
'… the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.' (Karat 1997, p. 691)
“The cost of less-than-user-friendly software can be astonishingly high — the combined result of unnecessarily high training and customer support costs, unnecessarily low productivity, and lost market share.” (Mayhew, 1999, p. x).

6. Usability Evaluation Debugging the Design
Goals of Usability Evaluation
Assess the extent of the system’s functionality
Assess the effect of the interface on the user
Identify specific problems with the design

7. Usability Evaluation
Evaluation should occur throughout the design process
Expensive mistakes can be avoided prior to major resource commitments
Most evaluation method don’t involve users directly
Depend upon designers, experts
Methods are NOT mutually exclusive

8. Results of Usability Evaluation
List of usability problems
Possible hints on how to fix
May not be able to fix all problems
Need to prioritize
Priorities are based on experimental data regarding impact of problems on performance
How many people experience the problem
How much time wasted (e.g., recovering from errors)

9. Usability Evaluation Methods
Discount usability engineering:
Usability inspection methods
Many methods
Scenarios
User and task observation
Simplified thinking aloud
Heuristic evaluation
Cognitive Walkthrough

10. General Usability Principles
Learnability
The ease with which new users can begin effective interaction and achieve maximal performance
Flexibility
The multiplicity of ways the user and system exchange information
Robustness
The level of support provided to the user in determining successful achievement and assessment of goals.

11. Learnability Predictability Synthesizability
Support for the user to determine the effect of future action based on past interaction history
Related principle: Operation visibility
Synthesizability
Ability of the user to assess the effect of past operations on the current state

12. Learnability Familiarity Generalizability Consistency
Extent to which a user’s knowledge and experience in other real-world or computer-based domains can be applied when interacting with the new system
Generalizability
Support for the user to extend knowledge of specific interaction within and across applications to other similar situations.
Consistency
Similarity in input-output behavior between circumstances, situations, or tasks

13. Flexibility Dialog initiative Multi-threading Task migration ability
Allowing the user freedom from artificial constraints on the input dialog imposed by the system
Related principle: system/user pre-emptiveness
Multi-threading
Ability of the system to support more than one user interaction at a time
Related principle: Concurrent vs. interleaving
Task migration ability
Ability to pass control for task execution back and forth between the system and the user

14. Flexibility Subsitutivity Customizability
Allow equivalent values of input and output to be arbitrarily assigned for each other
1 in = 2.5 cm
Customizability
Modifiability of the user interface
Related principle: Adaptability

15. Robustness Observability Recoverability
Ability of the user to evaluate the internal state of the system from its perceivable representation
Related principles: Browsability, defaults, reachability, persistence
Recoverability
Ability of the user to take corrective action once an error has been recognized
Related principles: Forward/backward recovery

16. Robustness Responsiveness Task conformance
How the user perceives the rate of communication with the system
Related principle: Stability
Task conformance
The degree to which the system services support all of the tasks the user wishes to perform in a way the user understands
Related Principle: Task completeness, adequacy

17. Usability Specification
Concentrates on features that contribute to the usability of the product.
Various attributes proposed as a measure of usability – for each attribute these six items are defined to form the usability specification;
Measuring Concept Measuring Method
Now Level Worst Case
Planned Level Best Case

18. Usability Specification
Measuring concept (Usability attribute)
“Operationalizing” the concept in terms of the product.
Measuring method (Measuring Instrument)
How you will measure the concept
Now (Current) level
Indicates the value in the present system
Planned (Target) level
Target for the design
Worst case
Lowest acceptable measurement for the task
Best case
Level which is agreed to be the best possible measurement

19. Usability Specification - Example
Feature: New dialing interface on phone allowing for quicker access to keypad and dialing.
Measuring Concept: Amount of time to dial number on phone
Measuring Method: Time to dial and place call
Now Level: 22 seconds
Planned level: 10 seconds
Worst Case: Unable to change average time required
Best Case: Planned level met in all cases

20. Usability Specification
Ways to set measurement levels in a usability specification
An existing system or previous version
Competitive systems
Carrying out the task without use of a computer system
An absolute scale
Your own prototype
User’s own earlier performance
Adapted from Whiteside, Bennett, and Holtzblatt, 1988

21. Example Usability Metrics
Objective
Effectiveness Measure
Efficiency
Measure
Satisfaction
Suitability to Task
Percentage of goals achieved
Time to complete task
Rating Scale (1-5)
Appropriate for trained users
Number of advanced features used
Efficiency gains in time for expert users
Learnability
Percent of functions learned
Amount of time required to learn
Error Tolerance
Percent of errors corrected
Amount of time spent correcting errors

22. Example Usability Metrics
Time to complete a task
% of task completed
Time spent in errors
% or number of errors
Number of commands used
Frequency of help use
Number of users preferring the system
Number of times users expresses frustration
% favorable or unfavorable comments

23. Usability Evaluation Results
Result of usability evaluation is a list of usability problems
Possible hints on how to fix
May not be able to fix all problems
Need to prioritize
Set priorities impact of problems on performance
How many people experience the problem
How much time wasted (e.g., recovering from errors)

24. Discount Usability Methods
User and task observation
Simplified thinking aloud
Severity ratings
Heuristic evaluation
Cognitive walkthroughs
Scenarios

25. User and Task Observation
Watching users in real work situations doing real tasks
Visits to sites where application is installed
Watch users perform their everyday tasks
Designers don’t interfere
Watch users work
Record any problems

26. Thinking Aloud User asked to use system while thinking aloud
By verbalizing thoughts users allow observer to determine:
Not just what they are doing, but why they are doing it.
Can add insight into a user’s thought process
Can’t determine why someone is having trouble just by observing, need to know their internal thoughts.

27. Severity Ratings
Evaluations should be conducted independently by all judges
Used to allocate resources to fix problems
Combination of
Frequency: How many users are affected?
Impact: How does the problem affect users in terms of achieving their goal?
Persistence: How often does it occur to the user? Every time, or only once?

28. Estimating Severity Few Many Small Low Severity Medium Severity Large
Proportion of Users Experiencing Problem
Few
Many
Small
Low Severity
Medium Severity
Impact of Problem
Large
Medium Severity
High Severity

29. Heuristic Evaluation Heuristic: Rule-of-thumb Procedure
Recognition over recall
Consistency and standards
Procedure
Small set (3-5) of evaluators examine the interface or mockup
Evaluators independently evaluate for compliance with usability principles (“heuristics”)
Different evaluators may find different problems
Evaluators only communicate afterwards
data are then aggregated

30. Cognitive Walkthrough
Used to assess learnability of interface
Several evaluators work together
Step through action sequences for each task
At each step ask: Why would a user do that?

31. Scenarios Cheap prototypes
Scenario: realistic story about a segment of work user must accomplish
Cut down on complexity of implementation by eliminating parts of the full system
Ultimate reduction in level of functionality and number of features
Because it’s small, can change scenario quickly
Good for quick and frequent user feedback

32. Scenario Design
Yu works as a hotel clerk. He receives a call from a customer. The customer would like to change some information
for a hotel reservation she made last week. Yu finds the woman’s reservation, and asks the
customer what information she would like changed. The customer indicates that she would like to add her companies name to the hotel
reservation given they will be
billed for the hotel stay.

33. Evaluation Matrix Real System? Real Users? Scenario Severity Ratings
Thinking Aloud NO
Heuristic Evaluation
Cognitive Walkthrough
Real System?
Observation
Thinking Aloud
Severity Ratings
Heuristic Evaluation
Cognitive Walkthrough
YES
YES NO
Real Users?

34. Objectives Define the term usability Define the term evaluation
Describe the three goals of evaluation
Describe why usability evaluation is important in the design of interactive systems

35. Objectives
Describe the following usability principles, and provide examples (more specific principles) of each:
Learnability
Flexibility
Robustness
Describe the term usability specification, and demonstrate the ability to generate example usability specifications

36. Objectives Describe the following usability evaluation methods:
Heuristic evaluation
Cognitive walkthrough
User and task observation
Severity ratings
Scenarios
Simplified thinking aloud