1. An Overview of Usability Evaluation
Gabriel Spitz
An Overview of Usability Evaluation

2. Outline What is usability evaluation ?
Why perform usability evaluation ?
Types of usability evaluations
What can we learn from:
Heuristic evaluation ?
Usability testing ?
How do we conduct :
Usability test ?

3. User-Interface design - Thinking
Articulate the Problem
Develop Requierments
Formulate a Conceptual Design
Create a Low Fidelity Design
Design a high Fidelity Product
Evaluate
Ideas, concepts, designs

4. What Is Usability Evaluation
A systematic process aimed at assessing the the fit between a UI design and human capabilities within a task context
It is a central element of the UI design process performed throughout the UI development process
Why?

5. Why Perform Usability Evaluations?
Find usability problems in an interface design
Assess compliance with style guide ( e.g., MS windows, IOS, Google Material, etc. )
Compare alternative UI components
Icon design
Input/output technologies
Assess usefulness of the software in overall job context

6. Scope of Usability Evaluation
Like in many scientific endeavors, there are many and different methods to evaluate
Methods vary in terms of
The purpose of the evaluation – Summative vs. Formative
Stage of the design
What is being evaluated

7. Summary - Evaluation Methods
Evaluation Categories
Evaluation Requirements
Evaluation Technique
Usability inspection methods
A static prototype
A UI design expert
Heuristic evaluation
Evaluation against guidelines
Cognitive walkthrough
User-based Evaluations
A dynamic prototype
A usability analyst
Questionnaires
Observational usability study
Formal usability study with quantitative data analysis
Controlled experiments
Analytic Evaluations
A UI designer with
expiries in analytic techniques
Keystroke level model
GOMS
Grammars

8. Questions of a Usability Evaluation
Is the system compliant with other applications running in the user environment?
We need to know the style guide knowledge
How fast can users learn to use the system?
At what speed can users perform various task?
How likely are users to complete a given task?

9. Usability Inspection Methods
Evaluation against guidelines
Heuristics evaluation

10. Evaluation Against Guidelines and Rules
A systematic process in which each UI element (e.g., menu choice, icon, button, pointer, radio button) is examined against an existing set of general guidelines
Mil STD 1476 F
Windows style guide
Performed by one or more UI design experts who have a thorough familiarity with general UI design guidelines and the product/corporate style guide

11. Guidelines and Rules
Guidelines are accepted principles for interface design
Rules specify the interface appearance or action

12. Examples of Guidelines
Displays should be consistently formatted
Displays should be uniquely identified
Use short simple sentences
Employ units of measurement that are familiar to the user

13. Examples of Design Rules
The character stroke width of a system font shall be at least 2 pixels thick
F10 (and Shift+Menu) exits the menu bar and returns the location cursor to the previous object with focus

14. Evaluating Against Guidelines
Pros
Provides information on basic design issues
Forces a complete and exhaustive review against all the available guidelines
Finds a broad range of usability problems
Cons
Do not assess whether system meets user's needs ( can be compliant and still have poor design)
Is highly time consuming
Guidelines/rules don't exist for all areas of UI design
Some commercial guidelines/rules are poorly chosen and include conflicts
Are not task oriented

15. Heuristic Evaluation
Popular and widely used structured review of a user interface
Objective is to generate a list of potential usability problems
Evaluator assumes the user's role and identifies problems from a user's perspective
Criteria for "a problem" is a set of recognized usability principles called "Heuristics"

16. Heuristics Identified by Nielsen (1993)
Use simple and natural dialogue
Speak the users' language
Minimize the users' memory load "‘
Be consistent
Provide feedback
Provide clearly marked exit
Provide shortcuts
Provide good error messages
Prevent errors

17. Conducting a Heuristic Evaluation
Collect background information
Identify typical users
Build usage scenarios
Review user feedback about predecessor products and usability goals for the current product
Inspect the flow of the interaction from screen to screen
Inspect screens one at a time against the heuristics
Generate an inspection problem report
Lists and prioritize the usability issues
Provide fixes and redesign suggestions
Provide estimates of the cost (time, labor) of implementing the suggested redesign

18. Who Should Inspect ?
Usability specialists often find more “problems” than evaluators with no usability experience (or computer experience only)
Usability specialists with knowledge about a particular kind of interface being developed (Double specialists) find more usability problems than “regular” usability specialists

19. How Many Inspectors ?
Single evaluator finds only about 35% of the problems
Increasing the number of evaluators from 2 to 5 increases the number of problems found up to around 75% of all the problems
Percent of Problems Found Number of inspectors Nielsen 1993

20. Pros and Cons of Heuristic Evaluation
Does not involve users
Finds a broad range of major and minor usability problems
Relatively inexpensive
Maximized by using multiple evaluators
Less intimidating to developers then usability testing
Cons
Subjective and dependent on HCI skills of the evaluators and their knowledge of the task and the users
Depends on how realistically and to what degree the system is exercised
Not exhaustive

21. Usability Testing

22. Usability Testing A set of methods of user-based evaluations
Questionnaires
Observational usability studies
Formal usability studies with quantitative data analysis
Controlled experiments
Observe and measure how users interact with an application
Focus on the direct feedback from end users interacting with the system
Should be the ultimate goal of every evaluation plan because it involves real end users

23. The Nature of Usability Testing
Merges several user-based evaluation methods into a single evaluation process
Observation
Interviews
Testing
Each method illuminates a different aspect of usability
Performed after a design (or parts of a design) have been fine tuned based on usability inspection techniques
Performed before a prototype is handed over to developers and a product is sent out

24. When is Usability Testing Useful
Test early (Formative Evaluation) to:
Evaluate an individual aspect of the design
Significantly affect the design
Provide quick answers to developer
may involve fewer users
may collect less data
Test late (Summative Evaluation) to:
Verify the entire application
Stable design
Full functionality
Asses the impact of the design on the user
Controlled variables

25. Where Is Usability Testing Performed
User office environment
Users are in their natural surroundings
Easier to recruit users
But --
Uncontrolled environmental setting
Interruptions
Variety of computer configuration
No observation by development team permitted
Usability Lab
Controlled setting
Consistent computer configuration
Data collection equipment
Permits unobtrusive observation by development team

26. Who Participates in Usability Testing ?
Evaluators
Usability specialists
Participants
Potential users
Observers
Members of the design/development team

27. How Many Participants to Include ?
At least 2 from each distinct user group
2 - 3 at earlier stages of the evaluation when focus is on gross usability issues
6 and up (per user group) at later stages of the evaluation when focus is on performance assessment
Remember, the objective in usability evaluation is not to uncover statistical differences, only design issues

28. Measures of Usability Time to complete task Completion rate
Number of errors
Types of errors
Severity of errors
Number of requests for help
Number of trials to become proficient in using the system
Comparative ratings
Subjective ratings

29. What to Expect From Test Participants
Do the unexpected
Have preconceived ideas
Do not always ask for help
Fail to follow instruction
Quickly develop habits
Are afraid of breaking system
Are apologetic

30. Data Collection Techniques
Video Taping
User’s interactions with the application
User’s facial expressions
Audio Taping
User comments
Observer comments
Data collection applications
Keystroke capture
Indexed videotape
Questionnaires
Interviews
Open ended
Structured

31. Summary
Start evaluation early in the design process and continue to evaluate throughout the development cycle
This will minimize the likelihood of a major usability problem emerging during the later phases of the development
Incorporate a variety of evaluation methods
One method cannot predict or identify all the potential usability issues
Include at least one user-based evaluation method in your evaluation plan