CEN3722 Human Computer Interaction Usability Evaluation Dr. Ron Eaglin

Objectives Define the term usability Define the term evaluation Describe the three goals of evaluation Describe why usability evaluation is important in the design of interactive systems

Objectives Describe the following usability principles, and provide examples (more specific principles) of each: Learnability Flexibility Robustness Describe the term usability specification, and demonstrate the ability to generate example usability specifications

Objectives Describe the following usability evaluation methods: Heuristic evaluation Cognitive walkthrough User and task observation Severity ratings Scenarios Simplified thinking aloud

Usability - Definition '… the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.' (Karat 1997, p. 691) “The cost of less-than-user-friendly software can be astonishingly high — the combined result of unnecessarily high training and customer support costs, unnecessarily low productivity, and lost market share.” (Mayhew, 1999, p. x).

Usability Evaluation Debugging the Design Goals of Usability Evaluation Assess the extent of the system’s functionality Assess the effect of the interface on the user Identify specific problems with the design

Usability Evaluation Evaluation should occur throughout the design process Expensive mistakes can be avoided prior to major resource commitments Most evaluation method don’t involve users directly Depend upon designers, experts Methods are NOT mutually exclusive

Results of Usability Evaluation List of usability problems Possible hints on how to fix May not be able to fix all problems Need to prioritize Priorities are based on experimental data regarding impact of problems on performance How many people experience the problem How much time wasted (e.g., recovering from errors)

Usability Evaluation Methods Discount usability engineering: Usability inspection methods Many methods Scenarios User and task observation Simplified thinking aloud Heuristic evaluation Cognitive Walkthrough

General Usability Principles Learnability The ease with which new users can begin effective interaction and achieve maximal performance Flexibility The multiplicity of ways the user and system exchange information Robustness The level of support provided to the user in determining successful achievement and assessment of goals.

Learnability Predictability Synthesizability Support for the user to determine the effect of future action based on past interaction history Related principle: Operation visibility Synthesizability Ability of the user to assess the effect of past operations on the current state

Learnability Familiarity Generalizability Consistency Extent to which a user’s knowledge and experience in other real-world or computer-based domains can be applied when interacting with the new system Generalizability Support for the user to extend knowledge of specific interaction within and across applications to other similar situations. Consistency Similarity in input-output behavior between circumstances, situations, or tasks

Flexibility Dialog initiative Multi-threading Task migration ability Allowing the user freedom from artificial constraints on the input dialog imposed by the system Related principle: system/user pre-emptiveness Multi-threading Ability of the system to support more than one user interaction at a time Related principle: Concurrent vs. interleaving Task migration ability Ability to pass control for task execution back and forth between the system and the user

Flexibility Subsitutivity Customizability Allow equivalent values of input and output to be arbitrarily assigned for each other 1 in = 2.5 cm Customizability Modifiability of the user interface Related principle: Adaptability

Robustness Observability Recoverability Ability of the user to evaluate the internal state of the system from its perceivable representation Related principles: Browsability, defaults, reachability, persistence Recoverability Ability of the user to take corrective action once an error has been recognized Related principles: Forward/backward recovery

Robustness Responsiveness Task conformance How the user perceives the rate of communication with the system Related principle: Stability Task conformance The degree to which the system services support all of the tasks the user wishes to perform in a way the user understands Related Principle: Task completeness, adequacy

Usability Specification Concentrates on features that contribute to the usability of the product. Various attributes proposed as a measure of usability – for each attribute these six items are defined to form the usability specification; Measuring Concept Measuring Method Now Level Worst Case Planned Level Best Case

Usability Specification Measuring concept (Usability attribute) “Operationalizing” the concept in terms of the product. Measuring method (Measuring Instrument) How you will measure the concept Now (Current) level Indicates the value in the present system Planned (Target) level Target for the design Worst case Lowest acceptable measurement for the task Best case Level which is agreed to be the best possible measurement

Usability Specification - Example Feature: New dialing interface on phone allowing for quicker access to keypad and dialing. Measuring Concept: Amount of time to dial number on phone Measuring Method: Time to dial and place call Now Level: 22 seconds Planned level: 10 seconds Worst Case: Unable to change average time required Best Case: Planned level met in all cases

Usability Specification Ways to set measurement levels in a usability specification An existing system or previous version Competitive systems Carrying out the task without use of a computer system An absolute scale Your own prototype User’s own earlier performance Adapted from Whiteside, Bennett, and Holtzblatt, 1988

Example Usability Metrics Objective Effectiveness Measure Efficiency Measure Satisfaction Suitability to Task Percentage of goals achieved Time to complete task Rating Scale (1-5) Appropriate for trained users Number of advanced features used Efficiency gains in time for expert users Learnability Percent of functions learned Amount of time required to learn Error Tolerance Percent of errors corrected Amount of time spent correcting errors

Example Usability Metrics Time to complete a task % of task completed Time spent in errors % or number of errors Number of commands used Frequency of help use Number of users preferring the system Number of times users expresses frustration % favorable or unfavorable comments

Usability Evaluation Results Result of usability evaluation is a list of usability problems Possible hints on how to fix May not be able to fix all problems Need to prioritize Set priorities impact of problems on performance How many people experience the problem How much time wasted (e.g., recovering from errors)

Discount Usability Methods User and task observation Simplified thinking aloud Severity ratings Heuristic evaluation Cognitive walkthroughs Scenarios

User and Task Observation Watching users in real work situations doing real tasks Visits to sites where application is installed Watch users perform their everyday tasks Designers don’t interfere Watch users work Record any problems

Thinking Aloud User asked to use system while thinking aloud By verbalizing thoughts users allow observer to determine: Not just what they are doing, but why they are doing it. Can add insight into a user’s thought process Can’t determine why someone is having trouble just by observing, need to know their internal thoughts.

Severity Ratings Evaluations should be conducted independently by all judges Used to allocate resources to fix problems Combination of Frequency: How many users are affected? Impact: How does the problem affect users in terms of achieving their goal? Persistence: How often does it occur to the user? Every time, or only once?

Estimating Severity Few Many Small Low Severity Medium Severity Large Proportion of Users Experiencing Problem Few Many Small Low Severity Medium Severity Impact of Problem Large Medium Severity High Severity

Heuristic Evaluation Heuristic: Rule-of-thumb Procedure Recognition over recall Consistency and standards Procedure Small set (3-5) of evaluators examine the interface or mockup Evaluators independently evaluate for compliance with usability principles (“heuristics”) Different evaluators may find different problems Evaluators only communicate afterwards data are then aggregated

Cognitive Walkthrough Used to assess learnability of interface Several evaluators work together Step through action sequences for each task At each step ask: Why would a user do that?

Scenarios Cheap prototypes Scenario: realistic story about a segment of work user must accomplish Cut down on complexity of implementation by eliminating parts of the full system Ultimate reduction in level of functionality and number of features Because it’s small, can change scenario quickly Good for quick and frequent user feedback

Scenario Design Yu works as a hotel clerk. He receives a call from a customer. The customer would like to change some information for a hotel reservation she made last week. Yu finds the woman’s reservation, and asks the customer what information she would like changed. The customer indicates that she would like to add her companies name to the hotel reservation given they will be billed for the hotel stay.

Evaluation Matrix Real System? Real Users? Scenario Severity Ratings Thinking Aloud NO Heuristic Evaluation Cognitive Walkthrough Real System? Observation Thinking Aloud Severity Ratings Heuristic Evaluation Cognitive Walkthrough YES YES NO Real Users?

Objectives Define the term usability Define the term evaluation Describe the three goals of evaluation Describe why usability evaluation is important in the design of interactive systems

Objectives Describe the following usability principles, and provide examples (more specific principles) of each: Learnability Flexibility Robustness Describe the term usability specification, and demonstrate the ability to generate example usability specifications

Objectives Describe the following usability evaluation methods: Heuristic evaluation Cognitive walkthrough User and task observation Severity ratings Scenarios Simplified thinking aloud