1. Chapter 7 Usability and Evaluation Dov Te’eni Jane M. Carey

2. Copyrights Te'eni & Carey Chapter 7 Usability and Evaluation Usability and evaluation are closely related concepts Usability means that the HCI design of the system supports the user’s cognitive and ergonomic limitations and is easy to use and learn Evaluation means to assess the system for functionality as well as usability We explore how to compare, assess, and improve interfaces

3. Copyrights Te'eni & Carey Mini-Case 1984 Olympic message system demonstrates: 1.You will not get it right the first time 2.You will get different answers when you use different methods of observation 3.You will need to come up with integrative solutions to problems Chapter 7 U sability and Evaluation

4. Copyrights Te'eni & Carey Designers of the OMS (Olympic Message System) followed 3 principles: –Early focus on user and tasks –Empirical measure on user’s behavior –Iterative design Chapter 7 Usability and Evaluation

5. Copyrights Te'eni & Carey Chapter 7 Usability and Evaluation Figure 7.2 Attitudes, use, performance, and satisfaction relationship

6. Copyrights Te'eni & Carey ISO (International Standards Organization) defines usability as “ a concept comprising the effectiveness, efficiency, and satisfaction with which specified users can achieve specified goals in a particular environment” Chapter 7 Usability and Evaluation

7. Copyrights Te'eni & Carey Chapter 7 Usability and Evaluation Table 7-1: Usability indicators based on performance 1. Goal achievement indicators (success rate, failure rate, accuracy, effectiveness). 2. Work rate indicators (speed, completion rate, efficiency, productivity, productivity gain). 3. Operability indicators of the user's ability to make use of the systems features (error rate, problem rate, function usage) 4. Knowledge acquisition indicators of the user's ability and effort in learning to use the system (learnability and learning).

8. Copyrights Te'eni & Carey Chapter 7 Usability and Evaluation Usability measures based on performance and HCI 1. Time to complete a task 2. Number of user commands to complete task 3. Fraction of task completed 4. Fraction of task completed in a given time 5. Number of errors 6. Time spent on errors 7. Frequency of online help used 8. Number of available commands not used 9. When task is repeated, ratio of successes to failures 10. Fraction of positive comments made by user 11. Fraction of good to bad features recalled by user 12. Number of expressions of frustration and satisfaction 13. Number of times user loses control over system 14. Number of times the user needs to devise a way of working around the problem/system.

9. Copyrights Te'eni & Carey Chapter 7 Usability and Evaluation Table 7-3: The usability engineering life cycle · Know the user · Analyze competing products · Set usability goals · Consider alternative designs · Engage in participatory design · Coordinate the total interface · Check against heuristic guidelines · Prototype · Evaluate interface · Design in iterations · Follow up with studies of installed systems

10. Copyrights Te'eni & Carey Methods of usability assessments include: –Thinking aloud (protocol analysis) –Observation –Interviews –Focus groups –Automatic logs –Questionnaires Chapter 7 Usability and Evaluation

11. Copyrights Te'eni & Carey Table 7-4: Attitude questionnaire 1. Using a microcomputer could provide me with information that would lead to a better decisions. 2. I wouldn't use a microcomputer because programming it would take too much time. 3. I'd like to use a microcomputer because it is oriented to user needs. 4. I wouldn't use a microcomputer because it is too time consuming. 5. Using a microcomputer would take too much time away from my normal duties. 6. Using a microcomputer would involve too much time doing mechanical operations (e.g., programming, inputting data) to allow sufficient time for managerial analysis. 7. A microcomputer would be of no use to me because of its limited computing power. 8. I'd like to learn about ways that microcomputers can be used as aids in managerial tasks. 9. Using a microcomputer would result in a tendency to over design simple tasks. 10. I wouldn't want to have a microcomputer at work because it would distract me from my normal job duties. 11. A microcomputer would give me more opportunities to obtain the information that I need 12. I wouldn't favor using a microcomputer because there would be a tendency to use it even when it was more time consuming than manual methods. 13. I'd like to have a microcomputer because it is so easy to use. 14. I'd hesitate to acquire a microcomputer for my use at work because of the difficulty of integrating it with existing information systems. 15. I'd discourage my company from acquiring microcomputers because most application packages would need to be modified before they could be useful in our specific situation. 16. It is easy to access and store data in a microcomputer. 17. A microcomputer would be of no use to me because of the limited availability of application program packages. 18. A microcomputer would be of no use to me because of its small storage capacity. 19. It is easy to retrieve or store information from/to a microcomputer. 20. Using a microcomputer would give me much greater control over important information.

12. Copyrights Te'eni & Carey Chapter 7 Usability and Evaluation Heuristic guidelines 1. Create simple and natural dialog 2. Speak the user's language 3. Minimize the user's memory load 4. Be consistent 5. Provide feedback 6. Provide clearly marked exits 7. Provide shortcuts 8. Provide specific, corrective and positive error messages 9. Minimize propensity for error Step 7 in the engineering life cycle is check against heuristic guidelines

13. Copyrights Te'eni & Carey Evaluation is a broader term than usability – it includes several goals: 1) assess the system's functionality against the intended specifications, 2) assess the system's effect on the user's behavior and attitude, 3) assess the system's impact on measures of performance that are related to the user or the objective of the system, 4) discover unintended problems and perhaps opportunities. Chapter 7 Usability and Evaluation

14. Copyrights Te'eni & Carey Evaluation techniques include: 1) Exploratory vs. model based. 2) Design or implementation. 3) Field study vs. laboratory testing. 4) Design vs. use. 5) Level of performance measures. 6) Degree of designed manipulation and intrusion. Chapter 7 Usability and Evaluation

15. Copyrights Te'eni & Carey Chapter 7 Usability and Evaluation * Task description from the first-time user's viewpoint. Include any special assumptions about the state of the system assumed when the user begins work. * Action sequence: Make a numbered list of the atomic actions that the user should perform to accomplish the task. * Anticipated users: Briefly describe the class of users who will use this system. Note what experience they are expected to have with similar or previous versions. * User's initial goals: List the goals the user is likely to form when starting the task. If there are other likely goals list them, and estimate for each what percentage of user are likely to have them. Cognitive Walkthrough (Polson et al, 1992)

16. Copyrights Te'eni & Carey Chapter 7 Usability and Evaluation Table 7-6: Measures for comparing displays (Scott & Findlay, 1991) · Typing mistakes made before correction · Mistakes remaining · Pauses of 3 seconds or more immediately before mode change · Other pauses of 3 seconds or more · Length of pause immediately before mode change · Attempting to type-over whilst in insert mode · Attempting to insert whilst in type-over mode

17. Copyrights Te'eni & Carey Ben Shneiderman’s Questionnaire for user satisfaction Chapter 7 Usability and Evaluation

18. Copyrights Te'eni & Carey Other methods of evaluation Metrics (Rengger, 1991) Chapter 7 Usability and Evaluation

19. Copyrights Te'eni & Carey Conclusion – Flow diagram from Olson and Moran (1996) Chapter 7 Usability and Evaluation

20. Copyrights Te'eni & Carey Table 7.8 ANSI/HFS 200 outline and status (1997) Chapter 7 Usability and Evaluation Section Status 1 Introduction draft 2 accessibility draft 3 presentation of information ISO document 4 user guidance ISO document 5 direct manipulation ISO document 6 color draft 7 forms fill-in ISO document 8 command languages ISO document 9 voice i/o Voice recognition, non-speech auditory output, interactive voice response drafts 10 visually displayed menus re-drafted ISO doc