1. An Overview of Usability Evaluation #15 1

2. Outline  What is usability evaluation ?  Why perform usability evaluation ?  Types of usability evaluations  What can we learn from:  Heuristic evaluation ?  Usability testing ?  How do we conduct :  Heuristic evaluation ?  Usability test ? 2

3. What Is Usability Evaluation  A systematic process aimed at assessing the fit between a UI design and human capabilities within a task context  It is a central element of the UI design process performed throughout the UI development process 3

4. Why Perform Usability Evaluations?  Find usability problems in an interface design  Assess compliance with style guide ( e.g., MS windows )  Compare alternative UI components  Icon design  Input/output technologies  Assess the worth/usefulness of the software in overall job context 4

5. Evaluation Methods 5 Evaluation CategoriesEvaluation Requirements Evaluation Technique Usability inspection methods A static prototype A UI design expert Heuristic evaluation Evaluation against guidelines Cognitive walkthrough User-based EvaluationsA dynamic prototype A usability analyst Questionnaires Observational usability study Formal usability study with quantitative data analysis Controlled experiments Analytic EvaluationsA UI designer with expiries in analytic techniques Keystroke level model GOMS Grammars

6. Questions of a Usability Evaluation (1)  Are the functions made available in a convenient, task oriented way?  we need to have task knowledge  Does the system anticipate the skill and knowledge of the user?  we need to have user knowledge  Does the design meet general rules of good user interface design?  we need to have UI knowledge 6

7. Questions of a Usability Evaluation (2)  Is the system compliant with other applications running in the user environment?  We need to know the style guide knowledge  How fast can users learn to use the system?  At what speed can users perform various task?  How likely are users to complete a given task ? 7

8. Usability Inspection Methods  Evaluation against guidelines  Heuristics evaluation 8

9. Evaluation Against Guidelines and Rules  A process in which each UI element (e.g., menu choice, icon, button, pointer, radio button) is examined against an existing set of general guidelines and a specific set of design rules (the style guide) applicable to a specific product  Mil STD 1476 F  Windows style guide  Performed by one or more UI design experts who have a thorough familiarity with general UI design guidelines and the product/corporate style guide 9

10. Guidelines and Rules  Guidelines are accepted principles for interface design  Rules specify the interface appearance or action 10

11. Examples of Guidelines  Guidelines  Displays should be consistently formatted  Displays should be uniquely identified  Use short simple sentences  Employ units of measurement that are familiar to the user 11

12. Examples of Design Rules  Design Rules  The character stroke width of a system font shall be at least 2 pixels thick  F10 (and Shift+Menu) exits the menu bar and returns the location cursor to the previous object with focus 12

13. Pro and Cons of Evaluating Against Guidelines  Pros  Provides information on basic design issues  Finds a broad range of usability problems  Cons  Dose not assess whether system meets user's and task needs ( can be compliant and still have poor design)  Time consuming  Guidelines/rules don't exist for all areas of UI design 13

14. Heuristic Evaluation  Popular and widely used structured review of a UI  Objective is to generate a list of potential usability problems  Evaluator assumes the user's role and identifies problems from a user's perspective  Criteria for "a problem" is a set of recognized usability principles called "Heuristics" 14

15. Heuristics Identified by Nielsen (1993)  Use simple and natural dialogue  Speak the users' language  Minimize the users' memory load "‘  Be consistent  Provide feedback  Provide clearly marked exit  Provide shortcuts  Provide good error messages  Prevent errors 15

16. Conducting a Heuristic Evaluation  Collect background information  Identify typical users, scenarios, previous feedback, usability goals  Inspect the flow of the interaction from screen to screen  Inspect screens one at a time against the heuristics  Generate an inspection problem report  Lists and prioritize the usability issues, fixes and/or redesigns 16

17. Who Should Inspect ?  Usability specialists often find more “ problems ” than evaluators with no usability experience (or computer experience only)  Usability specialists with knowledge about a particular kind of interface being developed (Double specialists) find more usability problems than “ regular ” usability specialists 17

18. How Many Inspectors ?  Single evaluator finds only about 35% of the problems  Increasing the number of evaluators from 2 to 5 increases the number of problems found up to around 75% of all the problems  Percent of Problems Found 5 Number of inspectors. Nielsen 1993 18

19. Types of Problems Uncovered by Heuristic Evaluation  Missing or difficult-to-access functionality  Limited or inappropriate task flow  Limited navigational cues  Inappropriate feedback  Cluttered screens 19

20. Pros and Cons of Heuristic Evaluation  Pros  Does not involve users, Relatively inexpensive  Finds a broad range of major and minor usability problems  Maximized by using multiple evaluators  Less intimidating to developers then usability testing  Cons  Subjective and dependent on HCI skills of the evaluators and their knowledge of the task and the users  Depends on how realistically and to what degree the system is exercised  Not exhaustive 20

21. 21 Usability Testing

22.  A set of methods of user-based evaluations  Questionnaires  Observational usability studies  Formal usability studies with quantitative data analysis  Controlled experiments  Observe and measure how users interact with an application  Focus on the direct feedback from end users interacting with the system  Should be the ultimate goal of every evaluation plan because it involves real end users 22

23. The Nature of Usability Testing  Merges several user-based evaluation methods into a single evaluation process  Observation  Interviews  Testing  Each method illuminates a different aspect of usability  Performed after a design (or parts of a design) have been fine tuned based on usability inspection techniques  Performed before a prototype is handed over to developers and a product is sent out 23

24. When is Usability Testing Useful  Test early to:  Evaluate an individual aspect of the design  Significantly affect the design  Provide quick answers to developer  may involve fewer users  may collect less data  Test late to:  Verify the entire application  Stable design  Full functionality  Asses the impact of the design on the user  Controlled variables 24

25. Where Is Usability Testing Performed  User office environment  Users are in their natural surroundings  Easier to recruit users  But --  Uncontrolled environmental setting  Interruptions  Variety of computer configuration  No observation by development team permitted  Usability Lab  Controlled setting  Consistent computer configuration  Data collection equipment  Permits unobtrusive observation by development team 25

26. Who Participates in Usability Testing ?  Evaluators  Usability specialists  Participants  Potential users  Observers  Members of the design/development team 26

27. How Many Participants to Include ?  At least 2 from each distinct user group  2 - 3 at earlier stages of the evaluation when focus is on gross usability issues  6 and up (per user group) at later stages of the evaluation when focus is on performance assessment  Remember, the objective in usability evaluation is not to uncover statistical differences, only design issues 27

28. Measures of Usability  Time to complete task  Completion rate  Number of errors  Types of errors  Severity of errors  Number of requests for help  Number of trials to become proficient in using the system  Comparative ratings  Subjective ratings 28

29. What to Expect From Test Participants  Do the unexpected  Have preconceived ideas  Do not always ask for help  Fail to follow instruction  Quickly develop habits  Are afraid of breaking system  Are apologetic 29

30. Data Collection Techniques  Video Taping  User ’ s interactions with the application  User ’ s facial expressions  Audio Taping  User comments  Observer comments  Data collection applications  Keystroke capture  Indexed videotape  Questionnaires  Interviews  Open ended  Structured 30

31. Summary  Start evaluation early in the design process and continue to evaluate throughout the development cycle  This will minimize the likelihood of a major usability problem emerging during the later phases of the development  Incorporate a variety of evaluation methods  One method cannot predict or identify all the potential usability issues  Include at least one user-based evaluation method in your evaluation plan 31