1. 1 Human-Computer Interaction Usability Evaluation: 1 Introduction and Analytic Methods

2. 2 Lecture Overview Definition and motivation Industrial practice and interest Types of evaluation Analytic methods Heuristic evaluation Keystroke level model Cognitive walkthrough

3. 3 Evaluation: Definition and Motivation Definition Gathering information about usability or potential usability of a system Motivation Suggest improvements or confirm acceptability of interface and/or supporting materials Ensure competitive reputation ‘Users will evaluate your interface sooner or later’ (Hix and Hartson, 1993) Match or exceed usability of competitor’s products (and statutory requirements)

4. 4 MUSiC Project - Metrics for Usability Standards in Computing Early 1990’s - European survey Generally high appreciation of importance of usability evaluation Knowledge of evaluation methods limited Lack of metrics a major problem (after time and money limitations) Intuitiveness of a product and ability to learn it quickly without manuals is a an increasingly important usability factor

5. 5 Reasons why Interface Evaluation is Often Omitted or Poorly Performed Assumption designer’s personal behaviour is ‘representative’ Implicit unsupported assumptions about human performance Acceptance of traditional/standard interface design Postponement of evaluation until ‘a more convenient time’ Lack of expertise in analysing experiments

6. 6 What to Evaluate Usability specifications at all lifecycle stages Initial designs (pre-implementation) Partial Integrated Prototype at various stages Final(?) implementation Documentation

7. 7 Formative Evaluation Repeatedly, as development proceeds Purpose: to support iterative refinement Nature: structured, but fairly informal Average of 3 major ‘design-test-redesign’ cycles, with many minor cycles to check minor changes The earlier poor design features or errors are detected, the easier and cheaper they are to correct

8. 8 Summative Evaluation Once, after implementation (or nearly so) Important in field or ‘beta’ testing Purpose: quality control - product is reviewed to check it meets Own specifications Prescribed standards, e.g. Health and Safety, ISO Nature: formal, often involving statistical inferences

9. 9 Where to Evaluate Designer’s mind Discussion workshops Representative workplace Experimental laboratory

10. 10 How to Evaluate: Evaluation Methods MethodInterfaceUser Involvement development AnalyticSpecificationNo users Expert Specification orNo users prototypeRole playing only ObservationalSimulation orReal users prototype SurveySimulation orReal users prototype ExperimentalNormally fullReal users prototype EmpiricalEmpirical costscostscostscosts

11. 11 Types of Data Quantitative data Objective measures Directly observed E.g. time to complete, accuracy of recall User performances or attitudes can be recorded in a numerical form Qualitative data Subjective responses Reports and opinions that may be categorized in some way but not reduced to numerical values

12. 12 Measurement Tools Semi-structured interview Questionnaire - personal /postal administration Incident diary Feature checklist Focus group Think-aloud Interactive experiment Compare on: Cost Number of subjects

13. 13 Analytic Evaluation Usable early in design Little or no advance planning Cheap Quick Focus on problems Lack of diagnostic output for redesign Encourages strengthening of existing solution Broad assumptions of users’ cognition Can be difficult for evaluator Advantages Disadvantages

14. 14 Analytic Evaluation: Heuristic Evaluation Assess design against known usability criteria e.g. Brown, 1994 Coffee break test Data overload Engineering model Help Mode test Dead ends Unable to complete Stupid questions Jotting test Standalone test Maintenance test Consistency test Reversibility Functionality test Knowledge of completion

15. 15 Analytic Evaluation: Keystroke Level Model (Card et al., 1980) Best known analytic evaluation technique Simple way of analysing expert user performance - usually of unit tasks - say 20 secs Applies constants to operations - total gives completion time for error free dialogue sequence Proven predictive validity (+ 20%) Human motor system is well understood No high-level mental activity

16. 16 KLM Constants OperatorMeaning Time(secs) KPress key (good) 0.12 0.28 1.20 (poor) BMouse button press Down or up0.10 Click0.20 PPoint with mouse1.10 (Fitt’s law: K log 2 (Distance/Size + 0.5) HHand to keyboard or mouse0.40 MMental preparation for physical1.35 action - 1 M per ‘chunk’ RSystem response time Measure Averages - modify to suit

17. 17 KLM - Worked Example (adapted from Browne, 1994) Operation Operator Time Decide to dealM1.35 System response, WindowsR2.00 Locate and grasp mouseH0.40 Point at optionP1.10 Press mouse buttonB0.10 Release mouse buttonB0.10 Identify source (customer)M1.35 Point at source on menuP 1.10 Press mouse buttonB 0.10 System response, windowR1.00 Release mouse buttonB 0.10 Ascertain productM1.35 Press mouse buttonB0.10 Release mouse buttonB0.10 System response, windowR1.00 Calculate quote (complex)M2.70 Return hand to keyboardH0.40 Type 6 quote characters K * 6 1.20 Total 15.55

18. 18 Analytic Evaluation: Cognitive Walkthrough Analyses design in terms of exploratory learning i.e. user Has rough plan Explores system for possible actions Select apparently most appropriate action Interpret system’s response and assess if progresses task Suits systems primarily learned by exploration e.g. walk-up-and-use Overall question - How successfully does this design guide the unfamiliar user through the performance of the task?

19. 19 Analytic Evaluation: Cognitive Walkthrough - Key Questions Simulation of exploration, selection and interpretation at each state of interaction Will the correct action be made sufficiently evident to the user? Will the user connect the correct action’s description with what he or she is trying to do? Will the user interpret the system’s response to the chosen action correctly, that is, will the user know if he or she has made a right or a wrong choice?

20. 20 Lecture Review Definition and motivation Industrial practice and interest Types of evaluation Analytic methods Heuristic evaluation Keystroke level model Cognitive walkthrough