Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Lecturer – Prof Jim Warren Lecture 24 - Usability Testing Based on Heim, Chapter 8

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-2 Chapter 8 Usability Testing What is Usability? What is Usability Testing? Design the Test Prepare for the Test Perform the Test Process the Data

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-3 What is Usability? Usability is the measure of the quality of a user’s experience when interacting with a product or system. (Usability.gov, 2006) Usability is a quality attribute that assesses how easy user interfaces are to use. (Nielsen, 2003) The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. (ISO standard , 1998)

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-4 What is Usability? Ease of learning—How fast can a user learn to accomplish basic tasks? Efficiency of use—How fast can an experienced user accomplish tasks? Memorability—Can a user remember enough to use it effectively the next time? Error frequency and severity—How often do users make errors, how serious are these errors, and how do users recover from these errors? Subjective satisfaction—How much does the user like using the system?

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-5 What is Usability? Quality of experience –Excellence of user experience –Nielsen’s ease of use Error Frequency/User Satisfaction –Quantitative analysis –Qualitative analysis

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-6 What is Usability Testing? A usability test is a structured process used to explore the interaction between an objective participant and a proposed design Paper and pencil (paper prototype) –Heuristic evaluation –Cognitive walkthrough Usability Lab (functional prototype) –Formal testing with subjects

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-7 What is Usability Testing? Attributes common to most usability tests: –The goal is to improve a product. –Participants are real users. –The participants do real tasks. –Participants are formally observed. –The data are analyzed. –Recommendations for improvement are made. Dumas and Redish (1999)

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-8 What is Usability Testing? A usability test has three basic components: –Participants—Actual users who are asked to perform realistic and representative tasks using a proposed design –Design—May be a fully functioning prototype or a simple paper prototype –Tester—There might be only one tester or there might be a testing team

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-9 What is Usability Testing? Constraints on Usability Testing –Time Design Prepare Administer (an hour to an hour and a half) Analyze the results –Finance Equipment and software Laboratory time Recording media Participant compensation Refreshments

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-10 What is Usability Testing? Constraints on Usability Testing –Personnel—Formal usability tests require at least four people. –Laboratory—To perform a formal usability test, a dedicated laboratory is required.

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-11 What is Usability Testing? Human Subjects Protocols –You must be fully aware of the regulations imposed by the various institutions and regulatory bodies that pertain to your experimental design –The U.S. Department of Health and Human Services Web site

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-12 What is Usability Testing? Advantages –Minimize help desk calls –Increase product loyalty –Provide benchmarks for future products Limitations –Artificial context –Not definitive of product acceptance –Skewed sample of users –Not always efficient

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-13 What is Usability? – Phases of a Usability Test Design the Test Prepare for the Test Perform the Test Process the Data

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-14 Design the Test Why: Define the Purpose What: Define Your Concerns and Goals What: Define the Tasks What: Create the Scenarios What: Define the Measurements How: Define the Test Method

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-15 Design the Test Where: Determine the Location of the Tests

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-16 Design the Test Who: Select Participants, Testers, and Observers

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-17 Design the Test Participants should be real users You do not always need to test a great many users The people you recruit should have the following basic characteristics: –Availability –Responsiveness –Objectivity

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-18 Design the Test Tester roles include the following: –Administrator –Moderator –Data logger –Technician –Prototype expert Potential observers include the following: –Other design team members not involved in the test –Clients –Programmers responsible for the final product

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-19 Prepare for the Test When: Create a Test Schedule –Project level You must build time for (iterative!) testing into the overall project plan right from the conception of the project; even if it’s just an estimate –Test preparation level Scheduling the testers, the participants, and the facility (if it’s specialised, or on site in the field) –Test and task execution level Must account for fatigue Organisation and preparation (if the user is going to do a 30-minute task, you’ll need an hour [to greet, settle in, debrief, etc.) – and more time between test participants)

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-20 Prepare for the Test Writing Scripts –Greeting the Participant –Preliminary Interview –Providing Instructions –Monitoring the Test –Debriefing the Participant Running a Pilot Test –Be organized –Be presentable

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-21 Perform the Test Test Phases –Pre-Test Greet the participant. Have the participant sign the informed consent form. Have the participant fill out any pre-test questionnaire. Proceed with scripts. –During the Test Maintain a log or observation check list for each task. Create a problem list to capture anything that is not covered by the check list. Notate problems and jot down any hypotheses that occur to you about the problems.

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-22 Perform the Test Test Phases –Post-Test Debrief the participant. –post-test questionnaire –verbal interview Thank the participant and provide compensation. Collect, summarize, and organize test data. Reset the room for the next participant.

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-23 Process the Data Activities Performed on the Day of the Test –Collecting Data –Summarizing Data –Organizing the Material Follow-Up Activities –Categorizing –Analyzing Quantitative Data Qualitative Data

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-24 Process the Data Documenting –Identify problems Severity Frequency Errors of omission Errors of commission –Prioritize problems –Theorize reasons –Theorize solutions –Identify successes –Identify areas of uncertainty

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Some test types in more detail Heuristic evaluation Talk aloud protocol Cognitive walkthrough (Wizard of Oz) The formal usability test [next lecture] Questionnaire/survey [next lecture] 1-25

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 26 Heuristic Evaluation Proposed by Nielsen and Molich. usability criteria (heuristics) are identified design examined by experts to see if these are violated

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 27 Heuristic Evaluation Rank by severity –0=no usability problem –1=cosmetic – fix if have extra time –2=minor – fixing is low priority –3=major – important to fix –4=usability catastrophe – imperative to fix Heuristics such as 10 from Nielsen –Visibility of system status –Match between system and real world –User control and freedom, etc. Heuristic evaluation `debugs' design

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 28 Nielsen’s 10 –Visibility of system status –Match between system and real world –User control and freedom –Consistency and standards –Error prevention –Recognition rather than recall –Flexibility and efficiency of use –Aesthetic and minimalist design –Help users recognize, diagnose and recover from errors –Help and documentation

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 29 When to use Heuristic Evaluation Particular advantage that it can be used very early –From first sketches and outline descriptions –May head off a mistake rather than having to fix it Called a ‘discount usability’ method, because it’s relatively cheap (doesn’t require a lot of time and effort)

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Talk aloud protocol (aka ‘protocol analysis’) Have a user try the system and speak out loud about what they’re doing –Not entirely natural (except for really annoying people!), but tends to work OK Note misconceptions –Is what the user says they’re doing, not actually what they’re doing Note vocabulary –What is the user calling objects and actions; should screen instructions be adjusted to align? Note problems –Most important to note when they go astray (fail to achieve their goal and/or get an error message) 1-30

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Talk aloud (contd.) Note aspirations –The user might spontaneously say what they wish they could do? Is that feature there, but just not obvious to the user? Is it a feature that could reasonably be added Variations for two –Have two users sit side-by-side at the computer and perform the task together –Then it becomes much more natural for them to ‘talk aloud’ –Potentially get extra rich insights if the users disagree about how to do something –More user input per unit time for the testers 1-31

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 32 Cognitive Walkthrough (aka Wizard of Oz) Proposed by Polson et al –evaluates design on how well it supports user in learning task –usually performed by expert in cognitive psychology –expert ‘walks through’ design to identify potential problems using psychological principles Based on the idea of a code walkthrough in conventional code testing –forms used to guide analysis –can be used to compare alternatives (A bit like a formalised version of the Talk Aloud Protocol, but generally an expert rather than a typical user is ‘driving’)

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 33 Cognitive Walkthrough (ctd) For each task walkthrough considers –what impact will interaction have on user? –what cognitive processes are required? –what learning problems may occur? Analysis focuses on goals and knowledge: does the design lead the user to generate the correct goals?

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 34 Pen-based interface for LIDS UA1: Press look up button SD1: Scroll viewpoint up UA2: Press steering wheel to drive forwards SD2: Move viewpoint forwards UA3: Press look down button SD3: Scroll viewpoint down UA = User Action SD = System Display

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 35 Pen interface walkthrough UA 1: Press look up button 1.Is the effect of the action the same as the user’s goal at this point? Up button scrolls viewpoint upwards. 2.Will users see that the action is available? The up button is visible in the UI panel. 3.Once users have found the correct action, will they know it is the one they need? There is a lever with up/down looking symbols as well as the shape above and below the word look. The user will probably select the right action. 4.After the action is taken, will users understand the feedback they get? The scrolled viewpoint mimics the effect of looking up inside the game environment.

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 36 Cognitive walkthrough results Fill out a form –Track time/date of walkthrough, who the evaluators were –For each Action, answer the four pro forma questions (as per previous slide) –Any negative answer to any question should be documented on a separate Problem Sheet, indicating how severe the evaluators think the problem is, and whether they think it’ll occur often

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 37 When to do a Cognitive Walkthrough Can be done at any stage in the development process once you have a ‘prototype’ or actual system implementation to work on –Can be done with paper prototype –Can be done with a shrink-wrapped product Focus on key tasks –Things that are done by most users –“Critical success factors” of the system –Consider something that matches the name of the product If it’s an client, do a cognitive walkthrough of the task of writing and sending an