Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley The Resonant Interface HCI Foundations for Interaction Design First Edition by Steven Heim Chapter 8: Usability Testing

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-2 Chapter 8 Usability Testing What is Usability? What is Usability Testing? Design the Test Prepare for the Test Perform the Test Process the Data

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-3 What is Usability? Usability is the measure of the quality of a user’s experience when interacting with a product or system. (Usability.gov, 2006) Usability is a quality attribute that assesses how easy user interfaces are to use. (Nielsen, 2003) The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. (ISO standard , 1998)

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-4 What is Usability? Ease of learning—How fast can a user learn to accomplish basic tasks? Efficiency of use—How fast can an experienced user accomplish tasks? Memorability—Can a user remember enough to use it effectively the next time? Error frequency and severity—How often do users make errors, how serious are these errors, and how do users recover from these errors? Subjective satisfaction—How much does the user like using the system?

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-5 What is Usability? Quality of experience –Excellence of user experience –Nielsen’s ease of use Error Frequency/User Satisfaction –Quantitative analysis –Qualitative analysis

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-6 What is Usability Testing? A usability test is a structured process used to explore the interaction between an objective participant and a proposed design Paper and pencil (paper prototype) –Heuristic evaluation –Cognitive walkthrough Usability Lab (functional prototype) –Formal testing with subjects

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-7 What is Usability Testing? Attributes common to most usability tests: –The goal is to improve a product. –Participants are real users. –The participants do real tasks. –Participants are formally observed. –The data are analyzed. –Recommendations for improvement are made. Dumas and Redish (1999)

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-8 What is Usability Testing? A usability test has three basic components: –Participants—Actual users who are asked to perform realistic and representative tasks using a proposed design –Design—May be a fully functioning prototype or a simple paper prototype –Tester—There might be only one tester or there might be a testing team

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-9 What is Usability Testing? Constraints on Usability Testing –Time Design Prepare Administer (an hour to an hour and a half) Analyze the results –Finance Equipment and software Laboratory time Recording media Participant compensation Refreshments

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-10 What is Usability Testing? Constraints on Usability Testing –Personnel—Formal usability tests require at least four people. –Laboratory—To perform a formal usability test, a dedicated laboratory is required.

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-11 What is Usability Testing? Human Subjects Protocols –You must be fully aware of the regulations imposed by the various institutions and regulatory bodies that pertain to your experimental design –The U.S. Department of Health and Human Services Web site

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-12 What is Usability Testing? Advantages –Minimize help desk calls –Increase product loyalty –Provide benchmarks for future products Limitations –Artificial context –Not definitive of product acceptance –Skewed sample of users –Not always efficient

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-13 What is Usability? – Phases of a Usability Test Design the Test Prepare for the Test Perform the Test Process the Data

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-14 Design the Test Why: Define the Purpose What: Define Your Concerns and Goals What: Define the Tasks What: Create the Scenarios What: Define the Measurements How: Define the Test Method

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Why: Purpose Statement What do you expect to gain from the test? What effect will the test results have on the project? How much of the design can be changed? Are there differences pf opinions between the team? 1-15

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Why: Purpose Statement 1-16 E.g., We want to determine how easy Powerpoint 2007 is to learn for 2003 users. –What do you expect to gain from the test? We expect to gain guidelines for improving the learnability of PPT 2007 –What effect will the test results have on the project? Reorganize the layout of the interface. –How much of the design can be changed? Offer new layout templates. –Are there differences of opinions between the team ? Most team members think that the layout should change to look more like Others think the menu should be renamed/reorganized.

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley What: Concerns and Goals What do we need to improve? –Look back at results of cognitive walkthroughs, heuristic evaluations, eval of paper prototype etc. Concern: –The edit controls have been relocated Goal: –Users will be able to paste with 2 clicks 1-17

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley What: Tasks What task should the user perform? – Task 1: Select a word, copy it, paste it next to the previous word NO – this is artificial. Usability tests involve real tasks. –Better: Task 1: Reorder bullet points of a slide to match a flowchart 1-18

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley What: Scenarios –A paragraph that tells a short story setting up the task. The user will read this. It should be: Short Complete Unambiguous Familiar 1-19

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley What: Define Measurements How do we measure usability of Powerpoint? –Quantitative Time taken Errors (?) –Number of incorrect click –Qualitative interviews, questionnaires 1-20

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-21 Design the Test Where: Determine the Location of the Tests

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Design the Test Diagnosis –Exploratory –Assessment Comparison –Compare to a competing interface approach Validation 1-22

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Why is Usability Testing Important? oj/judy/research/index.htm#importhttp:// oj/judy/research/index.htm#import 1-23

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Exercise: Design a study to evaluate the usability of the visual studio IDE Step 1: Write a purpose statement (10 min) –What do you expect to gain from the test? –What effect will the test results have on the project? –How much of the design can be changed? –Are there differences of opinions between the team? 1-24

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Exercise: Design a study to evaluate the usability of the visual studio IDE Step 2: Define concerns and goals (10 min) –What do we need to improve? Identify 4 concerns Define 4 corresponding goals for the user (hypotheses) 1-25

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Exercise: Design a study to evaluate the usability of the visual studio IDE Step 3: Define tasks that will test each of your hypotheses (10 min) –What will you have participants do? 1-26

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Exercise: Design a study to evaluate the usability of the visual studio IDE Step 4: Define scenarios (10 min) –A paragraph that tells a short story setting up the task. The user will read this. It should be: Short Complete Unambiguous Familiar 1-27

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Exercise: Design a study to evaluate the usability of the visual studio IDE Step 5: Define Measurements (5 min) –List ways (quantitative and qualitative) ways that we will assess task performance (e.g., errors and time). –The results of these measurements will tell us whether our hypotheses can be accepted 1-28

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-29 Design the Test Who: Select Participants, Testers, and Observers

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-30 Design the Test Participants should be real users You do not always need to test a great many users The people you recruit should have the following basic characteristics: –Availability –Responsiveness –Objectivity

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-31 Design the Test Tester roles include the following: –Administrator –Moderator –Data logger –Technician –Prototype expert Potential observers include the following: –Other design team members not involved in the test –Clients –Programmers responsible for the final product

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-32 Prepare for the Test When: Create a Test Schedule –Project level –Test preparation level –Test execution level –Task execution level

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-33 Prepare for the Test Writing Scripts –Greeting the Participant –Preliminary Interview –Providing Instructions –Monitoring the Test –Debriefing the Participant Running a Pilot Test –Be organized –Be presentable

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-34 Perform the Test Test Phases –Pre-Test Greet the participant. Have the participant sign the informed consent form. Have the participant fill out any pre-test questionnaire. Proceed with scripts. –During the Test Maintain a log or observation check list for each task. Create a problem list to capture anything that is not covered by the check list. Notate problems and jot down any hypotheses that occur to you about the problems.

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-35 Perform the Test Test Phases –Post-Test Debrief the participant. –post-test questionnaire –verbal interview Thank the participant and provide compensation. Collect, summarize, and organize test data. Reset the room for the next participant.

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-36 Process the Data Activities Performed on the Day of the Test –Collecting Data –Summarizing Data –Organizing the Material Follow-Up Activities –Categorizing –Analyzing Quantitative Data Qualitative Data

Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1-37 Process the Data Documenting –Identify problems Severity Frequency Errors of omission Errors of commission –Prioritize problems –Theorize reasons –Theorize solutions –Identify successes –Identify areas of uncertainty