Preparing and Running User Experiments By Mei Li, Pearl Ho, and Deepika Gandhi

How to Prepare and Run Usability Testing How to gather participants and prepare a proposal. Preparing the environment, test materials, and test team. How to measure usability and deal with participants.

Preparing Materials, Environment, and Test Team for Usability Tests Presenter: Pearl Ho From Dumas and Redding Chapters 11, 12, 14, 15, 16

Overview I. Selecting and organizing tasks to test II. Creating task scenarios III. Preparing test materials IV. Preparing the test environment V. Preparing the test team

I. Selecting and Organizing Tasks to Test How do you select which tasks to test? What resources do you need for each task?

Selecting Tasks Tasks that probe potential usability problems Use tasks suggested from concerns and experience Tasks that users will do with the product

Selecting Tasks Tasks that probe potential usability problems The more problems you find, the more successful your test will be Tasks suggested from concerns and experience Tasks that users will do with the product

Selecting Tasks Tasks that probe potential usability problems Tasks suggested from concerns and experience People who develop the product will have some sense of what will be difficult to use based on which parts were difficult to design or which parts where they disagreed about the best approach Tasks that users will do with the product

Selecting Tasks Tasks that probe potential usability problems Tasks suggested from concerns and experience Tasks that users will do with the product There are other tasks that users can do with a product over and above those that relate to usability problems, concerns, or goals. For example: New or modified tasks Tasks that are critical to the operation of the product Frequently done tasks Things done under pressure

Determining the Resources You Need for Each Task Estimating task times Listing the resources you need for tasks

Determining the Resources You Need for Each Task Estimating task times Time it will take to run the task during the test. Time that users will feel is acceptable for completing a task. Consider characteristics of participants who will be in the test Time for experienced person. This forms baseline Problems for typical participant and time to recover from them Time to set up tasks and time between tasks Estimate will probably be a range, not a single value Listing the resources you need for tasks

Determining the Resources You Need for Each Task Estimating task times Listing the resources you need for tasks Hardware – participants needs to do the task AND what you need to set up and run the task Software and data files – again for both participants and you Instructions and procedures you will need

Determining the Resources You Need for Each Task In addition to task times, for most tests, it takes about an hour to Start the test Give the prebriefing Give the participant a break halfway through the test Have the participant fill out the posttest questionnaire Give the debriefing

Creating the Final List of Resources Here’s an example of what a list of resources might look like for testing the task of sending a reply on an program Time3-12 minutes HardwareTwo PCs connected to network Software and data files software Mail account for participants Incoming mail message to reply to Existing account to send reply to Instructions and proceduresTest of reply (or info to create text) Procedure for restoring account to initial condition

II. Creating Task Scenarios What is a scenario? What makes a good scenario? Do you give participants written scenarios? How do you divide up tasks and scenarios? How do you make participants stop between tasks for timing purposes?

What is a scenario? You use scenarios to tell participants what you want them to do during the test. Scenarios describe the tasks in a way that takes some of the artificiality out of the test. It makes the task more realistic.

What makes a good scenario? Short User’s words, not product’s words Unambiguous Enough information to do the task Directly linked to your tasks and concerns

Do you always give participants written scenarios? Not always. You may: Have test team members pretend to be customers, supervisors, or colleagues and walk into the test room to deliver the scenarios in person Use the product to deliver the scenarios Mix modes, give them writing, but then add visits or calls

How do you divide up the tasks and scenarios for participants? It depends… In deciding how to match up scenarios and tasks, the major issue you have to consider is whether you want to separate measurements of time, errors, or other codes for particular tasks.

How do you make participants stop between tasks? If you want to time tasks separately, you need to get participants to stop between each task. You can give them one task at a time. Or you can just say after each task: “Please tell us when you have finished this task.” or “Please wait for us to tell you to turn the page.”

III. Preparing Test Materials The Legal Form - minimal risk, informed consent Questionnaires Participant Training

The Legal Form As a tester it is your responsibility to: Create a form that correctly states each party’s rights Ensure that test participants have read and understood what the form says Observe or witness participants signing the form

Concept of minimal risk Minimal risk means that “the probability and magnitude of harm or discomfort anticipated in the test are not greater, in and of themselves, than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.” If, in your judgment, any usability test could put participants at risk, you have three options: Eliminate the risk Follow the federal policies for such tests Don’t do the test

Informed consent Key to protecting both parties rights Participants should be given the opportunity to choose what will or will not happen to them

Three elements of any informed consent: Information Comprehension Voluntariness

Three elements of any informed consent: Information: Procedures you will follow The purpose of the test Any risks to the participant Provide the opportunity to ask questions Provide the opportunity to withdraw at any time Comprehension Voluntariness

Three elements of any informed consent: Information Comprehension You must convey the information clearly and completely. Do not rush through consent procedure, make it seem unimportant, or not allow questions. Voluntariness

Three elements of any informed consent: Information Comprehension Voluntariness Participants must be free from coercion and undue influence If participants are considering withdrawing from a test, remain calm and neutral in your manner so that you do not unduly influence them

Other Major legal issues: Nondisclosure, if product is still under development Waivers, permission to use tapes and questionnaires Participant’s rights, ex, right to be informed

Questionnaires Pretest questionnaire - gather info about the background of participant Posttask questionnaire - gather judgments and rating after each task Posttest questionnaire - gather judgments after completion of all tasks

Training Participants Reasons to train To ensure that all participants have the same level of skill or knowledge before they begin the tasks To provide some groups of participants with training that others do not get

Training Participants To provide effective training, you need to decide: The purpose of the training What skills and knowledge you want participants to learn How you will conduct the training What criteria of success you will use to measure the effectiveness of training

IV. Preparing the Test Environment Preparing the Product Data-logging material preparation

Preparing the Product Create the sample data you need. Plan for disaster. Save time and frustration. Preparing for manuals. Legible type and graphics Numbered pages High quality, professional looking binding or binder Put in table of contents, cross-references, and an index

Preparing Data-Logging Materials or Software The key to having quantitative measures of performance is recording the duration and frequency of events during the test. You can buy data-logging software or at least have stopwatch and sheet of paper that lists the task numbers. If you decide to videotape the test, keep in mind that it takes twice as long to run test and then watch the tape.

V. Preparing the Test Team How many people do you need to run a test? Should people switch roles during the test? Who should be on the test team What roles are there for test team members

How many people do you need to run a test? Three people make a good team. Some have five, including a technical person. Two can make up team if both are skilled usability specialists and participants don’t have to move around much.

Should people switch roles during the test? No. You don’t want differences in data based on differences in test team.

Who should be on the test team? Everyone on test team should be on the planning team, but not everyone planning has to be on test team. Product developers or usability specialists? Realize obvious tradeoffs and choose one that makes sense for you. Other people, such as managers, developers, writers, can participate in a test as observers.

What roles are there for test team members? Test administrator Briefer Camera operator Data operator Help desk operator Product expert Narrator

See the text book if you want specific tasks and a checklist for each role

QUESTIONS? On to Deepika…. During your User Requirements Activity