Running User Experiments Humeyra Topcu-Altintas David Girsault Feyi Agagu

Outline Recruiting Participants Recruiting Participants Selecting and Organizing Tasks to Test Selecting and Organizing Tasks to Test Creating Task Scenarios Creating Task Scenarios Deciding How to Measure Usability Deciding How to Measure Usability Prepare Test Materials Prepare Test Materials Prepare Testing Environment Prepare Testing Environment Prepare Test Team Prepare Test Team

Recruiting Participants

Finding appropriate participants Problems about wrong participant Problems about wrong participant More experienced More experienced Less experienced Less experienced Where should we go? Where should we go? The source, ex: new version of existing os -> customer list The source, ex: new version of existing os -> customer list bank teller system -> banks Should we use company employees? Should we use company employees? Tighter security Tighter security Lower costs Lower costs Recruiting is easier Recruiting is easier But, they may not represent the actual users But, they may not represent the actual users

Methods to find a participant Working with temporary agencies Working with temporary agencies Advertising Advertising Networking Networking Working with professional associations Working with professional associations

Building a database of qualified participants For people who might be appropriate for the types of usability test For people who might be appropriate for the types of usability test Select appropriate people from db to recruit for each test Select appropriate people from db to recruit for each test

Reaching and screening potential participants Sending out letters and questionnaires Sending out letters and questionnaires Sending initial letter first, questionnaires to people who express interest Sending initial letter first, questionnaires to people who express interest By telephone By telephone

Introductory letter (to get interest): Type of product Type of product The environment The environment How long the test will last How long the test will last How much we will pay How much we will pay Screening Questionnaire: Ask only what we need to know Ask only what we need to know Ask specific questions Ask specific questions Ask questions in an order that allows the recruiter to stop Ask questions in an order that allows the recruiter to stop

Deciding who should recruit participants Recruiter should be someone who Recruiter should be someone who understands usability testing understands usability testing is enthusiastic about the test is enthusiastic about the test

Planning how many calls it will take to find a participant More specialized skills -> More calls More specialized skills -> More calls More stringent criteria -> More calls More stringent criteria -> More calls

Knowing when to recruit and schedule participants A few weeks before the test. Because: A few weeks before the test. Because: people may not know what their work schedule will be people may not know what their work schedule will be they may forget they may forget

Knowing what to say when you are recruiting If it is the first contact, start the recruiting with the subjects of the introductory letter If it is the first contact, start the recruiting with the subjects of the introductory letter If the questionnaire has been already filled out, verify the information If the questionnaire has been already filled out, verify the information If not, find out quickly if the person appropriate or not If not, find out quickly if the person appropriate or not Don’t give detailed answers Don’t give detailed answers Don’t assume people show up w/o reminders Don’t assume people show up w/o reminders

Arranging for payment or other incentives If working with a temporary agency If working with a temporary agency It will charge for each participant It will charge for each participant Plus %33 (in general) for themselves Plus %33 (in general) for themselves How much should we pay to the participants? How much should we pay to the participants? The one that motivates people to come The one that motivates people to come Not have to be money, maybe gift certificates, travel to the test side, samples of the product, samples of other products etc. Not have to be money, maybe gift certificates, travel to the test side, samples of the product, samples of other products etc.

Making sure that participants show up 10% no show is common 10% no show is common What can we do about the people who forget What can we do about the people who forget Reschedule them Reschedule them What can we do to make sure participants come? What can we do to make sure participants come? To make the usability test sound interesting, nonthreatening To make the usability test sound interesting, nonthreatening To confirm the appointment and a day or two before the schedule time To confirm the appointment and a day or two before the schedule time

Selecting and Organizing Tasks to Test

Selecting tasks Not possible to test every task users can do with a product Not possible to test every task users can do with a product What tasks should we sample? What tasks should we sample? Tasks that probe potential usability problems Tasks that probe potential usability problems Tasks suggested from our concerns Tasks suggested from our concerns Tasks derived from other criteria Tasks derived from other criteria Tasks that users will do with the product Tasks that users will do with the product

Set a password for the mail account Set a password for the mail account Create a personal mailing list Create a personal mailing list Move messages between folders Move messages between folders Sorts items in folder Sorts items in folder Read any new mail message Read any new mail message Read a selected mail message Read a selected mail message Create and send a mail message Create and send a mail message Edit a message Edit a message File a mail message File a mail message Create a folder Create a folder Forward a message Forward a message Delete a mail message Delete a mail message Delete a folder Delete a folder Find a mail message Find a mail message Send a CC of a message Send a CC of a message Reply to a mail message Reply to a mail message Attach a file to a message Attach a file to a message Create a distribution list Create a distribution list

Determining the resources you need for each task How long it will take to do the task How long it will take to do the task What hardware, software, procedures, and other resources will we need to run What hardware, software, procedures, and other resources will we need to run

Creating the final list of tasks to test After completing task list with their requirements, eliminate some. After completing task list with their requirements, eliminate some. While eliminating, we should consider While eliminating, we should consider The objective of the task, whether we can achieve more than one obj. with it The objective of the task, whether we can achieve more than one obj. with it Whether a task uses expensive resources is worth testing Whether a task uses expensive resources is worth testing Whether time consuming tasks are more important than two or three tasks Whether time consuming tasks are more important than two or three tasks

Creating Task Scenarios

List of tasks for test  presentation to the participants List of tasks for test  presentation to the participants Scenario = embedding tasks in a short story Scenario = embedding tasks in a short story How to create good scenarios? How to create good scenarios?

What is a Scenario? Best defined by example: Best defined by example: “You’ve just bought a new combination telephone and answering machine. The box is on the table. Take the product out of the box and set it up so that you can make and receive calls.” Makes tasks more realistic Makes tasks more realistic Gives goal, not steps Gives goal, not steps

What Makes a Good Scenario? Short: no time waste on reading Short: no time waste on reading In the user’s words: easy to understand In the user’s words: easy to understand Unambiguous: task must be clear enough Unambiguous: task must be clear enough Enough information: same data as in a real situation Enough information: same data as in a real situation Directly linked to tasks: scenario tests a set of tasks Directly linked to tasks: scenario tests a set of tasks

Do you Always Give Participants Written Scenarios? Written scenarios  most of the time Written scenarios  most of the time RPG style scenarios  useful in some cases (customers, supervisors, colleagues…) RPG style scenarios  useful in some cases (customers, supervisors, colleagues…) Written version always necessary Written version always necessary

How do you Divide up the Tasks and Scenarios for Participants? Most important criteria = task measurement Most important criteria = task measurement Time Time Number of errors Number of errors Etc. Etc. Separating tasks  separate measurements Separating tasks  separate measurements Better precision Better precision

How do you Make Participants Stop Between Scenarios? Just tell them! Just tell them! Only provide scenarios one by one Only provide scenarios one by one For written scenarios, ask the user to stop when finished at the end of the instructions For written scenarios, ask the user to stop when finished at the end of the instructions

Deciding how to measure usability

Deciding how to Measure Usability Goal: Planning the observations and measurements for a usability test Goal: Planning the observations and measurements for a usability test Quantitative goals: Quantitative goals: vague ideas  measurable data vague ideas  measurable data Reasons for planning Reasons for planning Focus on important points during test Focus on important points during test Quantitative measures Quantitative measures

Understanding What You Can Measure Performance measures (quantitative) Performance measures (quantitative) Ex: time to finish a task, number of wrong menu choices Ex: time to finish a task, number of wrong menu choices Counting keystrokes is difficult Counting keystrokes is difficult Subjective measures Subjective measures Quantitative or qualitative Quantitative or qualitative Ex: user rating the product, collect spontaneous comments Ex: user rating the product, collect spontaneous comments

Matching Measures to your Goals and Concerns Performance measures  quantitative goals Performance measures  quantitative goals Ex: time requirement  time measurement Ex: time requirement  time measurement Commercial software available for collecting data Commercial software available for collecting data Taking notes could be as good Taking notes could be as good Measuring positive aspects useful Measuring positive aspects useful Good aspects of the product Good aspects of the product Rewarding for product developers Rewarding for product developers

Matching Measures to the Product’s Stage of Development Performance measures adapted to stage of development Performance measures adapted to stage of development Example 1: avoid time measurements with prototypes Example 1: avoid time measurements with prototypes Example 2: avoid measuring usability for partially implemented interfaces Example 2: avoid measuring usability for partially implemented interfaces

Setting Quantitative Criteria for Each Measure and Each Task Must set quantitative criteria for usability Must set quantitative criteria for usability Focus on users Focus on users Use previous data to set goals Use previous data to set goals Common sense Common sense 3 categories of results: excellent, acceptable, unacceptable  value ranges 3 categories of results: excellent, acceptable, unacceptable  value ranges Possible criteria modification Possible criteria modification

Setting Quantitative Criteria (continued) Different criteria for each task Different criteria for each task Measures can change with tasks Measures can change with tasks Small amount of time to get used to the environment Small amount of time to get used to the environment System response time  focus on users System response time  focus on users

Preparing Test Materials

The Concept of minimal Risk Minimal risk is defined as the probability of harm anticipated in the test are not greater than what participant encounters in everyday life. Minimal risk is defined as the probability of harm anticipated in the test are not greater than what participant encounters in everyday life. If there is risk involved If there is risk involved Eliminate the risk Eliminate the risk Follow the federal policies for such tests Follow the federal policies for such tests Don’t do the test Don’t do the test

The Legal Form Informed Consent – Information Informed Consent – Information Explain procedure you will follow Explain procedure you will follow Explain the purpose of the test Explain the purpose of the test Explain any risks Explain any risks Explain opportunity to ask questions Explain opportunity to ask questions

The Legal Form Informed consent - Comprehension Convey information clearly and completely Convey information clearly and completely Allow questions so participants can make informed decision Allow questions so participants can make informed decision If you rush through It seems unimportant If you rush through It seems unimportant

The Legal Form Informed consent - Voluntariness The participant must be free from coercion The participant must be free from coercion When participants wish to withdraw remain calm and neutral. When participants wish to withdraw remain calm and neutral.

The Legal Form When obtaining consent Develop a rapport Develop a rapport Neutral but friendly tone Neutral but friendly tone If video tapping have the camera on If video tapping have the camera on Body language must not show pressure Body language must not show pressure Be Professional Be Professional Never put participant at risk Never put participant at risk

The Legal Form Obtaining consent - Legal issues Non-disclosure: Participants can’t discuss about product. Non-disclosure: Participants can’t discuss about product. Waivers: You need participants permission to use tapes or questionnaire. Waivers: You need participants permission to use tapes or questionnaire. Participants rights: right to be informed, right to withdraw…. Participants rights: right to be informed, right to withdraw….

Questionnaires You need to anticipate question You need to anticipate question You need to write questions down You need to write questions down So you can ask same question So you can ask same question So you will not forget to ask the question So you will not forget to ask the question Put questions in specific and appropriate form Put questions in specific and appropriate form

Questionnaires There are several points where you would want to ask participants questions. They are; Pretest Pretest Post-task Post-task Post-test Post-test

Questionnaires Pretest Questionnaires Gather background information Gather background information Verify qualifications of participants Verify qualifications of participants Questions should be one page or less Questions should be one page or less How long have you worked with this product? How long have you worked with this product?

Questionnaires Post-task Questionnaires To obtain immediate reaction or judgments about each task To obtain immediate reaction or judgments about each task Measure changes in perception over time Measure changes in perception over time Did you use the manual to complete the task? Did you use the manual to complete the task?

Questionnaires Post-test Questionnaire To gather judgments and rating after test To gather judgments and rating after test To clarify any ambiguities that may have occurred To clarify any ambiguities that may have occurred General questions and specific questions General questions and specific questions What did you like least about the product? What did you like least about the product?

Training Scripts To ensure all participants have the same skill level To ensure all participants have the same skill level To provide participants with training that others do not have. To provide participants with training that others do not have.

Task Scenarios Package in such a way that it will allow u to observe and record data Package in such a way that it will allow u to observe and record data There must be a clear beginning and a clear ending. There must be a clear beginning and a clear ending.

Preparing Testing Environment

Arranging equipment and the props Arranging equipment and the props Preparing the products Preparing the products Creating sample data Creating sample data Planning for disaster Planning for disaster Preparing manuals Preparing manuals Adjusting the video and audio controls Adjusting the video and audio controls Preparing data logging materials or software Preparing data logging materials or software Labeling tapes and disks Labeling tapes and disks

Preparing Test Team

Preparing the Test Team Number of people to run test Three people should be enough! Three people should be enough! Two usability specialists Two usability specialists One member technical staff One member technical staff Some go for five but one should make sure they are not tripping over each other Some go for five but one should make sure they are not tripping over each other

Preparing the Test Team Can other people participate in the test? Observers can view in case they have any doubts about the test. Observers can view in case they have any doubts about the test. Should not switch roles during tests? No, people should not switch roles because it might make the results irregular No, people should not switch roles because it might make the results irregular

Preparing the Test Team Who should be on the test team? Testing is done by usability professional Testing is done by usability professional Testing is done by product developers Testing is done by product developers Testing is done by usability professionals and product developers Testing is done by usability professionals and product developers

Usability specialists as testers Advantages They know how to plan and conduct tests They know how to plan and conduct tests They know what to look for They know what to look for They know how to analyze data They know how to analyze data They become experts at using lab equip. They become experts at using lab equip.Disadvantages Not experts in each product Not experts in each product Not experts in how users for each product do their jobs Not experts in how users for each product do their jobs

Product developers as testers Advantages They see users using their products They see users using their products They are forced to think about usability They are forced to think about usability They already know the product They already know the product Don’t have to hire people Don’t have to hire peopleDisadvantage Developers must learn new tool Developers must learn new tool Developers are not trained observers Developers are not trained observers Testing takes time away from work Testing takes time away from work The cost may be higher The cost may be higher

Preparing the Test Team What roles are there for test team members? Test Administrator Test Administrator Briefer Briefer Camera operator Camera operator Data recorder Data recorder Help desk operator Help desk operator Product expert Product expert Narrator Narrator

Preparing the Test Team There should be checklists for each role There should be checklists for each role Test Administrator - Greet Guest observers Test Administrator - Greet Guest observers Test Administrator – Make sure someone is responsible for setting up for next session. Test Administrator – Make sure someone is responsible for setting up for next session. Briefer - Check participants name Briefer - Check participants name Briefer – Let participants see cameras Briefer – Let participants see cameras Product expert – Bring the product backup Product expert – Bring the product backup

THE END Questions??? – Science Library! Questions??? – Science Library!