Steps to Conduct a Usability Test Dr. Jennifer L. Bowie

What is Usability? “A function of particular users performing particular tasks in a particular environment” (Smith et al. 68) The “people who use the product can do so quickly and easily to accomplish their own tasks” (Dumas and Redish 4) User-centered design, not “user-friendly”

What is Usability Testing? An empirical study of a product’s usability by observing actual users do real tasks with the product Involves: Real users Real tasks Specific usability goals/concerns Observing and recording the testing Data analysis

Step 1: User Analysis & Profiles Who are your actual users? You may need to break your users into typical user categories. Consider: Demographics: age, sex, race, education level, cultural background, socioeconomic status,… Experience level with the product, with products of the same genre, with required technology,... Other things: motivation learning style subject matter knowledge location of use physical characteristics people with disabilities or impairments (from color blindness and learning disabilities to more severe disabilities)

Step 1: User Analysis & Profiles Con. Create user profiles: Break users into clear subgroups Profile/Define the characteristics of each subgroup Choose user profiles to test: Ideally users from all major profiles will be tested If limited testing: Choose profiles based on highest number of users in that profile or profiles that you think may have the greatest usability issues

Step 2: Decide what to Test 1. Choose an overall purpose Example: How useable is our new website? 2. Determine objectives or what you are testing for. Examples: Does our search engine provide usable results in the first 5 links returned? Are search results clear to the users? 3. Choose type of test: 1. Performance: Can they do it? 2. Understandability: Can they understand it? 3. Read-and-locate: Can they find it?

Step 2: Decide what to Test con. 4. Select tasks: Consider tasks with a high chance of user failure (complex tasks, one-of-a-kind tasks, highly abstract or technical tasks) Consider tasks with a high cost of user failure (tasks that require support, like help or support calls, to complete; tasks where data could be damaged or lost ) Consider: First impressions (look and feel) First tasks Tasks most performed Critical tasks Specific problem areas New task for the product 5. Select performance objectives (should be individualized for each task) Time: How long to complete tasks, to find things, to performance procedures Error/Success: user errors, attempts to do/find something, numbers of times section re-read, if the task was completed successfully

Step 3: Preparing for the Testing 1. Choose order of tasks: start easy 2. Create written test materials: Task list for users Written welcome speech/ Intro to be read to user Consent forms Observation forms Pre-task and post task questionnaires & interview questions Other materials 3. Recruit participants & determine “payment” 4. Define team member’s roles: Facilitator/Briefer (necessary): Often only team member to interact with users Observation recorder/note taker (necessary) Camera operator (optional) Help desk operator (optional) Test administer (optional) 5. Create written test plan 6. Practice: conduct walkthroughs of the testing and if possible pilot test (the pilot test users could even be a team member) 7. Prepare test environment (day of test)

Step 4: Conducting the test 1. Greet & Brief participant: a. Read/say welcome b. Emphasize that you are not testing them, but the product and that they should act as natural as possible c. Explain think-aloud protocol (if using) d. Emphasize how user tells you she has completed a task e. Stress that the testing is “anonymous” 2. Be unbiased (especially the Facilitator/Briefer) 3. Intervene carefully (avoid as much as possible) 4. Observe and record data 5. Debrief user

Step 5: Analyzing the Data 1. Collate data into findings: a. Choose an approach: Top-down approach: predetermine categories of findings (like navigation, design, terminology) and go through data looking for “hits” Bottom-up approach: put each observation on a sticky note/note card, sort into categories and label categories b. Determine time and errors/success Examine findings for each user, user profile, and task Use analysis techniques such as statistics (even averages help)

Step 5: Analyzing the Data con. 1. Analyze data: a. Determine cause of problems b. Determine scope/severity of problems c. Make recommendations/changes 2. Report Findings

Good Luck & Have Fun! Where to find out more: Barnum Usability Testing and Research Barker Writing Software Documentation, Chapter 6 “Conducting Usability Tests” Hom “General Concepts of Usability Testing”