Preparing a User Test Alfred Kobsa University of California, Irvine

Specifying Global Test Goals Specify (Intended) purpose of product Product development status Test goals (with priorities) User profiles (personae)

Selecting Tasks Test subjects should not merely “try out the system for n minutes”, but rather carry out selected tasks with the system. One cannot test every possible user task. Rather, usability tests must focus on representative and/or important tasks. Tasks should be selected that may be fraught with usability problems, as suggested from earlier concerns and usage experience; will be frequently carried out by users (20% is used 80% of time) are mission-critical; are performed under time pressure; or are new or have been modified in comparison with previous version or competitive program ☞ Brainstorm and then filter tasks using these criteria Test the comprehensibility of task descriptions Specify timeout for each task (possibly do not reveal to subjects)

Creating Scenarios Scenarios are created to contextualize user experiments (which in general yields more representative test results) Scenario descriptions should be - short - formulated in the words of the user / task domain - unambiguous - contain enough information for test subjects to carry out tasks - be directly linked to tasks and concerns ☛ User should read scenario descriptions (and experimenters should possibly read them aloud at the same time) ☛ Scenario descriptions should be tested (in the pilot test or even earlier) Examples:

Deciding how to measure usability Performance measures -time needed to carry out a task -Error rate -Task completion rate -Time spent on “unproductive” activities (navigation, looking up help, recovery after an error) -Frequency of “unproductive” activities -Counting keystrokes / mouse clicks -Etc. (see Dumas & Reddish) Measures of satisfaction User-provided: Observed(?): frustration / confusion / surprise / satisfaction -User ratings (e.g., SUS – System Usability Scale) -Comparisons with previous version / competitors’ software / current way of doing it -Behavioral intentions (use, buy(?), recommend to friend) -Free comments (after and during experiment)

Preparing the Test Materials Legal documents Informed consent form Non-disclosure form Waiver of liability form Permissions form (e.g., for video recordings) Instruction and training related materials Software / Powerpoint slides / video to be shown Summary of software functionality Write-up of oral instruction (Guided) training tasks Task-related materials Scenario description Task descriptions ( ☛ one task on each page, large font for task/page number) Pre-test and post-test questionnaires Experiment-related materials “Screener” with participant election criteria Experimental time sheet / log book To-do list for all experimenters

Preparing the testing environment Hardware equipment ☛ cater to users’ normal equipment; remove all potentially distracting programs Sample data ☛ make it look real Voice recording ☛ take care of ambient noise (also exceptional), direction of microphone,… Screen recording ☛ (mind a possible slowdown of tested program) Video recording ☛ take care of video angle, blocked view, glare, different sunlight over the day,… Time taking ☛ avoid races (between participants, or participants against stop watch) Lab layout (see Courage & Baxter) ☛ participants should not influence each other

Setting up a test team Typical roles Greeter: welcomes subjects, makes them relaxed, bridges wait time Briefer: informs about study, makes them sign forms Instructor / trainer: instructs them about the software Test administrator: tells subjects what to do Note taker Video operator Backup technician for emergencies Many of these roles can be combined in a single person No role-switching over the duration of an experiment, to ensure comparability The number of team members also depends on the number of parallel / overlapping subjects and the experimental design Teams of 2-3 are typical Tests are typically carried out by UI design team, and/or outside usability specialists Developers, managers, user representatives should be able to watch (invisible to the subjects, or in the background)