Preparing a User Test Alfred Kobsa University of California, Irvine

Selecting Tasks One cannot test every possible user task. Rather, usability tests have to focus on representative and/or important tasks. Tasks should be selected that may be fraught with usability problems, as suggested from concerns and experience; will be frequently carried out by users; are mission-critical; are performed under pressure; and are new or have been modified in comparison with previous version or competitive program

Creating Scenarios Scenarios are created to contextualize user experiments (which in general yields more representative test results) Scenario descriptions should be - short - formulated in the words of the user / task domain - unambiguous - contain enough information for test subjects to carry out tasks - directly linked to tasks and concerns ☛ User should read scenario descriptions (and experimenters should possibly read them aloud at the same time) ☛ Scenario descriptions should be tested (in the pilot test or even earlier)

Deciding how to measure usability Performance measures -time needed to carry out a task -Error rate -Time spent on “unproductive” activities (navigation, looking up help, recovery after an error) -Frequency of “unproductive” activities -Counting keystrokes / mouse clicks -Etc. (see Dumais & Reddish) Measures of satisfaction User-provided: Observed: frustration / confusion / surprise / satisfaction -User ratings -Comparisons with previous version / competitors’ software / current way of doing it -Behavioral intentions -Free comments

Preparing the Test Materials Legal documents Informed consent form Non-disclosure form Waiver of liability form Permissions form (e.g., for video-recorded material) Instruction and training related materials Software / Powerpoint slides / video to be shown Description of software functionality Write-up of oral instruction (Guided) training tasks Task-related materials Scenario description Task descriptions ( ☛ put only one task on each page) Pre-test and post-test questionnaires Experiment-related materials Experimental time sheet / log book To-do list for all experimenters

Preparing the testing environment Hardware equipment ☛ cater to users’ normal equipment; remove all potentially distracting programs Sample data ☛ make it look real Voice recording ☛ take care of ambient noise (also exceptional), direction of microphone,… Screen recording ☛ mind a possible slowdown of tested program Video recording ☛ take care of video angle, blocked view, glare, different sunlight over the day,… Time taking ☛ avoid races (between participants, or participants against stop watch) Lab layout ☛ participants should not influence each other

Setting up a test team Typical roles Greeter Briefer Instructor / trainer Narrator Test administrator Note taker Video operator Backup technician for emergencies (Help desk operator) Many of these roles can be combined in a single person No role-switching during an experiment The number of team members depends on the number of parallel / overlapping subjects and the experimental design Teams of three are typical Tests are typically carried out by UI design team, and/or outside usability specialists. Developers, managers, user representatives should be able to watch