Running a User Study Alfred Kobsa University of California, Irvine

Pilot Test A pilot test is extremely valuable for detecting problems in the planned user test. These problems may be harmless, but they may also obstruct the user test or render its results unusable. 2-5 pilot subjects are sufficient. Some of them should be people who are very much willing to criticize the experiment (e.g., good friends, colleagues). At least one of them should be a real test subject. Pilot test should be run two days before the actual study at the latest (and there are many good reasons to do them far earlier). Pilot test should include all procedures.

Problems that are frequently uncovered in pilot tests Subjects take considerably longer/shorter than expected. Task instructions are not well understood by subjects Subjects cannot carry out a task since some necessary functionality has not been explained to them Subjects use procedures for solving a task that are different from the one that the experimenters wanted to test Subjects overwrite each others’data files Experimenters overwrite data of previous experiments Software crashes Experimenters forget to do certain things Some materials are missing Users have troubles finding parking, the building entrance, the usability lab, etc.

Welcome, Briefing, Instruction and Training Welcome -Make participants feel comfortable / reduce anxiety -Bridge time until everyone has arrived -Let them show their ID (“for security/tax purposes”), and announce this beforehand -Ask them to switch off their cell phones. Briefing -Inform about purpose of the experiment (as far as this is possible) -Emphasize that it will help develop a better product -Encourage criticism and praise -Emphasize that the product is being tested and not they themselves -Emphasize that the people they will see are not the developers -Show them the lab (or movies or pictures of it) -Have them sign legal forms -Let them fill in a pre-questionnaire (e.g., demographics, pre-test) Instruction and Training -Instruction through video, Powerpoint, or instructor who follows a written script -Subsequent training tasks allow users to practice what they learned and to understand it better. -Guided training tasks: task description contains (partial) instructions on how to carry out a task, which subjects are asked to follow

Conducting the Test Setup -(Bring subjects to test computers) -Verify initial settings and materials, start recording devices. -(Instruct them to “think aloud” [this makes task completion times unreliable]) -Ask them to begin (and to let you know when they are done) -Enter users in logbook/timesheet (e.g., who used which computer) During the test -Watch test users and take notes, or monitor users from a distance -Record unusual occurrences in logbook/timesheet -Answer questions generically; avoid biasing participants -Inform users about timeouts -If applicable, remind them to think aloud if they become silent -If applicable: make sure that users don’t influence each other After the test -Post-questionnaire (e.g., opinions on software and on test procedures; post-test) -Payment, call escort -Prepare test station for new test subjects

At any time Be organized! Follow checklists!

Unexpected events A participant… does not arrive on time says he forgot his ID refuses to sign the informed consent or non-disclosure form refuses to be videotaped does not want to switch off his/her cell phone is called away in the middle of the test has a cell phone on that rings continuously does not have the required qualifications exceeds the time limit for a task without finishing it struggles mightily with a task becomes confrontational w/ other subjects or the experimenters takes over the group Software freezes, computer breaks down, etc. Outside interference (construction noise, vacuums, …)