Running a User Study Alfred Kobsa University of California, Irvine

Pilot Test A pilot test is extremely valuable for detecting problems in the planned user test. These problems may be harmless, but they may also render test results unusable. 2-5 pilot subjects are sufficient. Some of them should be people who are very much willing to criticize the experiment (e.g., good friends, colleagues). At least one of them should be a real test subject. Pilot test should be run two days before the actual study at the latest (and there are many good reasons to do them far earlier). Pilot test should include all procedures.

Problems that are frequently uncovered in pilot tests Subjects take considerably longer/shorter than expected. Task instructions are not well understood by subjects Subjects cannot carry out a task since some necessary functionality has not been explained to them Subjects use procedures for solving a task that are different from the one that the experimenters wanted to test Subjects overwrite each others’ data files Experimenters overwrite data of previous experiments Software crashes Experimenters forget to do certain things Some materials are missing Users have troubles finding parking, the building entrance, the usability lab, etc.

Welcome, Briefing, Instruction and Training Welcome -Make participants feel comfortable / reduce anxiety -Bridge time until everyone has arrived -Let them show their ID (“for security/tax purposes”), and announce this beforehand -Ask them to switch their cell phones off. Briefing -Inform about purpose of the experiment (as far as this is possible) -Emphasize that it will help develop a better product -Encourage criticism and praise -Emphasize that the product is being tested and not the user -Emphasize that the people they will see are not the developers -Show them the lab (or movies or pictures of it) -Have them sign legal forms -Let them fill in a pre-questionnaire (e.g., demographics, pre-test) Instruction and Training -Instruction through video, or through instructor who follows a written script -Subsequent training tasks allow users to practice what they learned and to understand it better. -Guided training tasks: task description contains (partial) instructions on how to carry out a task, which subjects are asked to follow

Conducting the Test Setup -(Bring subjects to test computers) -Verify initial settings and materials, start recording devices. -Ask them to begin (and to let you know when they are done) -Enter users in logbook/timesheet (e.g., who used which computer) During the test -Inform users about timeouts -Answer questions generically; avoid biasing participants -If applicable, remind them to think aloud if they become silent -Watch test users and take notes, or monitor users from a distance -Record unusual occurrences in logbook/timesheet -If applicable: remind users occasionally to think aloud -If applicable: make sure users don’t influence each other After the test -Post-questionnaire (e.g., opinions on software, test procedures; post-test) -Payment, escort -Prepare test station for new test subjects

At any time Be organized! Follow checklists!

Unexpected events A participant does not arrive in time A participant forgot(?) his or her ID A participant refuses to sign the informed consent form or the non-disclosure agreement A participant refuses to be videotaped A participant does not want to switch off his/her cell phone A participant is called away in the middle of the test A participant’s cell phone rings continuously A participant does not have the required qualifications A participant becomes confrontational with other participants or with the experimenters A participant takes over the group Software freezes, computer breaks down, etc. Outside interference (construction noise, vacuums, …)