1. User Testing & Experiments

2. Objectives Explain the process of running a user testing or experiment session. Describe evaluation scripts and pilot tests and explain why they are important Explain the difference between user testing and experiments. Describe the major building blocks of an HCI experiment. Identify flaws in experimental designs and suggest better alternatives

3. Procedure for User Testing From http://www.htc.net/~joegrant/web_pres_trial_v2/sld001.htm

11. Controlling bias Now you get to work with pop-up menus. I think you will really like them... I designed them myself! How else might the experimenter bias a user test?

12. Evaluation Script A script that you read instructions from –Ensures all participants receive exactly the same instructions –Ensures you don’t forget anything Can be awkward to read instructions –But you can explain to participants why you are doing this

13. Pilot test A “practice run” of your usability test or experimental session –Gives you practice –Evaluates procedures, tasks, instructions, scripts, questionnaires, timing etc. –Gives you initial data – is your data answering the questions you care about? Participants don’t need to exactly match target users

14. Experiments vs. User testing Experiments test hypotheses -- investigate the relationship between two or more variables. User testing checks that a system is usable by the intended user population for their tasks.

15. User testing Aim: improve products Few participants Results inform design Not perfectly replicable Controlled conditions Results reported to developers Research experiment Aim: discover knowledge Many participants Results validated statistically Replicable Strongly controlled conditions Scientific paper reports results to community

16. When to Do an Experiment? Comparison Important consequences of decision Prediction uncertain For research –Important to show causality

17. Exercise 1 Experiment or user testing? –Will comparison of images be less accurate if done successively on a single screen as compared to simultaneously on two screens? –Can radiologists learn to use a trackball and do they find it to be a satisfactory input device? –Do radiologists prefer linearly or hierarchically organized lists of x-rays? –Are the number of consultations between radiology and emergency increased by increased availability of workstations in both sites?

18. Experiment Terminology Hypothesis: A tentative prediction that is testable Independent variable: systematically manipulated by the experimenter Dependent variable: what you measure Confounding factor: varies with the independent variable, making it impossible to attribute causality to the independent variable

19. Exercise 2: Hypothesis Improve this hypothesis: Popup menus are fast to use File Edit View Insert New Open Close Save File Edit View Insert New Open Close Save

20. User performance is faster when selecting a single item from a pop-up vs. a pull down menu of 4 items, regardless of the subject’s previous expertise in using a mouse or using the different menu types

21. Exercise 3: Variable Types Name the independent & dependent variables: –Is reading speed faster from a high resolution or low resolution screen? –Will radiologists prefer a mouse, lightpen, or trackball as a pointing device? –Can air traffic controllers enter data faster with a QWERTY or ABC keyboard?

22. Exercise 4: Confounds Name the intended independent variable and the confounding variable. Suggest a better design. –In a study to compare legibility of 2 visual displays, one display is located near a window and the other is in a windowless room. –To determine whether computer training leads to more positive opinions of computers, attitudes of students in a 1 st year computer course are compared to attitudes of 1 st year students who are not taking a computing course.

23. Exercise 4 continued Volunteers compared 2 command selection techniques: function keys and mouse. All volunteers used the function key approach followed by the mouse approach. An experiment compares three colour scales: a blue-yellow scale, a greyscale, and a rainbow scale. All participants complete the task first with the blue-yellow scale, then with the greyscale, and then with the rainbow scale.

24. Experimental Designs Between-subjects: each participant sees only one condition Within-subjects: each participant sees all conditions *** Need to counter-balance order of conditions Matched: Between-subjects design where the groups are matched on some factor (e.g. IQ or computer experience) Mixed: Some factors between, some within

25. Exercise 4 continued Do people remember information better with 2D displays or 3D displays? Participants were randomly assigned to 2D and 3D display groups. Instructions & experimental task were the same for both groups.

26. Exercise 4 continued An experiment compares (1) maps with an overview window to (2) maps without an overview. So that they do not repeat the exact same task twice, participants use a map of California with (1) and a map of Oregon with (2).

27. Ways to control confounding factors Randomize Counter-balance Hold constant Eliminate Match or balance

28. Exercise 5: Choose a Design Design an experiment to test the effects of the following factors on the speed of selecting an item in a menu: –Menu type: pop-up or pull-down –Menu length: 3, 6, 9, 12, 15 items –Subject type (expert or novice) (Decide whether each factor will be within or between-subjects & how to control order effects.)

29. Key Points Run pilot tests Set participants at ease Use experiments when you need to show causality. Otherwise user testing may suffice. Experiments must be defined very carefully to avoid confounds.