ICS 463, Intro to Human Computer Interaction Design: 9. Experiments Dan Suthers

Experiments and Usability Tests Preview: Controlled Experiments –Control all but one variable –Very informative about specific issues –Extremely resource intensive –Questionable ecological validity Usability Tests –Less rigorous in terms of variables controlled –Informative about selected benchmarks –Somewhat resource intensive –Better but still questionable ecological validity

Experiments Decide why you are doing it Pick a testable hypothesis Define a method –Subjects –Materials –Procedures and Instructions Choose statistical tests Interpret results Pilot Study!

Variables Independent variable –Hypothesized causal factor –What you modify –“The input to the organism” Dependent variable –Hypothesized effect –What you measure –“The output from the organism”

Subjects or Participants Must balance for … –Age –Gender –Prior experience –Aptitude Consider incentive to participate Obtain informed consent –Aware of risks and benefits –Option to quit at any time

Experimental Designs at a glance Between Subjects: each experimental condition has different subjects Within Subjects: each condition has the same subjects One-way, 2x2, etc.

Between Subjects Designs Between subjects good for ensuring no cross-treatment effects, but it requires more subjects and raises issues of whether the groups are the same. Two ways to make the groups the same: Independent subject design: randomly assign to experimental conditions Matched subject design: design the groups to be similar by matching subjects

Within subjects designs Pre-post test: measure before & after treatment –Problem: confounds with time on task Repeated Measures: each subject tested in both experimental conditions –Problem: TOT and cross-condition effects –Solution: counterbalance order in which conditions given Half the subjects: Condition A, Condition B Other half: Condition B, Condition A –Can get complicated with more conditions

2x2 designs –Condition A, Condition 1 –Condition A, Condition 2 –Condition B, Condition 1 –Condition B, Condition 2 Good for finding interactions between two variables (e.g., using ANOVA) Balancing order of treatment gets complex

Critiquing Procedure Critique your own before you run it! –Instructions –Amount of practice –User’s interpretation of IV –Sufficient but not excessive task complexity –Users understand tasks –Sufficient but not excessive time on task

Critiquing Results Is size of effect meaningful? Are there alternative explanations? Are the results consistent? Compare dependent variables How general are the results?

Usability Engineering Instead of comparing treatment groups, specify quantitative objectives on a set of measures and test interface on representative set of users until these objectives are met Less concerned with experimental design and controlling variables More oriented towards design

Doing Usability Engineering Define usability goals –Choose metrics –Set quantitative levels to achieve Design: Use available information to make best choice –Analyze design solutions w.r.t. these metrics –Incorporate user-derived feedback into design Evaluate resulting design against metrics Iterate through Design and Evaluate until metric objectives achieved

More on Usability Engineering Benchmark Tasks: standard and representative set of tasks used in each iteration of evaluation Attitude metrics may also be used Design may require tradeoffs between measurable objectives This method is attractive to organizations desiring measurable objectives but is criticized for lack of ecological validity: use field observations to compensate