1 Lecture 5 Experimental Design Lecturer – Prof Jim Warren (with references to Dix et al. chapter 9)

2 Evaluating Implementations Requires an artefact: a simulation, a prototype, or a full implementation

3 Experimental evaluation controlled evaluation of specific aspects of interactive behaviour evaluator chooses hypothesis to be tested a number of experimental conditions are considered which differ only in the value of some controlled variable. changes in behavioural measure are attributed to different conditions

4 Experimental factors Subjects (i.e., the users, aka ‘participants’) –who – representative, sufficient sample not the programmer’s friend, boss, etc. huge variability in performance of individuals Variables –things to modify and measure Hypothesis –what you’d like to show Experimental design –how you are going to show it –Includes ‘Protocol’ – what the subjects do

5 Variables independent variable (IV) characteristic changed to produce different conditions e.g. interface style, number of menu items dependent variable (DV) characteristics measured in the experiment e.g. time taken, number of errors.

6 Hypothesis prediction of outcome –framed in terms of IV and DV e.g. “error rate will increase as font size decreases” null hypothesis: –states no difference between conditions –aim is to disprove this e.g. null hyp. = “no change with font size”

7 Experimental design “within groups” design (also called “repeated measures”) –each subject performs experiment under each condition –transfer of learning possible (practice makes performance better; or alternatively fatigue or boredom makes it worse) –less costly and less likely to suffer from user variation (each user is compared to themselves) between groups design –each subject performs under only one condition –no transfer of learning –more users required

8 Within v. Between Consider a test on the difference of beer v. vodka martinis on reaction time –Null hypothesis – no difference in increase in reaction time between the two beverages Design 1: –30 people try beer; 30 other people try vodka – D.V. is change in reaction time pre- v. post drinking Not bad – be sure to randomize who goes into beer group v. vodka group But ‘power’ of the experiment will be reduced due to the great variability of individuals in reaction to alcohol

9 Within v. Between (contd.) Design 2: –All 60 people first try beer, then immediately try vodka Problem of carryover effect Better Design: –All 60 try beer, then a week later try vodka Now each individual is compared with themselves Still possible problem of ordering effect (e.g., they might get a little better at the reaction time test) Best Design: –30 try beer, then a week later vodka; 30 try vodka and then a week later beer

10 Analysis of data Before you start to do any statistics: –look at data (e.g. average=5.25 – but 4.9 without the “outlier”) Choice of statistical technique depends on –type of data –information required Type of data –discrete finite number of values may be ordered, or unordered (e.g., colors) –continuous any value

11 ANOVA – analysis of variance Quite easy to test whether there’s a significant difference between groups in Excel –Need to invoke Tools/Add-ins/Analysis Toolpack to enable –Then just apply Tools/Data Analysis/ANOVA: Single Factor to the data

12 ANOVA from Excel If P-value < 0.05 then we usually say the result is ‘significant’ (result is more than expected chance variation) Say we have three columns of numbers representing the time to complete a task for 5, 5 and 7 users using three variations of an interface

13 When is a difference a difference? In the world of parametric stats, we look for a statistic to be large enough to be ‘significant’ –On the Gaussian (‘normal’) curve a |Z|=1.96 leaves 95% of the area of the curve behind so is a common ‘critical value’ for claiming significance

14 Parametric assumptions Parametric statistics assume that some mathematically elegant assumptions hold true for the data –E.g., ANOVA (and standard ‘regression’) assume, among other things, normally distributed random error –Trivia: The mathematical form of the probability density function for the normal distribution is remarkably formidable Centres on mean, , and is flattened by standard deviation,  Galton machine simulates normal distribution (aka ‘bell curve’) Exponential distribution models time between events happening with a constant average rate

15 So, what to measure? Usually one (or several) of these things: –Speed / efficiency How many (or whatever) per unit time can the user process with this interface? –Accuracy (errors) –Learnability (time to acquire the ability to do something in particular with the interface) And retention (how well they can do it some particular time later) –Satisfaction Subjective assessment – how does the user feel about the interface

16 Subjective assessment Very important –If the user seriously doesn’t like it, probably there’s something really wrong with the design (just maybe they can’t articulate what) Quantifiable –Yes/No, or better, Likert scale (ratings) through interview or questionnaire Need to ask the right questions –E.g., don’t have leading questions (BAD: “Is this the absolute worst system you have used ever?”) And need to ask the questions well (so user reliably expresses what they mean) –E.g., don’t trip them up with double negatives

17 Likert Scale Can be from 4 to 7 “points” Usually about agreement to a phrase –E.g., “I found the search function easy to use” –Strongly Agree, Agree, Neutral (optional), Disagree, Strongly Disagree May also be about importance –E.g., “A site search function is…” –“not very important” to “extremely important” Or a general assessment –E.g., “The performance of the search function was…” –Poor, Fair, Satisfactory, Good, Excellent Great to ask open-ended questions, too –E.g., “What was the best aspect of the search function?” –But it’s the Likert scale data that you can quantify See Heim – pages

18 Dimensions and validity When designing a questionnaire… –Have in mind a few underlying issues that you are trying to assess –Ask a few different questions that are coordinated around each issue –Ask different ways – vary whether positive or negative favours the issue in question –Ideally, verify the questionnaire by having people role-play particularly happy or angry, and middle-of- the-road users And see if they answer the questions the way you’d expect!

19 Questionnaire administration Face-to-face or telephone interviews –Esp. efficient (and unbiased) to have third party give the telephone survey Mail-out (including ) questionnaire Web-based questionnaire (maybe out URL) Even a questionnaire is an experiment on humans from the point of view of research ethics –Easier to achieve an ethical questionnaire administration if it’s truly anonymous

20 Response rates and bias Low response rate is a problem –Below 50% response rate, one wonders whether respondents were exceptional (happiest, angriest or a mix; but not the “normal” folks) –Better if you have some authority to motivate a response But back to issues of ethics – e.g., not truly anonymous if you know who to nag about non- response; and the “pressure” may be unfair Similar bias problems when you use volunteers for any experiment –Are these volunteers representative of your “normal” users?

21 Experimental studies on groups More difficult than single-user experiments Problems with: –subject groups –choice of task –data gathering –Analysis Unfortunately (in terms of experimental requirements) a lot of things that are interesting in the real world, involve computers mediating group behaviour

22 Subject groups larger number of subjects  more expensive longer time to `settle down’ … even more variation! difficult to timetable so … often only three or four groups

23 Groups (contd.): Data gathering several video cameras + direct logging of application problems: –synchronisation –sheer volume! one solution: –record from each perspective

24 Groups (contd.): Analysis N.B. vast variation between groups solutions: –‘within groups’ experiments (each group works under various conditions) –micro-analysis (e.g., gaps in speech) –anecdotal and qualitative analysis controlled experiments may `waste' resources! –Experiments dominated by dynamics of group formation –Field studies are apt to be more realistic

25 About statistics It’s an amazingly complex field –A lot of hidden complexities in running experiments and saying that the observed differences really make a difference ‘threats to validity’ – are those things that make it possible that your experimental conclusion is in error –Threats to internal validity: like carryover effects, or lack of randomization –Threats to external validity: like that your whole population of subjects were unusual in some way, or the task was not representative of real use of the tool –When the outcomes are serious (e.g., medical trials) professional statisticians are always used in design of the experiment as well as analysis and reporting of the findings –Plenty of texts and courses on stats available (the Wikipedia is pretty good on this topics, too – e.g., for ANOVA)

26 Unwanted biases in studies You can’t always take a study result at face value… must be attentive to what subjects are feeling Hawthorne effect –Worker is more productive when observed John Henry effect –Worker is [stubbornly] more productive when using his old tools (see Placebo effect –[Patient usually] gets some benefit just because they expect a benefit Pygmalion effect –Student performs better simply because they are expected to do so

Usability Analysis - Conclusion Remember: the ultimate goal is to learn –Learn what’s working and, most critically, what isn’t working for the end user –Do the usability testing that helps you make the best possible interface Test within your constraints –A quick talk aloud protocol session is far better than nothing and will probably find the most critical flaws –Then again, if it’s a “bet the business” interface, and it’s a big business, than organise testing on an appropriate scale! Hardest bit might be finding the time; nobody likes to delay a product release (but nobody wants to release a failure, either) 27