Quantitative Methods for Researchers Paul Cairns

Objectives  Statistical argument  Safe designs  A whizz through some stats  Time for questions 2

3 Statistical Argument  Inference is an argument form  Prediction is essential – Alternative hypothesis – “X causes Y”  No prediction – measuring noise

4 Gold standard argument 1.Collect data 2.Data variation could be chance (null) 3.Predict the variations (alternative) 4.Statistics give probabilities 5.Unlikely predictions “prove” your case

5 Implications  Must have an alt (testable) hyp  No multiple testing  No post hoc analysis  Need multiple experiments

6 Silver standard argument 1.Collect data 2.Data variations could be chance (null) 3.Are there “real” patterns in the data? 4.Use statistics to suggest (unlikely) patterns 5.Follow up findings with gold standard work

7 Fishing: This is bad science 1.Collect lots of data – DVs and IVs 2.Data variations could be chance 3.Test until a significant result appears 4.Report the tests that were significant 5.Claim the result is important

Statistical pit…  … is bottomless!  Safe designs – One (or two) IV – Two (or three) conditions – One primary DV  Other stuff is not severely tested 8

Choosing a test  What’s the data type?  Do you know the distribution?  Within or between  What are you looking for? 9

Seeing location  Boxplots  Median, IQR,  “Range”  Outliers 10

11

Distributions  Theoretical stance  Must have this!  Not inferred from samples 12

13 Parametric tests  Normal distribution  Two parameters  Null = one underlying normal distribution  Differences in location (mean)

t-test: null vs alternate 14

t-test  Two samples  Two means  Are means showing natural variation?  Compare difference to natural variation 15

Effect size  How interesting is the difference? – 2s difference in timings – Significance is not same as importance  Cohen’s d 16

ANOVA  Parametric  Multiple groups  Why not do pairwise comparison?  Get an F value  Follow up tests 17

ANOVA++  Multiple IV – So more F values!  Within and between  Effect size, η 2 – Amount of variance predicted by IV 18

Non-parametric tests  Unknown underlying distribution  Heterogeneity of variance  Non-interval data  Usually test location  Effect size is tricky! 19

Basic tests  Mann-Whitney  Wilcoxon  Kruskal-Wallis  Friedman  No accepted two-way tests 20

Choosing a test For your fantasy abstract, what test would you choose? Why? Would you change your design? 21

Questions  Specific problems  Specific tests  Other tests? 22

Useful Reading  Cairns, Cox, Research Methods for HCI: chaps 6  Rowntree, Statistics Without Tears  Howell, Fundamental Statistics for the Behavioural Sciences, 6 th edn.  Abelson, Statistics as Principled Argument  Silver, The Signal and the Noise 23

Multivariate  Multiple DV  Multivariate normal distribution – Normal no matter how you slice  MANOVA  Null = one underlying (mv) normal distribution 24

25

Issues  Sample size  Assumptions  Interpretation  Communication 26

Monte Carlo  Process but not distribution  Generate a really large sample  Compare to your sample  Still theoretically driven! 27

Example  Event = 4 heads in a row from a set of 20 flips of a coin  You have sample of 30 sets  18 events  How likely? – Get flipping! 28