1. Quantitative Methods for Researchers Paul Cairns paul.cairns@york.ac.uk

2. Objectives  Statistics for severe tests  Designing for analysis  A whizz through some stats  Time for questions Quant for RS, Paul Cairns

3. But first…  Does making game playing more sociable improve the player experience?  Experiments, please Quant for RS, Paul Cairns

4. A little example  Ten people played a game with AI on or with AI off. Get scores: Quant for RS, Paul Cairns AI OffAI On 2612 2315 3029 2021 2517 Mean24.818.8

5. Quant for RS, Paul Cairns Calculate probabilities  Statistical test – Transform data to a standard form – Calculate a statistic – Convert to probability  “Unlikely” events (low p) are good  Test depends on data

6. Mann-Whitney  Non-parametric test  Rank all scores (lowest =1, highest = 10)  Sum ranks of each group  Look up probability (in Howell)  Let’s do that now Quant for RS, Paul Cairns

7. Significance  What is unlikely?  Conventions: 0.05, 0.01  Is AI harder?  1 in 20 experiments give an incorrect result!

8. Statistical Argument  Inference is an argument form  What is being severely tested? – Alternative hypothesis – “X causes Y” – Null hypothesis  Statistics serve the test Quant for RS, Paul Cairns

9. Gold standard argument, mk 2 1.Specify your idea 2.Devise a severe test of hypothesis 3.Collect data rigorously 4.Statistics give probabilities of null 5.Unlikely predictions “prove” your case Quant for RS, Paul Cairns

10. Implications  Must have an idea under test  No multiple testing  No post hoc analysis  Need multiple experiments Quant for RS, Paul Cairns

11. Silver standard argument, mk 2 1.Specify some ideas 2.Devise an experiment to exhibit ideas 3.Collect data rigorously 4.Use statistics to suggest (unlikely) patterns 5.Follow up findings with gold standard work Quant for RS, Paul Cairns

12. Fishing: This is bad science 1.Collect lots of data – DVs and IVs 2.Data variations could be chance 3.Test until a significant result appears 4.Report the tests that were significant 5.Claim the result is important Quant for RS, Paul Cairns

13. Statistical pit…  … is bottomless!  Safe designs – One (or two) IV – Two (or three) conditions – One primary DV  Other stuff is not severely tested Quant for RS, Paul Cairns

14. Descriptive statistics  Central tendency (averages)  Spread  Correlations Quant for RS, Paul Cairns

15. Seeing location  Boxplots  Median, IQR,  “Range”  Outliers Quant for RS, Paul Cairns

17. Seeing correlation  Scatterplots  Linear relationships! Quant for RS, Paul Cairns

19. Distributions  Theoretical stance – Starting model  Must have this!  Not inferred from samples Quant for RS, Paul Cairns

20. Choosing a test  What’s the data type?  Do you know the distribution?  Within or between  What are you looking for? Quant for RS, Paul Cairns

21. Parametric tests  Normal distribution  Two parameters  Null = one underlying normal distribution  Differences in location (mean) Quant for RS, Paul Cairns

22. t-test: null vs alternate Quant for RS, Paul Cairns

23. t-test  Two samples  Two means  Are means showing natural variation?  Compare difference to natural variation Quant for RS, Paul Cairns

24. Effect size  How interesting is the difference? – 2s difference in timings – Significance is not same as importance  Cohen’s d Quant for RS, Paul Cairns

25. ANOVA  Parametric  Multiple groups  Why not do pairwise comparison?  Get an F value  Follow up tests Quant for RS, Paul Cairns

26. ANOVA++  Multiple IV – So more F values!  Within and between  Effect size, η 2 – Amount of variance predicted by IV Quant for RS, Paul Cairns

27. Non-parametric tests  Unknown underlying distribution  Heterogeneity of variance  Non-interval data  Usually test location  Effect size is tricky! Quant for RS, Paul Cairns

28. Basic tests  Mann-Whitney  Wilcoxon  Kruskal-Wallis  Friedman  No accepted two-way tests Quant for RS, Paul Cairns

29. Health warnings  Craft skill  Simpler is better – Doing it – Interpreting it – Communicating it  Experiments as evidence  Software packages are deceptively easy Quant for RS, Paul Cairns

30. Questions  Specific problems  Specific tests  Other tests? Quant for RS, Paul Cairns

31. Useful Reading  Cairns, Cox, Research Methods for HCI: chaps 6  Rowntree, Statistics Without Tears  Howell, Fundamental Statistics for the Behavioural Sciences, 6 th edn.  Abelson, Statistics as Principled Argument  Silver, The Signal and the Noise Quant for RS, Paul Cairns

32. Multivariate  Multiple DV  Multivariate normal distribution – Normal no matter how you slice  MANOVA  Null = one underlying (mv) normal distribution Quant for RS, Paul Cairns

34. Issues  Sample size  Assumptions  Interpretation  Communication Quant for RS, Paul Cairns

35. Monte Carlo  Process but not distribution  Generate a really large sample  Compare to your sample  Still theoretically driven! Quant for RS, Paul Cairns

36. Example  Event = 4 heads in a row from a set of 20 flips of a coin  You have sample of 30 sets  18 events  How likely? – Get flipping! Quant for RS, Paul Cairns