Quantitative Methods for Researchers Paul Cairns

Objectives  Statistics for severe tests  Designing for analysis  A whizz through some stats  Time for questions Quant for RS, Paul Cairns

But first…  Does making game playing more sociable improve the player experience?  Experiments, please Quant for RS, Paul Cairns

A little example  Ten people played a game with AI on or with AI off. Get scores: Quant for RS, Paul Cairns AI OffAI On Mean

Quant for RS, Paul Cairns Calculate probabilities  Statistical test – Transform data to a standard form – Calculate a statistic – Convert to probability  “Unlikely” events (low p) are good  Test depends on data

Mann-Whitney  Non-parametric test  Rank all scores (lowest =1, highest = 10)  Sum ranks of each group  Look up probability (in Howell)  Let’s do that now Quant for RS, Paul Cairns

Significance  What is unlikely?  Conventions: 0.05, 0.01  Is AI harder?  1 in 20 experiments give an incorrect result!

Statistical Argument  Inference is an argument form  What is being severely tested? – Alternative hypothesis – “X causes Y” – Null hypothesis  Statistics serve the test Quant for RS, Paul Cairns

Gold standard argument, mk 2 1.Specify your idea 2.Devise a severe test of hypothesis 3.Collect data rigorously 4.Statistics give probabilities of null 5.Unlikely predictions “prove” your case Quant for RS, Paul Cairns

Implications  Must have an idea under test  No multiple testing  No post hoc analysis  Need multiple experiments Quant for RS, Paul Cairns

Silver standard argument, mk 2 1.Specify some ideas 2.Devise an experiment to exhibit ideas 3.Collect data rigorously 4.Use statistics to suggest (unlikely) patterns 5.Follow up findings with gold standard work Quant for RS, Paul Cairns

Fishing: This is bad science 1.Collect lots of data – DVs and IVs 2.Data variations could be chance 3.Test until a significant result appears 4.Report the tests that were significant 5.Claim the result is important Quant for RS, Paul Cairns

Statistical pit…  … is bottomless!  Safe designs – One (or two) IV – Two (or three) conditions – One primary DV  Other stuff is not severely tested Quant for RS, Paul Cairns

Descriptive statistics  Central tendency (averages)  Spread  Correlations Quant for RS, Paul Cairns

Seeing location  Boxplots  Median, IQR,  “Range”  Outliers Quant for RS, Paul Cairns

Seeing correlation  Scatterplots  Linear relationships! Quant for RS, Paul Cairns

Distributions  Theoretical stance – Starting model  Must have this!  Not inferred from samples Quant for RS, Paul Cairns

Choosing a test  What’s the data type?  Do you know the distribution?  Within or between  What are you looking for? Quant for RS, Paul Cairns

Parametric tests  Normal distribution  Two parameters  Null = one underlying normal distribution  Differences in location (mean) Quant for RS, Paul Cairns

t-test: null vs alternate Quant for RS, Paul Cairns

t-test  Two samples  Two means  Are means showing natural variation?  Compare difference to natural variation Quant for RS, Paul Cairns

Effect size  How interesting is the difference? – 2s difference in timings – Significance is not same as importance  Cohen’s d Quant for RS, Paul Cairns

ANOVA  Parametric  Multiple groups  Why not do pairwise comparison?  Get an F value  Follow up tests Quant for RS, Paul Cairns

ANOVA++  Multiple IV – So more F values!  Within and between  Effect size, η 2 – Amount of variance predicted by IV Quant for RS, Paul Cairns

Non-parametric tests  Unknown underlying distribution  Heterogeneity of variance  Non-interval data  Usually test location  Effect size is tricky! Quant for RS, Paul Cairns

Basic tests  Mann-Whitney  Wilcoxon  Kruskal-Wallis  Friedman  No accepted two-way tests Quant for RS, Paul Cairns

Health warnings  Craft skill  Simpler is better – Doing it – Interpreting it – Communicating it  Experiments as evidence  Software packages are deceptively easy Quant for RS, Paul Cairns

Questions  Specific problems  Specific tests  Other tests? Quant for RS, Paul Cairns

Useful Reading  Cairns, Cox, Research Methods for HCI: chaps 6  Rowntree, Statistics Without Tears  Howell, Fundamental Statistics for the Behavioural Sciences, 6 th edn.  Abelson, Statistics as Principled Argument  Silver, The Signal and the Noise Quant for RS, Paul Cairns

Multivariate  Multiple DV  Multivariate normal distribution – Normal no matter how you slice  MANOVA  Null = one underlying (mv) normal distribution Quant for RS, Paul Cairns

Issues  Sample size  Assumptions  Interpretation  Communication Quant for RS, Paul Cairns

Monte Carlo  Process but not distribution  Generate a really large sample  Compare to your sample  Still theoretically driven! Quant for RS, Paul Cairns

Example  Event = 4 heads in a row from a set of 20 flips of a coin  You have sample of 30 sets  18 events  How likely? – Get flipping! Quant for RS, Paul Cairns