1. Understanding Science 13. Significance © Colin Frayn, 2015 www.frayn.net

2. Introduction We can learn from magic! Humans are… –Easily fooled –Bad at understanding statistics –Bad at recognising randomness Significance –What’s a real result? –What’s just an anomaly? –Which claims should we pay attention to? © Colin Frayn, 2015 www.frayn.net

3. Randomness Randomness doesn’t always look random Random sequences do contain “patterns” 9838150934321876154747913 0829992687913986434088388 4784429141680340521748650 9351414414141450088184099 You often find patterns in random numbers… …and random events This leads to lots of pseudoscience © Colin Frayn, 2015 www.frayn.net

4. Superstition Unlucky 13. Or 4. And lucky 8? Highly culturally correlated Superstitions are mostly caused by Confirmation Bias –We recognise and remember things more often if we expect them –We recognise and remember things less if we aren’t looking out for them –We are creating a false significance Some superstitions become self-perpetuating © Colin Frayn, 2015 www.frayn.net

5. Stastistical Significance Flipping a coin twice: –HH, HT, TH, TT (25%) Flipping a coin five times: –HHHHH, HHHHT, HHHTH, HHHTT, HHTHH etc.. (~3%) Standard Deviation –Measures average distance of samples from a mean value –Lower value means samples are more consistent –Higher value means samples are more variable © Colin Frayn, 2015 www.frayn.net

6. Normal Distribution Johann Carl Friedrich Gauss (1809) © Colin Frayn, 2015 www.frayn.net +1  +2  +3  -1  -2  -3  Measurement Probability of Measurement Greek Letter Sigma (  ) Standard Deviation 1  = 68% 2  = 95% 3  = 99.7%

7. Experiment Design Publication Bias –Only publish “interesting” studies –Can be done entirely by accident –Biases results Multiple Variables –Measure lots of things –Only report on the ones that changed significantly –Also biases results More “Researcher Degrees of Freedom” –Researcher decides when to stop measuring How to fix the problem –Require pre-registration of clinical trials –Must include full methodology and measures of success © Colin Frayn, 2015 www.frayn.net

8. Repetition and Replication How else can we avoid fake significance? Replicate studies –Studies should include methodology –Should give sufficient detail for exact replication –Failure to replicate could show original result was false –Replication increases confidence –But be careful of errors in the experiment design Independent confirmation –Find a different way to test the same phenomenon –This enhances confidence as before –Greatly reduces chance of design errors © Colin Frayn, 2015 www.frayn.net

9. Summary Significance is all about random chance –We propose a hypothesis –We create a test that distinguishes between our hypothesis being true or false –We measure the outcomes –We expect data in agreement with our hypothesis –Significance is related to how unlikely the results were to have been obtained if the hypothesis were false. Sigma () = standard deviation –More sigma = more significant Fake significance –Confirmation bias (e.g. superstition) –Publication bias –Researcher degrees of freedom Ways to avoid fake significance –Replication / repetition –Experiment design e.g. pre-register clinical trials, decide on outcomes up front © Colin Frayn, 2015 www.frayn.net