Type I & Type II errors Brian Yuen 18 June 2013

Definition of Type I error (α) Concluded a statistical significant effect size from the sample when this does not exist in the whole population (H0 is true) Reject the null hypothesis (H0) when it is true False positive result Level of significance - P value Usually allowed for 5% - arbitrary but acceptable rate! Repeatedly sample and test from the same population, you will expect to make 1 wrong conclusion (with statistically significant result) in 20 of these tests if the null hypothesis is in fact true

Definition of Type II error (β) Concluded a non-statistical significant effect size from the sample when this exists in the whole population (H1 true) Fail to reject the null hypothesis (H0) when it is not true False negative result Power of study = 100% - False negative rate Maximum accepted false negative rate at 20%  Minimum power of study at 80%

What does a significant result mean? Chance findings fall into this region Remember we are trying to find enough evidence to reject the null hypothesis i.e. to show that the observed effect size has exceeded our pre-set threshold of expected events, hence a statistical significant result Observed a significant result, are we lucky to find out null hypothesis is indeed incorrect? A genuine finding unlucky to observed an unlikely result/event? Type I error Findings fall in these 2 regions are too extreme to say these by chance

Can we eliminate these errors completely? Type I error Not really, there is always a pre-set (5%) chance for a Type I error You could minimise this by reducing the level of significance from 5% to 1%, but you would expect a larger sample size Type II error Not really, there is always a pre-set (20%) chance for a Type II error You could minimise this by reducing the error rate from 20% to 10%, but you would compromise this with a larger sample size

Can we ever work out if Type I error exist in published results? Yes, if the result is statistically significant, then there is a possibility of a Type I error Since we allowed for 5% chance to have this error, there is always a possibility of getting such error and there is no way we can get rid of this doubt We could work out if there is any hint of an unusual/unlikely result by comparing results from other similar studies Testing multiple outcomes issue P<0.05 means it is unlikely that the observed effect size is by chance (less than 5%) P<0.01 means it is unlikely that the observed effect size is by chance (less than 1%)

Can we ever work out if Type II error exist in published results? Yes, if the result is not statistically significant, then there is a possibility of a Type II error The key point is to identify if the author had performed a sample size calculation with a pre-defined power before the study starts on the one primary outcome they made conclusive statement whether they had recruited the desired number at the analysis stage whether they had pre-defined a clinically worthwhile effect size to be detected and was observed Then the chance of having a Type II error would be the pre-set value, i.e. 20%

Type I error (α) & Type II error (β) No True Difference True Difference No Observed Difference Well Designed Trial (100-) Type II Error () Observed Difference Type I Error () Well Powered Trial (100-)

Mind twisting quizzes! In a published paper, a statistically significant result of a fully powered 2-group study was reported on a single primary outcome. What can you comment on this result regarding Type I/II error? In a published paper, a non statistically significant result of a fully powered 2-group study was reported on a single primary outcome. What can you comment on this result regarding Type I/II error? You have reviewed 20 studies of a similar kind regarding the same outcome measure. You have found 5 studies with statistically significant results, while the others were not. What is your view on these results?

Reference http://www.bmj.com/about-bmj/resources- readers/publications/statistics-square-one/5- differences-between-means-type-i-an