Confidence intervals – the what, why and when Stats Club 9 Marnie Brennan

References Petrie and Sabin - Medical Statistics at a Glance: Chapter 11 Good Petrie and Watson - Statistics for Veterinary and Animal Science: Chapter 6 Good Dohoo, Martin and Stryhn – Veterinary Epidemiologic Research: Chapter 6 Other references – see the end of the presentation

What do you know about confidence intervals?

Follows on from hypothesis testing and p-values Determining if comparisons between groups are statistically significantly different Use a hypothesis test, and resulting p-values, to determine if sample groups are ‘related’ or not But what about the ‘certainty’ around the results we get? How certain are we that our sample results relate to our population of interest? There are other things you can measure to help decide as well as p-values

Confidence intervals (CIs) ‘A range of values for a variable of interest constructed so that this range has a specified probability of including the true value of the variable’ ‘ Shows the range within which the true treatment effect is likely to lie’ Davies and Crombie (2009) Specified probability = confidence level Range of values = confidence interval End points of the confidence interval = confidence limits

Calculate them for any estimate The way you calculate a confidence interval depends on what you are calculating: For the mean Normally distributed variables, or non-normally distributed variables For the proportion E.g. Binomial distribution Kappa measurement of agreement Correlations, regressions Odds ratios, relative risks etc.

So a bit like a p-value? Similar in that you can calculate a p-value or a CI for any estimate you are interested in But p-values just give you information as to whether the estimate is different Not the certainty behind this Some may say they are preferable to p-values Identify the range of possible effect sizes compatible with the data

An example of how to calculate them Generally it is an equation incorporating the standard error (but exact calculation will be estimate specific) E.g. For data that is normally distributed (and you want to work out the CI for the mean) Lower limit = Sample mean - 1.96 x SEM Upper limit = Sample mean + 1.96 x SEM E.g. if mean = 416 and SEM = 5.5, CI would be: 416 – 1.96 (5.5); 416 + 1.96(5.5) 405.2; 426.8 Would write this as ‘The mean was 416 (CI 405.2 – 426.8)’ Conventional to use the 95% level If we were to repeatedly re-run the same study and create a CI each time, 95% of the CIs would contain the true parameter value Can use others (90%, 99%) but this depends on what confidence you are looking for in your results

Petrie and Watson 2006

Why we use them Indicates the precision of the estimate we are interested in Helps us to decide whether something is biological importance versus statistical significance Can be used to test a null hypothesis about a parameter of interest Extrapolating from our sample to our population See previous Stats Club session

Hypothesis testing

How to interpret CIs Two main things to look at: Final important point The width of the CI If the width of the CI is really wide, indicates great uncertainty about where the point estimate sits Less confident about the results you’ve got If the CI crosses the point of ‘no difference’ E.g. If we are doing a case-control study and we are using odds ratios, the point of no difference would be 1 If the CI includes 1, it means we cannot tell what affect the variable is having Final important point Is the result relevant? ‘Biologically important’ versus ‘Statistically important’

Factors affecting CIs Sample size Smaller the sample, the wider the intervals Standard deviation (how variable the values are from the mean) More variability, the wider the intervals Selected confidence level (e.g. 95%, 90%) 99% interval is wider than a 95% interval or a 90% interval Must look at the sample biases when interpreting results Even with large sample sizes and narrow confidence intervals, the results can be misleading Usually if the P value is not significant, there will be an issue with the CIs as well (and vice versa)

du Prel et al. 2009

Example 1 What can we determine from these results? Buchholz et al. (2011) German outbreak of E.coli 0104:H4 associated with sprouts. The New England Journal of Medicine

Example 2 What can we determine from these results? Westgarth et al. (2015) Factors associated with daily walking of dogs. BMC Veterinary Research

Example 3 What can we determine from these results? Bucher et al. (2012) Do any spray or dip treatments, applied on broiler chicken carcasses or carcass parts, reduce Salmonella spp. Prevalence and/or concentration during primary processing? A systematic review –meta-analysis. Food Control

Calculating them Analysis specific – check your programme for details! How to do them in Microsoft Excel: http://www.gov.scot/Topics/Statistics/Browse/Health/scottish-health-survey/ConfidenceIntervals/ConfidenceIntervalExcel

Calculating in SPSS

Calculating in SPSS

Summary Confidence intervals can be useful to determine the uncertainty around a point estimate that you have calculated Gives you more information than just the p-value You should use/report both p-values and confidence intervals – and an understanding of where your sample has come from (bias) Statistical versus biological significance

Next time Power calculations and sample size…… References: Bucher, O., Rajić, A., Waddell, L.A., Greig, J. and McEwen, S.A., 2012. Do any spray or dip treatments, applied on broiler chicken carcasses or carcass parts, reduce Salmonella spp. prevalence and/or concentration during primary processing? A systematic review–meta-analysis. Food control, 27(2), pp.351-361. Davies, H.T. and Crombie, I.K., 2009. What are confidence intervals and p-values. London: Hayward Medical Communications. du Prel, J.B., Hommel, G., Röhrig, B. and Blettner, M., 2009. Confidence interval or p-value?: part 4 of a series on evaluation of scientific publications. Deutsches Ärzteblatt International, 106(19), p.335. Scottish Government (2017) Confidence intervals. Accessed at: http://www.gov.scot/Topics/Statistics/Browse/Health/scottish-health-survey/ConfidenceIntervals Last accessed 06.12.17