Sample size calculations Marie-Pierre Sylvestre mp.sylvestre@epimgh.mcgill.ca Material adapted from http://www.sgul.ac.uk/depts/phs/guide/size.htm June 2007

Why bother? Sample size calculations are important to ensure that estimates are obtained with required precision or confidence. E.g. a prevalence of 10% from a sample of size 20 ... 95%CI is 1% to 31%... ... a prevalence of 10% from a sample of size 400 ... 95%CI is 7% to 13% In studies concerned with detecting an effect if an effect deemed to be clinically or biologically important exists, then there is a high chance of it being detected, i.e. that the analysis will be statistically significant. If the sample is too small, then even if large differences are observed, it will be impossible to show that these are due to anything more than sampling variation.

Some terminology Significance level Cut-off point for the p-value, below which the null hypothesis will be rejected and it will be concluded that there is evidence of an effect. Typically set at 5%. One-sided and two-sided tests of significance Two-sided tests should be used unless there is a very good reason for doing otherwise. Power Power is the probability that the null hypothesis will be correctly rejected i.e. rejected when there is indeed a real difference or association. It can also be thought of as "100 minus the percentage chance of missing a real effect" - therefore the higher the power, the lower the chance of missing a real effect. Power is typically set at 80% or 90% but not below 80%. Effect size of clinical importance This is the smallest difference between the group means or proportions (or odds ratio/relative risk closest to unity) which would be considered to be clinically or biologically important. The sample size should be set so that if such a difference exists, then it is very likely that a statistically significant result would be obtained.

Example (1) Estimating a single proportion Scenario: The prevalence of dysfunctional breathing amongst asthma patients being treated in general practice is to be assessed using a postal questionnaire survey (Thomas et al. 2001). Required information: Primary outcome variable = presence/absence of dysfunctional breathing 'Best guess' of expected percentage (proportion) = 30% (0.30) Desired width of 95% confidence interval = 10% (i.e. +/- 5%, or 25% to 35%) Formula for sample size for estimation of a proportion is n = 15.4 * p * (1-p)/W2 where n = the required sample size p = the expected proportion - here 0.30 W = width of confidence interval - here 0.10 To get a feel

Example (2) Estimating a single proportion Here we have: n = 15.4 * 0.30 * (0.70)/ 0.102 = 324 "A sample of 324 patients with asthma will be required to obtain a 95% confidence interval of +/- 5% around a prevalence estimate of 30%. To allow for an expected 70% response rate to the questionnaire, a total of 480 questionnaires will be delivered." Note: The formula presented below is based on 'normal approximation methods', and, should not be applied when estimating percentages which are close to 0% or 100%. In these circumstances 'exact methods' should be used.

Prevalence/Proportion http://www.cs.uiowa.edu/~rlenth/Power/ Then, test for 1 proportion

Cohort studies Epi Info: http://www.cdc.gov/epiinfo/

Unmatched case-controls http://stat.ubc.ca/~rollin/stats/ssize/caco.html http://calculators.stat.ucla.edu/powercalc/binomial/case-control/index.php

Clinical Trials http://hedwig.mgh.harvard.edu/sample_size/quan_measur/assoc_quant.html

Simple Survival Analysis and Regression PS: http://biostat.mc.vanderbilt.edu/twiki/bin/view/Main/PowerSampleSize

Which variables should be included in the sample size calculation? The sample size calculation should relate to the study's primary outcome variable. If the study has secondary outcome variables which are also considered important (as is often the case), the sample size should also be sufficient for the analyses of these variables.

Allowing for response rates and other losses to the sample The sample size calculation should relate to the final, achieved sample. Therefore, the initial numbers approached in the study may need to be increased in accordance with the expected response rate, loss to follow up, lack of compliance, and any other predicted reasons for loss of subjects. The link between the initial numbers approached and the final achieved sample size should be made explicit.

Consistency with study aims and statistical analysis If the aim is to demonstrate that a new drug is superior to an existing one then it is important that the sample size is sufficient to detect a clinically important difference between the two treatments. However, sometimes the aim is to demonstrate that two drugs are equally effective. This type of trial is called an equivalence trial or a 'negative' trial. The sample size required to demonstrate equivalence will be larger than that required to demonstrate a difference. The sample size calculation should also be consistent with the study's proposed method of analysis, since both the sample size and the analysis depend on the design of the study.

Pitfalls to avoid (1) "The throughput of the clinic is around 50 patients a year, of whom 10% may refuse to take part in the study. Therefore over the 2 years of the study, the sample size will be 90 patients. " Although most studies need to balance feasibility with study power, the sample size should not be decided on the number of available patients alone. Where the number of available patients is a known limiting factor, sample size calculations should still be provided, to indicate either The power which the study will have to detect the desired difference of clinical importance, or The difference which will be detected when the desired power is applied.

Pitfalls to avoid (2) "Sample sizes are not provided because there is no prior information on which to base them." Where prior information on standard deviations is unavailable, sample size calculations can be given in very general terms, i.e. by giving the size of difference that may be detected in terms of a number of standard deviations.

Pitfalls to avoid (3) "A previous study in this area recruited 150 subjects and found highly significant results (p=0.014), and therefore a similar sample size should be sufficient here." Previous studies may have been 'lucky' to find significant results, due to random sampling variation. Calculations of sample size specific to the present, proposed study should be provided, including details of power significance level primary outcome variable effect size of clinical importance for this variable standard deviation (if a continuous variable) sample size in each group (if comparing groups)

References http://www.sgul.ac.uk/depts/phs/guide/size.htm