Sample Size August, 2007 Charles E. McCulloch Professor and Head, Division of Biostatistics Department of Epidemiology and Biostatistics

Outline Introduction Sample size calculation ingredients Statistical ideas Calculations Sample size in relation to If you don’t like the answer Caveats and what if’s Summary

Introduction: Salk Vaccine Trial In the 1950’s polio caused about 6% of the deaths among children aged 5-9. Its epidemic style of attack and its attack of children caused it to be of great concern. Jonas Salk (then of the University of Pittsburgh) developed a dead-virus vaccine. How large a study would be needed to prove it effective?

The prevalence of polio at the time was approximately 3 in 10,000 (.0003). It was hoped that the vaccine would reduce that by 40% to 1.8 in 10,000 (.00018). (Relative risk of 0.6). We’ll use accepted levels of accuracy to do the calculation (significance level of 5%, power of 80% - explanation in a bit). The study was a randomized, controlled, double- blind, prospective cohort study to test the ability of the vaccine to reduce the incidence of polio. Standard statistical analyses were to be performed (chi-square test). Can use the formula at the bottom of p.87 in DCR. Some Details

Required Sample Size for the Salk Vaccine Trial Using a one-sided test at a significance level of 0.05 and with a power of 0.8: Using a two-sided test at a significance level of 0.05 and with a power of 0.8:

Salk Vaccine Trial: Results GroupNPolio Rate per 100,000 Placebo201,22971 Vaccine200,74528 (RR = 0.4)

Sample Size Depends On W A D S hat you want to show ccuracy you need to achieve esign of your study tatistical analysis you will use

Sample Size Depends On What you want to show  Looking for a drop or a difference?  Simple hypothesis or complex one?  In a subgroup or overall?

Sample Size Depends On Accuracy you need to achieve  Analytic study: degree of proof required  Descriptive study: level of accuracy required

Sample Size Depends On Design of your study  Choice of measurements  How data are collected  How many treatment groups

Sample Size Depends On The statistical analysis you will use  Will you treat hemoglobin as continuous or dichotomize at 11 g/dl?  Will you compare two groups using a t-test or perform a regression or correlation analysis?  Will you use repeated measurements or just look at a change score from beginning to end?

Statistical Ideas Descriptive versus analytic studies Hypotheses (null and alternative) Level of significance Power Effect size Standardized effect size

Descriptive vs Analytic Probably analytic if you are using any of the following words: Case-control, cohort, experiment, cross-over, more than, less than, in contrast to, association between, a cause of, compared with, odds ratio, relative risk, correlated with, change in. The Salk trial was

Hypotheses Null hypothesis – there is no difference (notation H 0 ) Alternative hypothesis – what we wish to prove (notation H A or H 1 ) Hypothesis test: A formal decision rule for using the data to decide whether or not to reject the null hypothesis

Hypotheses – Salk rial Null hypothesis – there is no difference (notation H 0 ) Alternative hypothesis – what we wish to prove (notation H A or H 1 )

Hypotheses – One vs Two-sided Alternatives One-sided: When looking for a difference only in one direction. More focused, smaller sample size required. Two-sided: When looking for a difference in both directions. Safer, larger sample size. Salk trial: Reality check: What would you do if the results went dramatically in the reverse of the direction you predicted?

Significance Level Probability of rejecting the null hypothesis when it is, in fact, true. (notation , also  -level) Salk trial: Probability of claiming the vaccine is better or worse than placebo (when using a hypothesis test) when, in fact, it is the same as the control. Typically Sometimes 0.01 (if rejecting the null when true is serious). Sometimes 0.10 (if rejecting the null when true isn’t so serious).

Power Probability of rejecting the null hypothesis when it is, in fact, false Salk trial: Probability of claiming the vaccine is better or worse than placebo (when using a hypothesis test) when, in fact, it is better or worse than the control. If there is an effect in the accessible or target population will we always show it when we conduct the study?, i.e, proving your point

Thumbtack Experiment Do thumbtacks land point up or point down? H A : Pr(point up)>1/2 H 0 : Pr(point up)=1/2 Usual test: Reject H 0 if 8 or more points up out of 10 How often will this prove the point?

Conduct the Experiment 0 up, 10 down 0 up, 10 down 1 up, 9 down 2 up, 8 down 3 up, 7 down 4 up, 6 down 5 up, 5 down 6 up, 4 down 7 up, 3 down 8 up, 2 down 9 up, 1 down 10 up, 0 down

What is The Power? If the true heads up probability is 0.6, the power is only 0.17

Power (continued) Notation:  = 1- power. (Probability of failing to prove the point when it is true). Power is typically in the range of 0.8 and (Why not 1.00?) So  is typically in the range of 0.20 to 0.05.

Is It the Thumbtacks? The thumbtack “experiment” mimics the exact behavior of a hypothesis test with a binary outcome. So calculations for the thumbtacks apply equally as well to any experiment. Thumbtacks are versatile …

Effect Size How big is the effect we are looking for? Salk trial:

Standardized Effect Size How big is the effect we are looking for in relation to the variability in the data? With dichotomous outcome data, the variability is built in (and built in to the formulas) – so we don’t have to worry about specifying it. For the Salk trial it is (as we will see this is verrrry small for a standardized effect size) For continuous outcome data we need to specify the standard deviation of the measurements.

The Brass Tacks: Calculations W – Set up hypotheses/specify description A – Specify , power, standardized effect size or Confidence, desired precision D – Consider the design which will lead to the S – Statistical analysis, which gives the proper formulas or tables.

Example Prospective cohort study to test whether non- aggressive vs aggressive treatment of severe back pain (hospitalization and drug therapy) affects the two year cost of treatment. Descriptive or analytic?

(W)hat you want to show Null hypothesis? Alternative hypothesis? (one or two sided?)

(A)ccuracy Significance level = 0.05 Power = 0.80 (so  = = 0.20) E=Effect size = $200 S=Standard deviation = $500

(D)esign Two independent samples t-test (S)tatistical analysis Continuous outcome, two groups

Look up in Table 6A, p.84

n=100 per group

Specifying the Effect Size Previous study, similar outcome Clinically important difference Write down a hypothetical data set and calculate it Pilot data Try a wide range of reasonable values

Specifying the Standard Deviation Previous study, similar outcome Range/4 (In about 25 patients if data not too skewed) Write down a hypothetical data set and calculate it Pilot data Try a wide range of reasonable values Specify the standardized effect size

Sample Size in Relation to … For a fixed , power, and standard deviation, sample size changes inversely proportional to the square of the effect size. For a fixed , power, and effect size, sample size changes proportional to the square of the standard deviation.

If You Don’t Like the Answer: Design: Dichotomous vs continuous outcomes

If You Don’t Like the Answer: Design: Accuracy of measurement Design: paired/matched/longitudinal Design: consider unequal group sizes One-sided vs. two-sided alternatives Increase effect size by design or wishful thinking Reduce power

Caveats Dropouts (plan ahead) Use the appropriate standard deviation (e.g., use the standard deviation of the changes in a change score analysis). Will sample sizes really be equal? Are the assumptions met for the planned statistical analysis? Other statistical analyses

What if’s Multiple outcomes Multiple treatment groups Multivariate adjustment for other variables (adjust for age of the patients before comparing the treatment and control groups). Ordinal categorical outcomes (healthy, moderately ill, severely ill, dead). Survival analysis and censoring (time to re- hospitalization) Hierarchical or clustered data Equivalence studies (does a generic drug deliver the same dose of the active ingredient?

What if’s What if you find yourself in one of these situations? Multiple outcomes: multiple calculations Multiple groups: Do calculations for several of the important pairwise comparisons. Some other scenarios are covered in the text and references to even more are given.

Other Resources CTSI offers a free hour of consulting on design issues (including sample size) through its BREAD program: Professional sample size calculation programs are available, e.g., NQuery Advisor. I recommend the free program PS Power, which is good for simple situations. Available for download at: owerSampleSize owerSampleSize Spreadsheet calculators available on the course web site:

Summary W A D S Calculate N early and often. Be watchful of caveats/what if’s.