Introduction to sample size and power calculations Afshin Ostovar Bushehr University of Medical Sciences

Questions 1. How many samples do we need? 2. How much chance do we have to reject the null hypothesis when the alternative is in fact true? (what’s the probability of detecting a real effect?) 3. Can we quantify how much power we have for given sample sizes?

Why don’t we take the sample size as large as possible? 1. Economy: o Why should we include more than we have to? o Every trial costs! 2. Ethics: o We should never punish more test persons or animals than necessary 3. Statistics: o We can proof almost every effect, if we only have sufficiently large sample size o Stress field: statistical significance vs. clinical relevance

What do we want to do? We have to find the correct sample size to detect the desired effect. Not too small - Not too large

What do we need on the way?  How does a test work?  What means ”Power of a test”?  What determines the sample size?  How do we handle this in practical tasks?

A short introduction to hypotheses testing Strategy: 1. Formulate a hypothesis Expected heights equal vs. Expected heights different Null hypothesis vs. Alternative H0 vs. H1 2. Find an appropriate test statistics 3. Compute the observed test statistics 4. Reject the null hypothesis H0 if Test statistic is too large. But what does this mean: ”too large”?

Possible results of a single test Test Decision RejectAcceptReality Type I ErrorRightH0 True RightType II ErrorH0 False

Wrong decisions: o Rejection even if H0 is true (type I error) o No rejection even if H0 is false (type II error) What do we want? o Reduction of the wrong decisions. o ⇒ Minimal probability for both types of errors.

Two opaque jars, each holding 100 beads, some blue and some white

Statistical approach o Idea: What is the probability that everything happened by accident? o Solution: The p-value The p-value is for a given data set the probability to get the observed test statistics or worse assuming that the null hypothesis is true We reject the null hypothesis if the p-value is under a given significance level (usual convention = 0.05) The probability of type I error (incorrect rejection) will be lower than the used significance level.

Power and Sample Size o Recall: α Probability for wrong rejections (type I error) o Definition β: Probability for wrong acceptations (type II error) The power of a test is the probability to detect a wrong hypothesis Power = 1 − β

Principles of Sample Size 1. Variability 2. Effect size 3. Significance level 4. Power of the test 5. Type of the test

Variability o e.g. standard deviation of both samples for a t-test o taken some experience former or pilot studies

Effect Size o What effect (mean difference, risk, coefficient size) is clinically relevant? o Which effect must be detectable to make the study meaningful? o This is a choice by the researcher (not statistically determined!)

Significance level o usually set to = 0.05 o Adjustments must be taken into account (e.g. multiple testing)

Power of the test o often used 1 − β = 0.8 o This is a choice by the researcher (not statistically determined!)

Type of the Test o Different test for the same problem often have different power

What to put in a grant application 1. Bare essentials:  Proportions  P1  The effect size (from which P2 and the standard error can be calculated)  Means  Population mean for the controls  The effect size  Standard deviation for either the cases or controls

What to put in a grant application

Some points  The variable you choose for the calculation should be the most important one to the interpretation of the study.  Sometimes it is wise to do calculations for a number of outcome variables to see if they all give about the same sample size or if there are some variables which are more likely to yield precise results than others.

Some points, cont’d

Altman Nomogram

Sample Size Calculation software  SPSS  STATA  PASS  Gpower  PS  Epiinfo  …

What this workshop covers  Means 1. Estimation 2. Comparing with a constant 3. Comparing two independent groups 4. Comparing ≥ two independent groups 5. Comparing before and after situations  Proportions 1. Estimation 2. Comparing with a constant 3. Comparing two independent groups  Correlation & Regression

What this workshop does not cover  Complex formulas  Special cases:  Survival analysis  Survey sampling methods  Crossover trials  Non-inferiority trials  Missing data  Subgroup analysis  Non-parametric tests  …