The Importance of Adequately Powered Studies William S. Weintraub, MD Christiana Care Health System

I have no real or apparent conflicts of interest to report. William S. Weintraub, MD I have no real or apparent conflicts of interest to report.

Power and P Value Power: The probability of finding a difference if there really is one P Value: The probability of finding a difference is one does not really exist Both are inferential statistics

Inferential Statistics? Sample from a large unknown population Sample Underlying Population

What do P Value and Power Mean? Power is the Sensitivity Probability of finding a difference if there is one P Value is the False Positive Rate (1- Specificity) The probability of finding a difference is one does not exist What we really want in Predictive Value: Given the results of the trail, the probability that there is a difference between arms

Frequentist vs Bayesian Statistics Frequentist approaches offer statistical inference without knowledge of the underlying population However the metrics are limited and often misinterpreted Bayesian statistics offer meaningful metrics, but require assessment of prior knowledge of the underlying population

Types of Error Finding a difference if there really isn’t one, α error, p less than specified limit (often 0.05). Find no difference if there really is one, β error. Both are related to small sample size.

How Do You Assure Adequate Power During Design Power Calculation: Determine power required (generally 0.80 to 0.95) Determine p value required (generally 0.05 two tailed) Determine clinically meaningful difference Algorithm will find patients required to achieve power

Example: AIM HIGH The Role of Niacin in Raising HDL-C to Reduce Cardiovascular Events in Patients with Atherosclerotic Cardiovascular Disease and Optimally Treated LDL-C Randomized Trial of Niacin+Simvastatin vs Placebo+Simvastatin, composite outcome With an estimated sample size of 3,300, the trial had 99% power to detect a 25% reduction in primary outcome in the niacin arm compared to placebo arm, assuming a 0.089 hazard rate in the placebo arm and using one-sided P value of .025.

Study Design ER Niacin + 40-80 mg/day simvastatin Placebo + 40-80 mg/day simvastatin R Follow to end of study -2 -1 1 2 3 6 12 Months Relative to Randomization

Endpoints Primary Outcome Composite (Time to First Occurrence): Coronary Heart Disease Death Non-Fatal MI Ischemic (Non-Hemorrhagic) Stroke Hospitalization for ACS Symptom-Driven Revascularization Secondary Composite Endpoints: CHD Death, Non-Fatal MI, Ischemic Stroke, or Hospitalization for High-Risk ACS CHD Death, Non-Fatal MI or Ischemic Stroke Cardiovascular Mortality

Statistical Analyses Event-driven trial with projected 800 primary outcomes; 2.5-7 year follow-up (mean 4.6 years) 85% power to detect a 25% reduction in the 5- component primary endpoint (one-sided test of significance; alpha level=0.025) Pre-specified, conservative asymmetric boundaries for potential early stopping based on efficacy/lack of efficacy Trial stopped on 5/25/11: lack of efficacy and concern of ischemic stroke imbalance with niacin after a 36-month average follow-up

After the Trail Use Confidence Interval to Assess Power Original Power Calculation Offers No Additional Information

Primary Outcome 50 Combination Therapy 40 Monotherapy 30 HR 1.02, 95% CI 0.87, 1.21 Log-rank P value= 0.79 Cumulative % with Primary Outcome 16.4% 20 16.2% 10 1 2 3 4 N at risk Time (years) Monotherapy 1696 1581 1381 910 436 Combination Therapy 1718 1606 1366 903 428

Confidence Interval / Credible Interval 95% Confidence Interval: If the procedure for computing a 95% confidence interval is used over and over, 95% of the time the interval will contain the true parameter value. 95% Credible Interval: Given the assumptions of the prior distribution and results, there is a 95% probability that the true value lies within the interval

What About Additional Endpoints and Subgroups? Watch out for multiple comparisons Generally underpowered as power calculations are for the composite overall

Primary and Secondary Endpoints All Cardiovascular Death non-fatal MI or ischemic stroke Composite of CHD Death, hospitalization for high-risk ACS) (CHD death, non-fatal MI, ischemic stroke, Original Primary Endpoint or Cerebral Revascularization Symptom-Driven Coronary Hospitalization for ACS Ischemic Stroke Non-fatal MI CHD Death Primary Endpoint P=0.11 0.5 1 1.5 2 2.5 3 3.5 Niacin better Niacin worse

Pre-Specified Subgroups Overall Age ≥ 65 years Age < 65 years Men Women Diabetes No Diabetes Metabolic Syndrome No Metabolic Syndrome Prior MI No Prior MI ON Statin at Entry OFF Statin at Entry 0.5 1 1.5 2 Niacin better Niacin worse

Conclusions Underpowered studies are subject to β errors, failing to find a difference when there is one Small studies are also subject to α error, finding a difference when there isn’t one – sometimes called the “tyranny of small numbers” Power calculation is critical to design, but look at the data after the trial