Tim Wiemken, PhD MPH CIC Assistant Professor Division of Infectious Diseases University of Louisville, Kentucky Planning Your Study Statistical Issues

Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

Video

Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

Types of Analysis DescriptiveDescribes a sample

Types of Analysis Descriptive

Types of Analysis Inferential/An alytical Analyzes associations, predictions, differences

Types of Analysis Inferential/An alytical STATISTICAL SIGNIFICANCE

Types of Analysis Inferential/An alytical

Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

Hypothesis Testing It is very hard to prove something causes something else

Hypothesis Testing 2.Hypothesis testing for inferential statistics It is much easier to disprove that something doesn’t cause something else.

Types of Analysis 2.Hypothesis testing for inferential statistics EXAMPLE Innocent until proven guilty (misnomer)

Hypothesis Testing 2.Hypothesis testing for inferential statistics EXAMPLE Innocent until proven guilty (misnomer) Should be: Innocent until proven not innocent

Hypothesis Testing 2.Hypothesis testing for inferential statistics a. Assume innocence.

Hypothesis Testing 2.Hypothesis testing for inferential statistics a. Assume innocence. b. Disprove innocence beyond a reasonable doubt.

Hypothesis Testing 2.Hypothesis testing for inferential statistics a. Assume innocence. b. Disprove innocence beyond a reasonable doubt. c. The conclusion you are left with is guilty.

Hypothesis Testing 2.Hypothesis testing for inferential statistics Hypothesis tests work the same way

Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis

Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis Not guilty Guilty

Hypothesis Testing 2.Hypothesis testing for inferential statistics Does azithromycin cause cardiac complications?

Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis

Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death

Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis Hard to prove Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death

Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis Easy to to disprove Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death

Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis If disproven, only option is to accept the alternative! Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death

Hypothesis Testing Disproving (rejecting) a null hypothesis is done using statistical tests. Bivariate Tests (univariate) Chi-squared Student’s t-test Fisher’s Exact test Mann-Whitney U-test ANOVA

Hypothesis Testing Disproving (rejecting) a null hypothesis is done using statistical tests. Bivariate Tests (univariate) Chi-squared Student’s t-test Fisher’s Exact test Mann-Whitney U-test ANOVA Multivariate Tests (regression) Linear regression Logistic regression Cox Regression Poisson Regression

Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

Error Reject Null Do not reject Null TrueFalse Null Hypothesis (reality) StudyStudy Correct

Error Do not reject the null when it is indeed true Null hypothesis Azithromycin is not associated with cardiac death This is the truth Correct!

Error Reject Null Do not reject Null TrueFalse StudyStudy Correct Null Hypothesis (reality)

Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Reject the null when it is false Reject! This is the truth Correct!

Error Reject Null Do not reject Null TrueFalse StudyStudy Correct Error I (α) Null Hypothesis (reality)

Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Reject the null when it is true = incorrectly conclude there is an association Reject incorrectly! This is the truth Incorrect!

Error Reject Null Do not reject Null TrueFalse Null Hypothesis StudyStudy Correct Error I (α) Error II (β)

Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Do not reject the null when it is false= incorrectly conclude there is no association This is the truth Incorrect! Do not reject

Error What you need to know α= level of significance Pretty much always 0.05 or 5%

Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Reject the null when it is true = incorrectly conclude there is an association Reject incorrectly! This is the truth Incorrect!

Error Reject Null Do not reject Null TrueFalse StudyStudy Correct Error I (α) Null Hypothesis (reality) Willing to accept a 5% chance of rejecting the null when it is true

Error Reject Null Do not reject Null TrueFalse StudyStudy Correct Error I (α) Null Hypothesis (reality) 1 in 20 studies (5%) will lead to a false conclusion that there is an association when there actually is not

Error What you need to know β Type II error – usually 0.2 or 20%

Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Do not reject the null when it is false= incorrectly conclude there is no association This is the truth Incorrect! Do not reject

Error Reject Null Do not reject Null TrueFalse Null Hypothesis StudyStudy Correct Error I (α) Error II (β) Willing to accept a 20% chance of not rejecting the null when it is false

Error Reject Null Do not reject Null TrueFalse Null Hypothesis StudyStudy Correct Error I (α) Error II (β) 1 in 5 studies (20%) will lead to a false conclusion that there is not an association when there actually is

Power What you need to know β More importantly, 1-β = statistical power

Power What you need to know 1-β Power = the ability to detect an association when one is present

Power Reject Null Do not reject Null TrueFalse Null Hypothesis StudyStudy Correct Correct (Power) Error I (α) Error II (β) 4 in 5 studies (80% Power) will detect an association when one is really present

Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

Effect Size Effect Size = Smallest difference between your study groups worth saying is statistically significant.

Effect Size 1.Effect size is critical to be able to properly calculate the sample size.

Effect Size 2.Utilize data from pilot studies or literature to determine an effect size

Effect Size Usually takes the format: –20% mortality in group 1 –10% mortality in group 2 10% Effect Size (20%-10%)

Effect Size Usually takes the format: –Average blood glucose in group 1: 200mg/dL –Average blood glucose in group 2: 150mg/dL Effect Size = 50mg/dL

Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

Example 1 Design: Unblinded randomized controlled trial

Example 1 Design: Unblinded randomized controlled trial Population: Stool transplantation versus Oral Vancomycin for C. difficile infection

Question: Is stool transplantation more effective than oral vancomycin at preventing C. difficile infection relapse? Example 1 Design: Unblinded randomized controlled trial Population: Stool transplantation versus Oral Vancomycin for C. difficile infection

From the literature: Stool transplantation will reduce relapse from X% (oral vancomycin alone) to Y% (stool transplantation alone). Example 1

Null hypothesis: ?? Example 1 Alternative hypothesis: ??

Null hypothesis: Stool transplant is not associated with less C. difficile relapse than oral vancomycin. Example 1 Alternative hypothesis: Stool transplants are associated with less C. difficile relapse than oral vancomycin.

Example 1 Sample Size: How many patients do you need?

What you need to know: What percent of relapse do you expect in each group? Example 1 Sample Size: How many patients do you need?

What you need to know: What percent of relapse do you expect in each group? Example 1 Sample Size: How many patients do you need? Pilot study or literature review (or guess)!

Example 1 Back to our example. Are we testing the difference between proportions or means?

Example 1 Back to our example. Are we testing the difference between proportions or means?

Example 1 How many patients do you need? Left side: 20% (standard of care) 10% (treatment group) Right side: 20% (standard of care) 15% (treatment group)

Example 1 How many patients do you need? Left side: 20% (standard of care) 10% (treatment group) 199 per group Total = 398 Right side: 20% (standard of care) 15% (treatment group) 905 per group Total = 1809

Example 1 If we are “powered” to detect a 10% difference 199 per group Total = 398 If there is actually a 5% difference, we won’t have enough power to detect the difference = lack of significance may be type II error

Example 2 Hypothesis: Patients who come to clinic at least 3 times a year will have an increase in CD4+ cell count. The average CD4 will be X if they do come to clinic and Y if they do not.

What you need to know: What is the average CD4 count you expect in each group? Example 2 Sample Size: How many patients do you need?

What you need to know: What is the average CD4 count you expect in each group? Example 2 Sample Size: How many patients do you need? Left side: 750 (≥3 visits/year) 500 (<3 visits/year) Right side: 750 (≥3 visits/year) 650 (<3 visits/year)

What you need to know: What is the standard deviation for both groups? Example 2 Sample Size: How many patients do you need? Left side: 200 Right side: 200

Example 2 Sample Size: How many patients do you need? Left side: Total: 21 Right side: Total: 126

Example 2 We have seen what mean alterations do to the sample size. Left side: Total: 21 Right side: Total: 126

Example 2 What happens if we change the standard deviation? Use 300 instead of 200

Example 2 What happens if we change the standard deviation? Use 300 instead of 200 Left side: Total: 46 (from 21) Right side: Total: 283 (from 126)

Example 2 What happens if we change the standard deviation? Use 300 instead of 200 Use 100 instead of 200

Example 2 What happens if we change the standard deviation? Use 300 instead of 200 Use 100 instead of 200 Left side: Total: 6 (from 21) Right side: Total: 32 (from 126)

At the end of the day… FOR AN UNDERPOWERED STUDY You don’t know if there is not an association or if your non-significant findings are due to your small sample size. Under- powered I’m not significant!

At the end of the day… Studies can be OVERPOWERED Any difference can be found to be statistically significant. Over- powered I’m ALWAYS significant!

At the end of the day… Studies can be OVERPOWERED There is a difference between statistically significant and clinically significant.

Conclusion Sample size estimations are important

Conclusion Obtaining appropriate estimates of proportions or means/standard deviations is CRITICAL

Conclusion Improper estimation of these values will lead to error

Conclusion Call your statistician early in the planning process.

Conclusion Remember your statistician is not a magician. He or she will give you a sample size based the values you provide.

Questions?