1. Three Statistical Issues (1) Observational Study (2) Multiple Comparisons (3) Censoring Definitions

2. Non-Randomized Study Benefit of Randomized Study: Characteristics that may be associated with outcome tend to be balanced across treatment groups Limitation of Observational Study: Characteristics that may be associated with outcomes may not be balanced. –Example: “Sicker” patients could tend to be assigned to one of the treatments more often than the others. –As a result, crucial to compare the groups as thoroughly as possible with respect to covariates. –Where balance does not exist, analyses should be adjusted for imbalances.

3. Examples from Paper

4. How to adjust? Regression Methods: –Cox proportional hazards model (survival analysis) –Linear regression (continuous outcomes) –Logistic regression (binary outcomes, e.g. response rate) –Other regression methods (“Generalized Linear Models”)

5. Multiple Comparisons Not just comparing treatment 1 versus treatment 2. Three comparisons: –treatment 1 vs 2 –treatment 1 vs 3 –treatment 2 vs 3 Recall type 1 and type 2 errors…..

6. Type 1 Error The probability of concluding there is a difference in treatments when there truly is NO difference. Also called the “alpha” level of the test. Determines what is a “significant” p-value. Generally set to 0.05 So, 5% of the time we will say there IS a difference when there really is not.

7. Type II error The probability of concluding there is NO difference in treatments when there truly is a difference. Also, called the “beta” level Also, Power = 1 - beta (so power is concluding that there is a difference when there truly is a difference) Generally, we like to have beta is less than 20%. So, 20% of the time we will say there is no difference when there really is a difference

8. Back to Type 1 Error If we are making one comparison, we have a 5% chance of making the error of concluding that there is a difference when there really is no difference. What if we have three comparisons? 3x5% = 15% chance of finding some difference between treatments if no treatment differences exist.

9. Solutions Some say…. –Note to “take into consideration when drawing conclusions” –Easy way to handle it Other places (e.g. Harvard)…. –Bonferroni and other “corrections” –Essentially dividing p-value among tests so that the sum of the type I errors is 0.05 –Example: For three comparisons, allow Type 1 error for each comparison of 5/3 = 1.33%. So only conclude that there is a significant difference between two treatments if pvalue is less than 0.013.

10. Another Solution: ANOVA type tests: –Ho: no differences between treatments –Ha: at least one treatment is different than some other –Get just one pvalue. –Tells you that there is some difference, but no information about which is different, or how many are different.

11. Example var1 var2 var3 var4 var5 1. 5 6 7 8 9 2. 6 7 8 9 10 3. 7 8 9 10 11 4. 8 9 10 11 12 5. 9 10 11 12 13 6. 10 11 12 13 14 anova var group Number of obs = 30 R-squared = 0.4068 Root MSE = 1.87083 Adj R-squared = 0.3119 Source | Partial SS df MS F Prob > F -----------+---------------------------------------------------- Model | 60.00 4 15.00 4.29 0.0089 | group | 60.00 4 15.00 4.29 0.0089 | Residual | 87.50 25 3.50 -----------+---------------------------------------------------- Total | 147.50 29 5.0862069 pvalue

12. Pvalues for Individual T-tests: 10 comparisons