1. Comparing Three or More Groups: Multiple Comparisons vs Planned Comparisons Robert Boudreau, PhD Co-Director of Methodology Core PITT-Multidisciplinary Clinical Research Center for Rheumatic and Musculoskeletal Diseases

2. First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95  Prob [ H ≤ 40 ] + P[ H ≥ 60] = 0.05 -------------------------------------------------------------------------------------------------------

3. First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95  Prob [ H ≤ 40 ] + P[ H ≥ 60] = 0.05 ------------------------------------------------------------------------------------------------------- Experiment: 20 people flip their own coin 100 times Q: Approx how many will get 40 or fewer heads or 60+ heads? or 60+ heads?

4. First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95  Prob [ H ≤ 40 ] + P[ H ≥ 60] = 0.05 ------------------------------------------------------------------------------------------------------- Experiment: 20 people flip their own coin 100 times Q: Approx how many will get less than 40 heads or 60+ heads? Answer: One or 60+ heads? Answer: One

5. First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95  Prob [ H ≤ 40 ] + P[ H ≥ 60] = 0.05 ------------------------------------------------------------------------------------------------------- Experiment: 20 people flip their own coin 100 times Q: Approx how many will get less than 40 heads or 60+ heads? Answer: One (1/20 = 5%) or 60+ heads? Answer: One (1/20 = 5%)

6. First a simple thought experiment Experiment: 20 people flip their own coin 100 times One (1/20=0.05) will flip an unusually small or unusually large # heads (on average) One (1/20=0.05) will flip an unusually small or unusually large # heads (on average) Q: Can we conclude that this person “X” flips an “unfair” coin, or was this explainable by “chance”?

7. Controlling Experiment-wise Error Experiment: 20 people flip their own coin 100 times  Person X’s confidence interval didn’t cover 0.5 Q: What alpha level should be used so that 95% of the time all 20 confidence intervals each cover 0.5? (i.e. so that the correct conclusion is drawn about (i.e. so that the correct conclusion is drawn about every single coin) every single coin)

8. Controlling Experiment-wise Error Experiment: 20 people flip their own coin 100 times  Person X’s confidence interval didn’t cover 0.5 Q: What alpha level should be used so that 95% of the time all 20 confidence intervals each cover 0.5? (i.e. so that the correct conclusion is drawn about (i.e. so that the correct conclusion is drawn about every single coin) every single coin)  Equivalent to drawing a “wrong” conclusion about at least one of the coins only 5% of the time (Experiment-wise Type I error)

9. Controlling Experiment-wise Error Q: What alpha level should be used so that there’s a 95% probability that all 20 confidence intervals each cover 0.5? (aka Experiment-wise correct conclusion) Experiment-wise α=0.05, solve for comparison-wise α*: α = Prob[ At least one C.I. misses 0 ] α = Prob[ At least one C.I. misses 0 ] = 1 – Prob[ All C.I.’s cover 0 ] = 1 – Prob[ All C.I.’s cover 0 ] = 1 – (1 – α* ) 20 = 1 – (1 – α* ) 20 Sidak: Comparison-wise α* = 1 – (1 – α) 1/n n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = 0.00256 n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = 0.00256

10. Controlling Experiment-wise Error Q: What alpha level should be used so that there’s a 95% probability that all 20 confidence intervals each cover 0.5? Sidak: Comparison-wise α* = 1 – (1 – α) 1/n n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = 0.00256 n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = 0.00256 Bonferroni: α* = α/n ( 0.05/20=0.0025)

11. Controlling Experiment-wise Error Mathematically: α/n < 1 – (1 – α) 1/n Mathematically: α/n < 1 – (1 – α) 1/n Bonferroni < Sidak (i.e. higher α-level) But usually very close  Sidak slightly more powerful Bonferroni works in all situations to guarantee control of experimentwise error (but may be conservative) Bonferroni works in all situations to guarantee control of experimentwise error (but may be conservative) Sidak (derived assuming independence) can under- control in presence of high correlations Sidak (derived assuming independence) can under- control in presence of high correlations

12. Comparison of Adverse Effect of 4 Drugs on Systolic BP

15. Unadjusted pairwise t-tests (α = 0.05 each comparison) critical value of t=2.13145

16. Pairwise t-tests (Bonferroni) critical value of t=3.03628 Pairwise t-tests (Bonferroni) critical value of t=3.03628

17. Pairwise t-tests (Sidak) critical value of t=3.02585 Pairwise t-tests (Sidak) critical value of t=3.02585

18. Comparison of critical values Scheffe: * Designed for arbitrary post-hoc testing * Controls experimentwise error for all possible simultaneous comparisons and contrasts

19. Comparison of Adverse Effect of 4 Drugs on Systolic BP (v2) Note: For Drug 4, I’ve subtracted 6 from the previous values s s

20. Comparison of Adverse Effect of 4 Drugs on Systolic BP (v2) ANOVA F-test

21. Unadjusted pairwise t-tests (v2) (α = 0.05 each comparison) critical value of t=2.13145

22. Pairwise t-tests (Bonferroni) (v2) critical value of t=3.03628 Pairwise t-tests (Bonferroni) (v2) critical value of t=3.03628

23. Pairwise t-tests (Sidak) (v2) critical value of t=3.02585 Pairwise t-tests (Sidak) (v2) critical value of t=3.02585

24. Tukey’s Studentized Range Test Related in concept to Scheffe’s Method Related in concept to Scheffe’s Method Designed for all pairwise comparisons exclusively Designed for all pairwise comparisons exclusively (recall: Scheffe applies to all possible simultaneous pairwise comparisons and contrasts) pairwise comparisons and contrasts) Exact experimentwise error coverage if sample sizes equal Exact experimentwise error coverage if sample sizes equal Critical values smaller than Bonferroni or Sidak Critical values smaller than Bonferroni or Sidak  More powerful in finding differences

25. Pairwise t-tests (Tukey) (v2) critical value of t=2.88215 Pairwise t-tests (Tukey) (v2) critical value of t=2.88215

26. Comparison of Adverse Effect of 4 Drugs on Systolic BP

27. Dunnett’s Method (Comparison vs a Control) Related in concept to Scheffe and Tukey Methods Related in concept to Scheffe and Tukey Methods Designed for pairwise comparisons vs a single control exclusively Designed for pairwise comparisons vs a single control exclusively Exact experimentwise error coverage of those comparisons if sample sizes equal Exact experimentwise error coverage of those comparisons if sample sizes equal Critical values smaller than Bonferroni, Sidak or Tukey Critical values smaller than Bonferroni, Sidak or Tukey  More powerful in finding differences vs control

28. Comparison vs Control (Dunnett) (v2) critical value of t=2.61702 Comparison vs Control (Dunnett) (v2) critical value of t=2.61702

29. Controlling for Multiple Comparisons in Exploratory Analyses Caterina Rosano, Howard J. Aizenstein, Stephanie Studenski, Anne B. Newman. Caterina Rosano, Howard J. Aizenstein, Stephanie Studenski, Anne B. Newman. A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007 A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007

30. Controlling for Multiple Comparisons in Exploratory Analyses A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007 A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007

31. Controlling for Multiple Comparisons in Exploratory Analyses

32. c

33. Thank you ! Any Questions? Robert Boudreau, PhD Co-Director of Methodology Core PITT-Multidisciplinary Clinical Research Center for Rheumatic and Musculoskeletal Diseases