Comparing Three or More Groups: Multiple Comparisons vs Planned Comparisons Robert Boudreau, PhD Co-Director of Methodology Core PITT-Multidisciplinary Clinical Research Center for Rheumatic and Musculoskeletal Diseases

First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95  Prob [ H ≤ 40 ] + P[ H ≥ 60] =

First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95  Prob [ H ≤ 40 ] + P[ H ≥ 60] = Experiment: 20 people flip their own coin 100 times Q: Approx how many will get 40 or fewer heads or 60+ heads? or 60+ heads?

First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95  Prob [ H ≤ 40 ] + P[ H ≥ 60] = Experiment: 20 people flip their own coin 100 times Q: Approx how many will get less than 40 heads or 60+ heads? Answer: One or 60+ heads? Answer: One

First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95  Prob [ H ≤ 40 ] + P[ H ≥ 60] = Experiment: 20 people flip their own coin 100 times Q: Approx how many will get less than 40 heads or 60+ heads? Answer: One (1/20 = 5%) or 60+ heads? Answer: One (1/20 = 5%)

First a simple thought experiment Experiment: 20 people flip their own coin 100 times One (1/20=0.05) will flip an unusually small or unusually large # heads (on average) One (1/20=0.05) will flip an unusually small or unusually large # heads (on average) Q: Can we conclude that this person “X” flips an “unfair” coin, or was this explainable by “chance”?

Controlling Experiment-wise Error Experiment: 20 people flip their own coin 100 times  Person X’s confidence interval didn’t cover 0.5 Q: What alpha level should be used so that 95% of the time all 20 confidence intervals each cover 0.5? (i.e. so that the correct conclusion is drawn about (i.e. so that the correct conclusion is drawn about every single coin) every single coin)

Controlling Experiment-wise Error Experiment: 20 people flip their own coin 100 times  Person X’s confidence interval didn’t cover 0.5 Q: What alpha level should be used so that 95% of the time all 20 confidence intervals each cover 0.5? (i.e. so that the correct conclusion is drawn about (i.e. so that the correct conclusion is drawn about every single coin) every single coin)  Equivalent to drawing a “wrong” conclusion about at least one of the coins only 5% of the time (Experiment-wise Type I error)

Controlling Experiment-wise Error Q: What alpha level should be used so that there’s a 95% probability that all 20 confidence intervals each cover 0.5? (aka Experiment-wise correct conclusion) Experiment-wise α=0.05, solve for comparison-wise α*: α = Prob[ At least one C.I. misses 0 ] α = Prob[ At least one C.I. misses 0 ] = 1 – Prob[ All C.I.’s cover 0 ] = 1 – Prob[ All C.I.’s cover 0 ] = 1 – (1 – α* ) 20 = 1 – (1 – α* ) 20 Sidak: Comparison-wise α* = 1 – (1 – α) 1/n n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = n=20 “comparisons”: α* = 1 – (1-.05) 1/20 =

Controlling Experiment-wise Error Q: What alpha level should be used so that there’s a 95% probability that all 20 confidence intervals each cover 0.5? Sidak: Comparison-wise α* = 1 – (1 – α) 1/n n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = Bonferroni: α* = α/n ( 0.05/20=0.0025)

Controlling Experiment-wise Error Mathematically: α/n < 1 – (1 – α) 1/n Mathematically: α/n < 1 – (1 – α) 1/n Bonferroni < Sidak (i.e. higher α-level) But usually very close  Sidak slightly more powerful Bonferroni works in all situations to guarantee control of experimentwise error (but may be conservative) Bonferroni works in all situations to guarantee control of experimentwise error (but may be conservative) Sidak (derived assuming independence) can under- control in presence of high correlations Sidak (derived assuming independence) can under- control in presence of high correlations

Comparison of Adverse Effect of 4 Drugs on Systolic BP

Unadjusted pairwise t-tests (α = 0.05 each comparison) critical value of t=

Pairwise t-tests (Bonferroni) critical value of t= Pairwise t-tests (Bonferroni) critical value of t=

Pairwise t-tests (Sidak) critical value of t= Pairwise t-tests (Sidak) critical value of t=

Comparison of critical values Scheffe: * Designed for arbitrary post-hoc testing * Controls experimentwise error for all possible simultaneous comparisons and contrasts

Comparison of Adverse Effect of 4 Drugs on Systolic BP (v2) Note: For Drug 4, I’ve subtracted 6 from the previous values s s

Comparison of Adverse Effect of 4 Drugs on Systolic BP (v2) ANOVA F-test

Unadjusted pairwise t-tests (v2) (α = 0.05 each comparison) critical value of t=

Pairwise t-tests (Bonferroni) (v2) critical value of t= Pairwise t-tests (Bonferroni) (v2) critical value of t=

Pairwise t-tests (Sidak) (v2) critical value of t= Pairwise t-tests (Sidak) (v2) critical value of t=

Tukey’s Studentized Range Test Related in concept to Scheffe’s Method Related in concept to Scheffe’s Method Designed for all pairwise comparisons exclusively Designed for all pairwise comparisons exclusively (recall: Scheffe applies to all possible simultaneous pairwise comparisons and contrasts) pairwise comparisons and contrasts) Exact experimentwise error coverage if sample sizes equal Exact experimentwise error coverage if sample sizes equal Critical values smaller than Bonferroni or Sidak Critical values smaller than Bonferroni or Sidak  More powerful in finding differences

Pairwise t-tests (Tukey) (v2) critical value of t= Pairwise t-tests (Tukey) (v2) critical value of t=

Comparison of Adverse Effect of 4 Drugs on Systolic BP

Dunnett’s Method (Comparison vs a Control) Related in concept to Scheffe and Tukey Methods Related in concept to Scheffe and Tukey Methods Designed for pairwise comparisons vs a single control exclusively Designed for pairwise comparisons vs a single control exclusively Exact experimentwise error coverage of those comparisons if sample sizes equal Exact experimentwise error coverage of those comparisons if sample sizes equal Critical values smaller than Bonferroni, Sidak or Tukey Critical values smaller than Bonferroni, Sidak or Tukey  More powerful in finding differences vs control

Comparison vs Control (Dunnett) (v2) critical value of t= Comparison vs Control (Dunnett) (v2) critical value of t=

Controlling for Multiple Comparisons in Exploratory Analyses Caterina Rosano, Howard J. Aizenstein, Stephanie Studenski, Anne B. Newman. Caterina Rosano, Howard J. Aizenstein, Stephanie Studenski, Anne B. Newman. A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007 A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007

Controlling for Multiple Comparisons in Exploratory Analyses A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007 A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007

Controlling for Multiple Comparisons in Exploratory Analyses

c

Thank you ! Any Questions? Robert Boudreau, PhD Co-Director of Methodology Core PITT-Multidisciplinary Clinical Research Center for Rheumatic and Musculoskeletal Diseases