1. PSY 1950 Post-hoc and Planned Comparisons October 6, 2008

2. Preamble Presentations Tutoring Problem 1e: If you decide to reject the null hypothesis, you know the probability that you are making the wrong decision Visual depiction of F-ratio

3. Subpopulations Cournot (1843): “...it is clear that nothing limits... the number of features according to which one can distribute [natural events or social facts] into several groups or distinct categories.” e.g., the chance of a male birth: –Legitimate vs. illegitimate –Birth order –Parent age –Parent profession –Parent health –Parent religion “… usually these attempts through which the experimenter passed don’t leave any traces; the public will only know the result that has been found worth pointing out; and as a consequence, someone unfamiliar with the attempts which have led to this result completely lacks a clear rule for deciding whether the result can or can not be attributed to chance.”

4. Large Surveys and Observational Studies Abundant data Limited a priori hypotheses e.g., Genome Superstruct Project (GSP) –Genetic testing –Cognitive testing –Structural brain imaging –Functional brain imaging

5. ANOVA One-way ANOVA –k(k-1)/2 possible pairwise comparisons –e.g., with 5 levels, 10 possible comparisons Factorial ANOVA –The issue above plus –Multiple possible main effects/interactions –e.g., with a 2 x 2 x 2, 7 possible effects

6. Families Set of hypotheses = Family Type I error rate for a set of hypotheses = Familywise error rate –e.g., across pairwise comparisons in one-way ANOVA If no mean differences exist, what is the chance of finding a significant one? –e.g., across main effects/interactions in factorial ANOVA If no main effects or interactions exist for a particular ANOVA, what is the chance of finding a significant one –e.g., whole experiment with multiple ANOVAs If no effects exist for the entire experiment, what is the chance of finding a significant one?

7. Family Size "If these inferences are unrelated in terms of their content or intended use (although they may be statistically dependent), then they should be treated separately and not jointly” –Hochberg and Tamhane (1987) e.g., suicide rates for 50 states, with 1225 possible pairwise comparisons –From a federal perspective, how big is the family? –How about from a state perspective?

8. Familywise  If family consists of two independent comparisons with  =.05, AND if both corresponding null hypotheses are true: –The probability of NOT making a Type I error on both tests is:.95 x.95 =.9025 –The probability of making one or more type I errors is: 1 -.9025 =.0975 If family consists of c independent comparisons with  =.05, AND if all corresponding null hypotheses are true: –The probability of NOT making a Type I error on all tests is: (1 -.05) c –The probability of making one or more Type I errors is: 1 - (1 -.05) c

9. A Priori vs. Post-hoc Comparisons A priori comparisons –Chosen before data collection –Limited, deliberate comparisons Post hoc (a posteriori) comparisons –Conducted after data collection –Exhaustive, exploratory comparisons

10. Significance of Overall F Prerequisite for some tests (e.g., Fisher’s LSD) Efficient test of overall null hypothesis Need MS within for many tests

11. A Priori Comparisons Single stage tests –Multiple t-tests –Linear contrasts –Bonferroni t (Dunn’s test) –Dunn-Sidak test Multistage tests –Bonferroni/Holm

12. Multiple t-tests Replace s 2 pooled with MS within Use df within

13. Linear Contrasts Compare more than one mean with another mean

15. Bonferroni t (Dunn’s Test) If c independent tests are performed  corrected =  / c p corrected = p x c Imprecise math –e.g., for p corrected =.05 with c = 21, p corrected  1.05 – p corrected = 1 - (1 -.05) c Bonferroni, C. E. (1936). Teoria statistica delle classi e calcolo delle probabilit. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8, 3-62. Perneger, T.V. (1998). What is wrong with Bonferroni adjustments. BMJ,136,1236-1238.

16. Dunn-Sidak Test Identical to Bonferroni, except uses correct math Less conservative than Bonferroni –e.g., for p corrected =.05 with c = 10: p Bonferroni =.50 p Sidak =.40

17. Multistage Bonferroni (e.g., Holm) Calculate t for all c contrasts of interest Order results based on |t| |t 1 | > |t 2 | > |t 3 | Apply different Bonferroni corrections for  or p based on position in above sequence, stopping when t is insignificant –For t 1, c 1 = 3; if p 1 >.05/3, then… –For t 2, c 2 = 2; if p 2 >.05/2, then… –For t 1, c 1 = 3; use  =.05/1

18. Post-hoc Comparisons Fisher’s LSD Tukey’s test Newman-Keuls test The Ryan procedure (REGWQ) Scheffe’s test Dunnett’s test

19. Fisher’s LSD Test LSD = Least significant difference Two-stage process: –Conduct ANOVA If F is nonsignificant, stop If F is significant… –Make pairwise comparisons using Ensures familywise  =.05 for complete null Ensures familywise  =.05 for partial null when c = 3

20. Studenized Range Statistic (q) If M l and M s represent the largest and smallest means and r is the number of means in the set: Order means from smallest to largest Determine r, calculate q, lookup p

21. Tukey’s HSD Test Determines minimum difference between treatment means that is necessary for significance HSD = honestly significant difference

22. Scheffe Not for post-hoc pairwise comparisons Not for a priori comparisons Howell: “I can’t imagine when I would ever use it, but I have to include it here because it is such a standard test”

23. Newman-Keuls (S-N-K) Test Readjusts r based upon means tests Doesn’t control for familywise  =.05

24. Comparing Different Procedures

25. Which Test? One contrast –Simple: t-test –Complex: linear contrast Several contrasts –A priori: Multistage Bonferroni (e.g., Holm) –Post-hoc: Fisher’s LSD Many contrasts –Ryan REGRQ or Tukey Find critical values for different tests –with a control: Dunnett –planned: Bonferroni –not planned: Scheffé

27. Imaging Data 200,000 tests on 200,000 voxels 1000 false positives when  =.05 Bonferroni? –No, requires voxel independence