1. Multiple Comparison Procedures
Comfort Ratings of 13 Fabric Types
A.V. Cardello, C. Winterhalter, and H.G. Schultz (2003). "Predicting the Handle and Comfort of Military Clothing Fabrics from Sensory and Instrumental Data: Development and Application of New Psychophysical Methods," Textile Research Journal, Vol. 73, pp

2. Treatments
Means and standard deviations of 45 comfort ratings for 13 military fabrics. Fabric types:
10R - 50/50 Nylon/combed cotton, ripstop poplin weave
11A - 50/50 Nylon/Polyester, oxford weave (Australian)
12T - 50/50 Nylon/cotton, twill weave
13P - 92/5/3 Nomex/Kevlar/P140, plain weave
14N Cotton (former flame retardant treated)
15B - 77/23 Cotton sheath/synthetic core, twill (UK)
16C combed cotton, ripstop poplin (former hot weather BDU)
17C - 65/35 Wool/Polyester, plain weave (Canada-unlaundered)
18L - 65/35 Wool/Polyester, plain weave (Canada-laundered)
19N - 92/5/3 Nomex/Kevlar/P140, oxford weave
20J - Carded cotton sheath/nylon core, plain weave (Canada)
Pima cotton ripstop poplin (experimental)
/50 Nylon carded cotton ripstop poplin weave

3. Multiple Comparisons
Individual & Combined Null Hypotheses (H0  H01… H0k)
Comparisonwise Error Rate  Pr(Reject H0i|H0i True)
Experimentwise Error Rate  Pr(Reject any H0i|All H0i True)
False Discovery Rate  E(# False Rejects/Total Rejections)
Strong Familywise Error Rate  Pr(Any False Discoveries)
Simultaneous Confidence Intervals  Pr(All Correct)=1-e
Multiple Comparison Procedures control Type 1 Error Rate other than per comparison

4. Data and Analysis of Variance

5. Bonferroni Based Methods
Construct P-values for all k test statistics
Order P-values from smallest p(1) ≤ … ≤ p(k)
Bonferroni: Reject H0(i) if p(i) ≤ e/k
Holm (Controls Strong FWER): Reject H0(i) if p(j) ≤ e/(k-j+1)  j ≤ I
False Discovery Rate: Reject H0(i) if p(j) ≤ je/k for some j ≥ i
(Assumes independent tests, not the case for this example)
Example: Comparing all k=13(12)/2=78 pairs of fabrics

6. Fabric Example j=1,…,26

7. Fabric Example j=27,…,52

8. Fabric Example j=53,…,78

9. Scheffe’s Method for All Contrasts
Can be used for any number of contrasts, even those suggested by data. Conservative (Wide CI’s, Low Power)

10. Example – Scheffe’s Method – All Pairwise Tests/CIs

11. Tukey’s Method for All Pairwise Comparisons
Makes use of the Studentized Range Distribution
Pr{(max(Y1,…,Yn)-min(Y1,…,Yn))/S ≥ q(a,n,n)} = a
{Y1,…,Yn }  S n  degrees of freedom for S

12. Tukey’s Method for All Pairwise Comparisons

13. Bonferroni’s Method for All Pairwise Comparisons
Adjusts type I error rate for each test to e/(# of tests)
Increases Confidence levels of CI’s to (1-(e/(# of CIs)))

14. Bonferroni’s Method for All Pairwise Comparisons

15. SNK Method for All Pairwise Comparisons
Controls False Discovery Rate at e
Uses Different Critical Values for different ranges of means

16. SNK Method for All Pairwise Comparisons

17. Fisher’s Protected LSD for All Pairwise Comparisons
Controls Experimentwise Error Rate at e
Only Conducted if F-test is significant (P-value ≤ e)

18. Fisher’s Protected LSD for All Pairwise Comparisons

19. Multiple Comparisons with Best Treatment/Control
Pr{subset of treatments contains the best} = 1- e

20. Multiple Comparisons with Best Treatment/Control
Treatments 13P, 15B, and 11A all lie within of the highest mean