Multiple Comparison Procedures Comfort Ratings of 13 Fabric Types A.V. Cardello, C. Winterhalter, and H.G. Schultz (2003). "Predicting the Handle and Comfort of Military Clothing Fabrics from Sensory and Instrumental Data: Development and Application of New Psychophysical Methods," Textile Research Journal, Vol. 73, pp

Treatments Means and standard deviations of 45 comfort ratings for 13 military fabrics. Fabric types: 10R - 50/50 Nylon/combed cotton, ripstop poplin weave 11A - 50/50 Nylon/Polyester, oxford weave (Australian) 12T - 50/50 Nylon/cotton, twill weave 13P - 92/5/3 Nomex/Kevlar/P140, plain weave 14N Cotton (former flame retardant treated) 15B - 77/23 Cotton sheath/synthetic core, twill (UK) 16C combed cotton, ripstop poplin (former hot weather BDU) 17C - 65/35 Wool/Polyester, plain weave (Canada-unlaundered) 18L - 65/35 Wool/Polyester, plain weave (Canada-laundered) 19N - 92/5/3 Nomex/Kevlar/P140, oxford weave 20J - Carded cotton sheath/nylon core, plain weave (Canada) Pima cotton ripstop poplin (experimental) /50 Nylon carded cotton ripstop poplin weave

Multiple Comparisons Individual & Combined Null Hypotheses (H 0  H 01 … H 0k ) Comparisonwise Error Rate  Pr(Reject H 0i |H 0i True) Experimentwise Error Rate  Pr(Reject any H 0i |All H 0i True) False Discovery Rate  E(# False Rejects/Total Rejections) Strong Familywise Error Rate  Pr(Any False Discoveries) Simultaneous Confidence Intervals  Pr(All Correct)=1-  Multiple Comparison Procedures control Type 1 Error Rate other than per comparison

Data and Analysis of Variance

Bonferroni Based Methods Construct P-values for all k test statistics Order P-values from smallest p (1) ≤ … ≤ p (k)  Bonferroni: Reject H 0(i) if p (i) ≤  k  Holm (Controls Strong FWER): Reject H 0(i) if p (j) ≤  k-j+1)  j ≤ i  False Discovery Rate: Reject H 0(i) if p (j) ≤ j  k for some j ≥ i (Assumes independent tests, not the case for this example) Example: Comparing all k=13(12)/2=78 pairs of fabrics

Fabric Example j=1,…,26

Fabric Example j=27,…,52

Fabric Example j=53,…,78

Scheffe’s Method for All Contrasts Can be used for any number of contrasts, even those suggested by data. Conservative (Wide CI’s, Low Power)

Example – Scheffe’s Method – All Pairwise Tests/CIs

Tukey’s Method for All Pairwise Comparisons Makes use of the Studentized Range Distribution Pr{(max(Y 1,…,Y n )-min(Y 1,…,Y n ))/S ≥ q( ,n, )} =   Y 1,…,Y n }  S  degrees of freedom for S

Tukey’s Method for All Pairwise Comparisons

Bonferroni’s Method for All Pairwise Comparisons Adjusts type I error rate for each test to  /(# of tests) Increases Confidence levels of CI’s to (1-(  /(# of CIs)))

Bonferroni’s Method for All Pairwise Comparisons

SNK Method for All Pairwise Comparisons Controls False Discovery Rate at  Uses Different Critical Values for different ranges of means

SNK Method for All Pairwise Comparisons

Duncan’s Method for All Pairwise Comparisons More powerful than SNK method, at cost of increasing  for longer stretches (does not control experimentwise error rate) Uses Different Critical Values for different ranges of means

Duncan’s Method for All Pairwise Comparisons

Fisher’s Protected LSD for All Pairwise Comparisons Controls Experimentwise Error Rate at  Only Conducted if F-test is significant (P-value ≤ 

Fisher’s Protected LSD for All Pairwise Comparisons

Multiple Comparisons with Best Treatment/Control Pr{subset of treatments contains the best} = 1- 

Multiple Comparisons with Best Treatment/Control Treatments 13P, 15B, and 11A all lie within of the highest mean