1. Everyday is a new beginning in life. Every moment is a time for self vigilance.

2. Multiple Comparisons Error rate of control Pairwise comparisons Comparisons to a control Linear contrasts

3. Multiple Comparison Procedures Once we reject H 0 :   =   =...  c in favor of H 1 : NOT all  ’s are equal, we don’t yet know the way in which they’re not all equal, but simply that they’re not all the same. If there are 4 columns, are all 4  ’s different? Are 3 the same and one different? If so, which one? etc.

4. These “more detailed” inquiries into the process are called MULTIPLE COMPARISON PROCEDURES. Errors (Type I): We set up “  ” as the significance level for a hypothesis test. Suppose we test 3 independent hypotheses, each at  =.05; each test has type I error (rej H 0 when it’s true) of.05. However, P(at least one type I error in the 3 tests) = 1-P( accept all ) = 1 - (.95) 3 .14 3, given true

5. In other words, Probability is.14 that at least one type one error is made. For 5 tests, prob =.23. Question - Should we choose  =.05, and suffer (for 5 tests) a.23 Experimentwise Error rate (“a” or  E )? OR Should we choose/control the overall error rate, “a”, to be.05, and find the individual test  by 1 - (1-  ) 5 =.05, (which gives us  =.011)?

6. The formula 1 - (1-  ) 5 =.05 would be valid only if the tests are independent; often they’re not. [ e.g.,  1 =  2  2 =  3,  1 =  3 IF accepted & rejected, isn’t it more likely that rejected? ] 1 2 21 3 3

7. When the tests are not independent, it’s usually very difficult to arrive at the correct  for an individual test so that a specified value results for the experimentwise error rate (or called family error rate). Error Rates

8. There are many multiple comparison procedures. We’ll cover only a few. Pairwise Comparisons Method 1: (Fisher Test) Do a series of pairwise t-tests, each with specified  value (for individual test). This is called “Fisher’s LEAST SIGNIFICANT DIFFERENCE” (LSD).

9. Example: Broker Study A financial firm would like to determine if brokers they use to execute trades differ with respect to their ability to provide a stock purchase for the firm at a low buying price per share. To measure cost, an index, Y, is used. Y=1000(A-P)/A where P=per share price paid for the stock; A=average of high price and low price per share, for the day. “The higher Y is the better the trade is.”

10. } 1 12 3 5 12 5 6 2 7 17 13 11 7 17 12 3 8 1 7 4 3 7 5 24 13 14 18 14 19 17 R=6 CoL: broker 4 21 10 15 12 20 6 14 Five brokers were in the study and six trades were randomly assigned to each broker.

11.  =.05, F TV = 2.76 (reject equal column MEANS) “MSW”

12. 0 For any comparison of 2 columns,  /2 CLCL CuCu Y i -Y j AR: 0 + t  /2 x  MSW x  1 + 1 nini njnj df w (n i = n j = 6, here) MSW : Pooled Variance, the estimate for the common variance

13. In our example, with  =.05 0  2.060 (  21.2 x  1 + 1 ) 0  5.48 66 This value, 5.48 is called the Least Significant Difference (LSD). When same number of data points, R, in each column, LSD = t  /2 x  2xMSW. R

14. Col: 3 1 2 4 5 5 6 12 14 17 Summarize the comparison results. (p. 443) 1. Now, rank order and compare: Underline Diagram

15. Step 2: identify difference > 5.48, and mark accordingly: 5 6 12 14 17 3 1 2 4 5 3: compare the pair of means within each subset: Comparison difference vs. LSD 3 vs. 1 2 vs. 4 2 vs. 5 4 vs. 5 ****** <<<<<<<< * Contiguous; no need to detail 5

16. Conclusion : 3, 1 2, 4, 5 Can get “inconsistency”: Suppose col 5 were 18: 3 1 2 4 5 5 6 12 14 18 Now: Comparison |difference| vs. LSD 3 vs. 1 2 vs. 4 2 vs. 5 4 vs. 5 * <<><<<>< Conclusion : 3, 1 2 4 5 ??? 6

17. Broker 1 and 3 are not significantly different but they are significantly different to the other 3 brokers. Conclusion : 3, 1 2 4 5 Broker 2 and 4 are not significantly different, and broker 4 and 5 are not significantly different, but broker 2 is different to (smaller than) broker 5 significantly.

19. Fisher's pairwise comparisons (Minitab) Family error rate = 0.268 Individual error rate = 0.0500 Critical value = 2.060  t _  /2 (not given in version 16.1) Intervals for (column level mean) - (row level mean) 1 2 3 4 2 -11.476 -0.524 3 -4.476 1.524 6.476 12.476 4 -13.476 -7.476 -14.476 -2.524 3.476 -3.524 5 -16.476 -10.476 -17.476 -8.476 -5.524 0.476 -6.524 2.476 Minitab: Stat>>ANOVA>>One-Way Anova then click “comparisons”. Col 1 < Col 2 Col 2 = Col 4

20. Minitab Output for Broker Data Grouping Information Using Fisher Method broker N Mean Grouping 5 6 17.000 A 4 6 14.000 A 2 6 12.000 A 1 6 6.000 B 3 6 5.000 B Means that do not share a letter are significantly different.

21. Pairwise comparisons Method 2: (Tukey Test) A procedure which controls the experimentwise error rate is “TUKEY’S HONESTLY SIGNIFICANT DIFFERENCE TEST ”.

22. Tukey’s method works in a similar way to Fisher’s LSD, except that the “LSD” counterpart (“HSD”) is not t  /2 x  MSW x  1 + 1 nini njnj t  /2 x  2xMSW R = or, for equal number of data points/col ( ), but tuk X  2xMSW, R where t uk has been computed to take into account all the inter-dependencies of the different comparisons.  /2

23. HSD = t uk  /2 x  2MSW R _______________________________________ A more general approach is to write HSD = q  x  MSW R where q  = t uk  /2 x  2 --- q = (Y largest - Y smallest ) /  MSW R ---- probability distribution of q is called the “Studentized Range Distribution”. --- q = q(c, df), where c =number of columns, and df = df of MSW

24. With c = 5 and df = 25, from table (or Minitab): q = 4.15 t uk = 4.15/1.414 = 2.93 Then, HSD = 4.15   also    x 

25. In our earlier example: Rank order: 3 1 2 4 5 5 6 12 14 17 (No differences [contiguous] > 7.80)

26. Comparison |difference| >or< 7.80 3 vs. 1 3 vs. 2 3 vs. 4 3 vs. 5 1 vs. 2 1 vs. 4 1 vs. 5 2 vs. 4 2 vs. 5 4 vs. 5 * <<>><>><<<<<>><>><<< 9 12 * 8 11 * 5 * (contiguous) 7 3, 1, 24, 5 2 is “same as 1 and 3, but also same as 4 and 5.”

27. Tukey's pairwise comparisons (Minitab) Family error rate = 0.0500 Individual error rate = 0.00706 Critical value = 4.15  q _  (not given in version 16.1) Intervals for (column level mean) - (row level mean) 1 2 3 4 2 -13.801 1.801 3 -6.801 -0.801 8.801 14.801 4 -15.801 -9.801 -16.801 -0.199 5.801 -1.199 5 -18.801 -12.801 -19.801 -10.801 -3.199 2.801 -4.199 4.801 Minitab: Stat>>ANOVA>>One-Way Anova then click “comparisons”.

28. Minitab Output for Broker Data Grouping Information Using Tukey Method broker N Mean Grouping 5 6 17.000 A 4 6 14.000 A 2 6 12.000 A B 1 6 6.000 B 3 6 5.000 B Means that do not share a letter are significantly different.

29. Special Multiple Comp. Method 3: Dunnett’s test Designed specifically for (and incorporating the interdependencies of) comparing several “treatments” to a “control.” Example: 1 2 3 4 5 6 12 5 14 17 Col } R=6 CONTROL Analog of LSD (=t  /2 x  2 MSW ) R D = Dut  /2 x  2 MSW R From table or Minitab

30. D= Dut  /2 x  2 MSW/R = 2.61 (  2(21.2) ) = 6.94 - Cols 4 and 5 differ from the control [ 1 ]. - Cols 2 and 3 are not significantly different from control. 6 In our example: 1 2 3 4 5 6 12 5 14 17 CONTROL Comparison |difference| >or< 6.94 1 vs. 2 1 vs. 3 1 vs. 4 1 vs. 5 6 1 8 11 < >

31. Dunnett's comparisons with a control (Minitab) Family error rate = 0.0500  controlled!! Individual error rate = 0.0152 Critical value = 2.61  Dut_  /2 Control = level (1) of broker Intervals for treatment mean minus control mean Level Lower Center Upper --+---------+---------+---------+----- 2 -0.930 6.000 12.930 (---------*--------) 3 -7.930 -1.000 5.930 (---------*--------) 4 1.070 8.000 14.930 (--------*---------) 5 4.070 11.000 17.930 (---------*---------) --+---------+---------+---------+----- -7.0 0.0 7.0 14.0 Minitab: Stat>>ANOVA>>General Linear Model then click “comparisons”.

32. What Method Should We Use? Fisher procedure can be used only after the F-test in the Anova is significant at 5%. Otherwise, use Tukey procedure. Note that to avoid being too conservative, the significance level of Tukey test can be set bigger (10%), especially when the number of levels is big.

33. Contrast Example 1 1 2 34 Placebo Sulfa Type S 1 Sulfa Type S 2 Anti- biotic Type A Suppose the questions of interest are (1) Placebo vs. Non-placebo (2) S 1 vs. S 2 (3) (Average) S vs. A

34. In general, a question of interest can be expressed by a linear combination of column means such as with restriction that  a j = 0. Such linear combinations are called contrasts.

35. Test if a contrast has mean 0 The sum of squares for contrast Z is where R is the number of rows (replicates). The test statistic Fcalc = SSC/MSW is distributed as F with 1 and (df of error) degrees of freedom. Reject E[C]= 0 if the observed Fcalc is too large (say, > F 0.05 (1,df of error) at 5% significant level).

36. Example 1 (cont.): a j ’s for the 3 contrasts P vs. P: C 1 S 1 vs. S 2 :C 2 S vs. A: C 3 1 2 3 4 -3 1 1 1 0 -1 1 0 0 -1 -1 2 PS1S1 S2S2 A

37. top row middle row bottom row                            Calculating

38. Y. 1 Y. 2 Y. 3 Y. 4 111 5.33 0.50 8.17 0 0 Placebo vs. drugs S 1 vs. S 2 Average S vs. A P S 1 S 2 A 14.00 56710 0 1 -3 2

39. 42.64.504.00 8.1765.36 5.33

40. Tests for Contrasts Source SSQ df MSQ F Error C1C2C3C1C2C3 42.64 4.00 65.36 111 42.64 4.00 4.0065.36 8.53 8.53.80.8013.07 140 28 5 F 1-.05 (1,28)=4.20

41. Example 1 (Cont.): Conclusions The mean response for Placebo is significantly different to that for Non-placebo. There is no significant difference between using Types S1 and S2. Using Type A is significantly different to using Type S on average.