1 Advanced Research Methods in Psychology - lecture - Matthew Rockloff Scheffé Post-Hoc Comparisons

2 When to use a Scheffé Posthoc comparison … 1 Whenever an ANOVA model is used to examine the differences among more than 2 groups, a posthoc procedure can be used to compare differences between all pairs of means. Posthoc comparisons are very similar to t-tests. However, posthoc comparisons are more appropriate for multiple tests, because they help control type-I error.

3 When to use a Scheffé Posthoc comparison … 2 Type I-error is the chance of wrongly accepting differences between means as significant. In a t-test, this chance is controlled to be at most 5%. In other words, we only accept 2 means as significantly different if the p-value in the t-test is less than “.05.”

4 When to use a Scheffé Posthoc comparison … 3 In a study that has several groups, we could do several t-tests to compare all the differences between means. However, each t-test has a separate 5% chance of making a wrong conclusion by falsely declaring 2 means significantly different. When we do several tests, the chance of making at least one wrong conclusion starts to expand dramatically.

5 When to use a Scheffé Posthoc comparison … 4 To understand “why” consider a simple toss of a fair coin. Each toss has a 50% chance of yielding “heads.” What if we tossed the coin 100 times? What is the chance of having at least one “heads?” Pretty darn likely!

6 When to use a Scheffé Posthoc comparison … 5 In the same way, the more t-tests we perform, the more likely we’ll get at least one conclusion wrong. Fortunately, posthoc comparisons have been con structured to adjust for this problem. They are more conservative than t-tests and control for Type-I error.

7 Example 8.1 We’ll return once again to the diet example. However, this time there are 3 different diets; including pizza, beer and cream. The outcome is weight gain. The research question follows: “Is there any difference in weight gain between the 3 diets?”

8 Example 8.1 (cont.) The first step is to create an ANOVA table similar to our previous example. See last week’s example for a detailed explanation of how to calculate a Oneway ANOVA. However, a brief hand-worked solution is provided on the next slide 

9 Calculations continued next slide  Weight Gains: PizzaBeerCream S 2 xj = N=15 n=5

10 ANOVA table Source of Variance Sum of Squares Degrees of Freedom Mean SquareF-ratio Critical Value Reject Decision SVSSdfMSFCVReject? BTW Yes WITH TOT4814 Example 8.1 (cont.)

11 Example 8.1 (cont.) Before we can do posthoc comparisons, we must interpret the ANOVA table. Since we now have 3 conditions, the conclusion we make is slightly different: There was model significance for the ANOVA, F(2,12) = 30.00, p <.05, indicating at least one significant difference among the means.

12 Example 8.1 (cont.) The ANOVA table provides us with what is called the “omnibus F-test” or likewise “model significance.” In the Oneway ANOVA, this test shows that there is at least one significant difference between a pair of means. Of course, there are 3 unique pairs of means in our example. The omnibus F-test does not indicate which pairs are significantly different. Therefore, we need a posthoc comparison to examine the difference between all pairs.

13 The Scheffé posthoc calculation by hand There are several types of posthoc comparisons, most of which are named after dead statisticians (e.g., Tukeys HSD, Fishers LSD and Ducans Multiple Range Test). Scheffé is the most conservative of the bunch. It controls experiment wide error rate to 5%, or in other words, there at most a 5% chance of making any type-I errors using the procedure. While this is desirable, it also makes the Scheffé procedure less sensitive.

14 The Scheffé posthoc calculation by hand (cont.) Using Scheffé, we are more exposed to making the Type-II error of not identifying significant difference when they do exist. Other procedures make a different tradeoff between exposure to type-I and type-II error. As such, none is “better” than the other, only different in exposures to this natural tradeoff.

15 The Scheffé posthoc calculation by hand (cont.) The first step to using the Scheffé procedure involves calculating the differences in all pairs of means: L1 = L2 = L3 = In our example: L1 = -2 L2 = -4 L3 = -2

16 The Scheffé posthoc calculation by hand (cont.) Next, we need to compare these differences to Scheffé’s critical “S” value:, where F α is the critical value of the ANOVA table (see previous calculation).

17 The Scheffé posthoc calculation by hand (cont.) In our example, the critical “S” is: Since the absolute value of each difference (L1, L2 and L3) exceeds the critical value, we can declare significant difference between all pairs of means. Summarizing our findings from the ANOVA and the posthoc comparisons, we conclude: (see next slide) = 1.44

18 We conclude … There was model significance for the ANOVA, F(2,12) = 30.00, p <.05, indicating at least one significant difference among the means. Scheffé posthoc comparisons showed that weight gain was higher in the Cream condition (M = 6.00) than the Beer condition (M = 4.00), p <.05 (two- tailed). In turn, weight gain was lower in the Pizza condition (M = 2.00) than either the Beer Condition, p <.05 (two-tailed), or the Cream condition, p <.05 (two-tailed).

19 Alternative conclusion … The conclusion may still be somewhat hard to read. Therefore, an alternative conclusion can refer the reader to a figure that illustrates the differences: See next two slides

20 Alternative conclusion: pt 1 … There was model significance for the ANOVA, F(2,12) = 30.00, p <.05, indicating at least one significant difference among the means. In addition, a Scheffé posthoc comparison showed that all means were significantly different, p <.05 (two-tailed). Weight gain was highest in the Cream condition, followed by the Beer condition and the Pizza condition (see figure 8.1). (See next slide.)

21 Alternative conclusion: pt 2 …

22 Example 8.1 Using SPSS As before, we need to setup variables in SPSS: IndependentVariable = diet (1 = Pizza, 2 = Beer, 3 = Cream) DependentVariable = wtgain (weight gain)

23 Example 8.1 Using SPSS (cont.) The data is entered into the SPSS data view as demonstrated opposite 

24 Example 8.1 Using SPSS (cont.) The SPSS syntax includes commands for both the Oneway ANOVA, and an additional command to graph the results:

25 Example 8.1 Using SPSS (cont.) The results appear in the SPSS output viewer: This ANOVA table simply reproduces the Omnibus test illustrated previously.

26 Example 8.1 Using SPSS (cont.) The table above compares all possible mean pairs of the three conditions. As indicated above, there are only three unique pairs, so this table is repetitive in sections (i.e., Pizza-Beer p-value = Beer-Pizza p-value).

27 Example 8.1 Using SPSS (cont.) Condition means which are significantly different (p <.05) appear in separate columns. In our example, the means for each condition appear in separate columns. The table opposite is a simple representation of which “means” are significantly different from one another.

28 Example 8.1 Using SPSS (cont.) The graph opposite is simply a reproduction of the figure illustrated earlier. It can be modified within SPSS to improve its appearance and make it conform to APA style.

29 The conclusion (again) Our conclusions can be restated given the exact probabilities provided in SPSS: There was model significance for the ANOVA, F(2,12) = 30.00, p <.05, indicating at least one significant difference among the means. Scheffé posthoc comparisons showed that weight gain was higher in the Cream condition (M = 6.00) than the Beer condition (M = 4.00), p =.01 (two- tailed). In turn, weight gain was lower in the Pizza condition (M = 2.00) than either the Beer Condition, p =.01 (two-tailed), or the Cream condition, p <.01 (two-tailed).

30 Advanced Research Methods in Psychology - lecture - Matthew Rockloff Scheffé Post-Hoc Comparisons Thus concludes 