ANOVA and Multiple Comparison Tests BUSI 6480 Lecture 3 ANOVA and Multiple Comparison Tests

Contrasts Among Means A contrast (or comparison) among means is a difference among means. The symbol yi is used to denote the i-th contrast: yi = mj – mk, estimated as: Sometimes contrasts involve the comparison of the average of two experimental groups with a control group:

Contrast equations, pairwise comparisons, and orthogonal contrasts Contrasts can be expressed as linear combinations of the group means, in the form of contrast equations: When all coefficients cj except two are equal to zero, the contrast becomes a pairwise comparison: Contrasts that are mutually non-redundant are called orthogonal contrasts. In those, the coefficient matrix of the contrast equations is of full rank.

A priori and a Posteriori contrasts Confirmatory data analysis typically involves a priori tests, which are tests the designer of the experiment had in mind during the planning phase. Exploratory data analysis typically involves a posteriori tests. These are used in the preliminary stages of a research program, when the researcher does not have the knowledge to formulate testable models.

Error rates When the experiment involves multiple contrasts, the probability of making the Type I error (false significance) becomes harder to control. For example, for c independent contrasts, Probability of one or more Type I errors = 1-(1-a)c A number of multiple comparison procedures have been developed. These aim to control the per-contrast, familywise, or per-family error rate, and also have a good combination of: conceptual simplicity, ease of computation, excellent power, availability of confidence intervals, and robustness (see Table 4.1.2).

Contrasts Contradicting ANOVA Results Is it possible to get a 'significant' result from a post test even when the overall ANOVA was not significant? Yes it is possible. The exception is Scheffe's test. It is intertwined with the overall F test. If the overall ANOVA has a p-value greater than 0.05, then the Scheffe's test won't find any significant post tests. In this case, performing post tests following an overall nonsignificant ANOVA is a waste of time but won't lead to invalid conclusions. But other multiple comparison tests can find significant differences (sometimes) even when the overall ANOVA showed no significant differences among groups.

Contrasts Contradicting ANOVA Results How can I understand the apparent contradiction between an ANOVA saying, in effect, that all group means are identical and a post test finding differences? The overall one-way ANOVA tests the null hypothesis that all the treatment groups have identical mean values, so any difference you happened to observe is due to random sampling. Each post test tests the null hypothesis that two particular groups have identical means. The post tests are more focused, so have power to find differences between groups even when the overall ANOVA is not significant.

Contrasts Contradicting ANOVA Results Are the results of the overall ANOVA useful at all? ANOVA tests the overall null hypothesis that all the data come from groups that have identical means. If that is your experimental question -- does the data provide convincing evidence that the means are not all identical -- then ANOVA is exactly what you want. More often, your experimental questions are more focused and answered by multiple comparison tests (post tests). Note that the multiple comparison calculations all use the mean-square result from the ANOVA table. So even if you don't care about the value of F or the p-value, the post tests still require that the ANOVA table be computed.

Difference Between Estimate and Contrast Statements in SAS Contrast in SAS proc glm data=myAnovaData; class Level; model Resp = Level ; Estimate '1and2Versus3' -.5 -.5 1; Contrast '1and2Versus3' -.5 -.5 1; ods output overallanova = atable; run; OUTPUT Contrast DF Contrast SS Mean Square F Value Pr > F 1and2Versus3 1 3.75000000 3.75000000 0.53 0.4732 Standard Parameter Estimate Error t Value Pr > |t| 1and2Versus3 -0.75000000 1.03091114 -0.73 0.4732

Different Ways of Representing Contrast Coefficients proc glm data=myAnovaData; class Level; model Resp = Level ; Means Level/ Sidak Bon; estimate 'Diff 1_2_3 minus 4_5' Level -2 -2 -2 3 3/ divisor = 6; estimate 'Diff 1_2_3 minus 4_5' Level -.33 -.33 -.34 .5 .5; estimate 'Diff 1_2_3 minus 4_5' Level -2 -2 -2 3 3; ods output overallanova = atable;

Data Input: Y in One Column, Grouping Variable in Another You always need to input data with Y in one column and grouping variable in another whether using Minitab, SAS or SPSS. One way to translate data to single column with Y and grouping variable using SAS: Data myAnovaData; Set Treat1 Treat2 Treat3; If _N_ <= 30 then Level = 3; If _N_ <= 20 then Level = 2; If _N_ <= 10 then Level = 1; Keep resp Level;

Data Input in SAS Data Treat1; set Learning; resp = a1; Level = 1; Data Treat2; resp = a2; Level = 2; Data Treat3; resp = a3; Level = 3; Data myAnovaData; Set Treat1 Treat2 Treat3; Keep resp Level; Another Way to Translate Data to Single Column with Y and Grouping Variable Using SAS:

The One-Way Procedure in SPSS Quick to use, but doesn’t have as many options as the Univariate GLM

Univariate Contrasts in SPSS Analyze > General Linear Model > Univariate But you can only select one contrast per run, unlike the one-way ANOVA.

Univariate Contrasts in SPSS Simple Contrast Compares Each Level to a Reference Category. NOTE: The Change button needs to be clicked to select contrast type.

Contrast types in SPSS Definitions of Contrast Types in SPSS are given below. If different contrasts are requested, they must be programmed.

Deviation Contrasts Deviation contrasts compares a level with the mean of the other three levels.  A contrast is not made for the reference category.

Simple Contrasts Simple contrasts compare each cell with a reference cell. The reference cell can be either the last level of the factor or the first level of the factor

Difference Contrasts Difference contrasts compare:  level 1 with level 2, level 3 with the mean of previous two levels, level 4 with the mean of previous three levels, and so forth.

Helmert contrasts Helmert contrasts compare:  the first level of the factor with all later levels, the second level with all later levels, the third level with all later levels, and so forth.

Repeated Contrasts Repeated contrasts compare consecutive pairs of levels, for example level 1 with level 2, level 2 with level 3, level 3 with level 4, and so forth.

Omega-square Omega squared is an estimate of the model variance accounted for by the variability of the treatment levels. For a contrast, w2 can be computed as: w2 = (SS(contrast) - MSerror)) / (MSerror + SStotal) Not difficult to compute, if you know the group means and ANOVA output : SS(contrast) = (estimate of contrast)2 / (sum of contrast coefficients squared/group size)

Partial Eta and Eta Squared Eta Squared: It is the proportion of the total variance that is attributed to an effect.   It is calculated as the ratio of the effect variance (SSeffect) to the total variance (SStotal): h2 = SSeffect / SStotal Partial Eta Squared: The formula differs from the Eta squared formula in that the denominator includes the SSeffect plus the SSerror rather than the SStotal. hp2 = SSeffect / (SSeffect + SSerror) Partial Eta squared, hp2 - same as R-square for one-way ANOVA.

Random Effects Model ANOVA Same ANOVA table as for Fixed Effects Model – This is true for only the one-way ANOVA. Interpretation is different. Since effects for a random effects model were selected at random, multiple comparisons and contrasts are not of interest.

Random Factors: No multiple comparisons Note that in SPSS, the Post Hoc dialog box doesn’t show any variables under Factor(s) when the grouping variable is placed in Random Factors.

Random Factors: Intraclass correlation For random effects, use Intra-class correlation instead of omega squared. Easy computation: rI = (F-1)/ ((n-1) + F)

Box Plots The generic SAS code is: PROC BOXPLOT DATA=test;    PLOT response * group; RUN; QUIT; In SPSS, click Graphs > Boxplots.