1. Handout Ten: Mixed Design Analysis of Variance EPSE 592 Experimental Designs and Analysis in Educational Research Instructor: Dr. Amery Wu Handout Ten: Mixed Design Analysis of Variance EPSE 592 Experimental Designs and Analysis in Educational Research Instructor: Dr. Amery Wu 1

2. Today, we will introduce the analysis and assumptions of a mixed design. This design explains/predicts quantitative data using two independent variables- one (explicit) between groups and one (implicit) within-subjects (repeated measures). Each IV has two or more conditions. Where We Are Today Mixed Design Analysis of Variance Measurement of Data QuantitativeCategorical Type of the Inference Descriptive AB Inferential CD 2

3. 3 An Example to Contextualize the Learning 2x3 Mixed Design ANOVA

4. 4 An Example to Contextualize the Learning 2x3 Mixed Design ANOVA Questions: 1. What is the DV? 2. What is the between-groups factor? How many levels are there? 3. What is the within-subjects factor? How many levels are there? Lab Activity: Using your own words, state as many (non- redundant) research questions as you can based on this research design.

5. 5 1. Does exercise has an effect on the pulse? (Dose the pulse change over time (before, during, and after exercise)? 2. Is there a difference in the pulse between the two dietary groups? 3. Dose the pattern of temporal change in pulse due to exercise differ (be moderated) between dietary groups? Alternatively, is the relationship between diet and pulse differs (be moderated) by the timing of when the pulse is taken? An Example to Contextualize the Learning 2x3 Mixed Design ANOVA

6. 6 Design Experimental Observational Data Quantitative Categorical Model Descriptive/Summative Explanatory/Predictive Inference Descriptive vs. Inferential Relational vs. Causal Research Question Quantitative Methodology Network

7. 7 Lab Activity There are six groups of samples separated by three columns (purple, blue, and yellow) and by the 2 categories of the diet column (in green and red texts). Questions: In what way, the observation of the six groups is supposed to be dependent and independent?

8. An research design that incorporates both between- groups (i.e., independent) factor and within-subject (i.e., dependent, correlated, or repeated measures) factors. Hence, for a mixed design, there is at least two IVs with a possibility of inclusion of another interaction IV: “between* within” (i.e., two-way factorial design with interaction). Mixed design is also known as split-plot design Mixed (between & within) Experimental Design 8

9. 9 For the between-groups factor, the group membership of the participants are independent (i.e., exclusive). Participants in one group differ from those in another groups. Hence, the observation of the DV scores are unrelated across the groups. Examples for between-group factors are gender groups, randomly assigned treatment groups, or teachers in different school districts. Mixed (between & within) Experimental Design

10. 10 For the within-subjects factor (e.g., repeated measures), the observation of the participants under one condition is dependent of that of another because the participants are selected to be related by design. Examples for within-subject factors are repeated measures groups, matched groups, twin groups, group scores by the same raters, or dyads groups. Mixed (between & within) Experimental Design

11. What is a Mixed Design Analysis Mixed Design ANOVA is an inferential statistics that tests experimental treatment effects (or simply mean differences for observation data) with a mix of between- group and within-subject factors. A full factorial mixed design with one between and one within factors is similar to a two-way factorial ANOVA. The only difference is that one of the IV now is a within- subjects factor. A full two-way factorial ANOVA may include testing of the following effects: 1.the main effect of the between-groups factor 2.the main effect of the within-subjects factor 3.interaction effect- “between x within” factor 11

12. 12 See Excel file “Exercise_Diet_Pulse_Mixed Design with Interaction.xlsx” for computing F statistics for the mixed design ANOVA.

13. 13 Within (Time) BeforeWarm-upAfter BetweenVege77.67121.67173.56113.96 (Diet)Meat97.33146.56215.56110.51 94.88143.44210.31138.72 Means – Cross Tabulation between by within Questions: 1.By comparing which values, is the main effect of the between group factor (diet) examined? 2.By comparing which values, is the main effect of the within-subject factor (time) examined? 3.By comparing which values, is the interaction effect of between*within (diet*time) examined? Example- 2x3 Mixed Design

14. 14 The between-factor effect is investigated by comparing the between-factor marginal means- the mean DV scores among the groups for the between-factor(e.g., treatment groups or vegetarians vs. meat eaters). The F statistics is calculated using the error term for the between-factor (see Excel file). Between-factor Effect Vege 77.67121.67173.56 113.96 Meat 97.33146.56215.56 110.51

15. 15 The within-factor effect is investigated by comparing the within-factor marginal means- the mean DV scores among the groups for the within- factor (e.g., repeated measures of same individuals; comparing their own repeated DV scores under different treatment conditions). The F statistics is calculated using the error term for the within-factor (see Excel file). Within-factor Effect BeforeWarm-upAfter 77.67121.67173.56 97.33146.56215.56 94.88143.44210.31

16. 16 “Between x Within” Interaction Effect The “between x within” interaction effect is investigated by conditional mean differences (i.e., simple main effects)- comparing the mean differences in the DV score among the between-factor groups for each of the within-factor groups (alternatively, among the within-factor groups for each of the between-factor groups). Take the Pulse data for example, the interaction effect is investigated by comparing the mean differences between the diet groups (vege vs. meat) for each of the three within groups (before, warm-up, and after exercise). The F statistics is calculated using the error term for the within-factor. Within BeforeWarm-upAfter BetweenVege 77.67121.67173.56 Meat 97.33146.56215.56 diff19.6724.8942.00

17. Data Assumptions about Mixed Effect ANOVA The DV scores for each of the PxQ groups are all normally distributed (P= # of groups for the between- factor and Q= # of groups for the within-factor). Independence and equal (homogeneity) variances assumption for the between-factor groups. Normal distribution and sphericity assumption for the within-factor groups. The difference scores between any pairs of the repeatedly measured are normally distributed. Also, the variances of the difference scores are equal (sphericity). With Q groups, there are Q(Q-1)/2 difference scores. 17

18. 18 Interpretation of the Mixed Design ANOVA-Between Interpretation of the between-factor effect makes sense only if the interaction effect is non-significant. The between-factor effect is interpreted the same way as is the one-way independent ANOVA. The omnibus F test for the between-factor examines whether there is any statistically significant mean difference in the DV between any pairs of between- factor groups. If the omnibus F test is statistically significant, do post- hoc pairwise t-tests to find out which pair(s) of mean difference is statistically significant. If the equal variances assumption is not met, remember to choose a post-hoc pairwise t-test that does not assume equal variances.

19. 19 Interpretation of the within-factor main effect makes sense only if the interaction effect is non-significant. The within-factor main effect is interpreted the same way as is the repeated measures ANOVA (see Handout Four). The omnibus F test for within-factor examines whether there is any statistically significant mean difference in the DV between any pairs of within-factor groups. If the sphericity assumption is violated (Greenhouse- Geisser’s ἓ less than 0.75), pick one of the alternative F-tests that corrects for violation from sphericity (i.e., Greenhouse- Geisser, Huynh-Feldt, or Lower-bound in SPSS). If the selected omnibus F test is statistically significant, do post-hoc pairwise t-tests to find which pair(s) of mean difference is statistically significant. If sphericity assumption is violated, choose Bonferroni or Sidak’s post hoc t-test for correction(see Handout Four). Interpretation of the Mixed Design ANOVA- Within

20. 20 Interpretation of the Mixed Design ANOVA- Interaction If the omnibus F test for the interaction effect is non- significant, interpret the each of the significant main effect separately. Note that once the omnibus F test for the interaction effect is significant, it is inappropriate to interpret the between- or within-factor main effects even if these main effects are significant (always interpret the highest order of the effect that is significant). SPSS drop-down menu does not provide a function to test the simple main effects. With SPSS, to test and interpret the interaction effect in terms of simple main effects (i.e., conditional mean differences). See later slides.

21. 21 Mixed Design ANOVA by SPSS Remember to hit “Add”

22. 22 Mixed Design ANOVA SPSS Output  First, examine whether the interaction effect is significant within the F table of within-subjects effects.  To do so, we need to check whether the sphericity assumption (for with-factor) is met. Because the assumption is not met (using Greenhouse-Geisser Epsilon <0.75 criterion), we should use an F test that does not assume sphericity.

23. 23 Mixed Design ANOVA SPSS Output Because the “Time x Diet” Interaction effect is significant, the main effects of time and diet should be interpreted in the context of the other by interpreting the simple mail effects.

24. 24 One has to use the following syntax to produce the output: GLM PulseBefore PulseWarmUp PulseAfter BY Diet /WSFACTOR=Time 3 /MEASURE=Pulse /METHOD=SSTYPE(3) /EMMEANS=TABLES(Diet*Time) compare (diet) adj (Bonferroni) /CRITERIA=ALPHA(.05). Testing Simple Main Effects Using SPSS Syntax

25. 25 Interpretation of the Mixed Design ANOVA- Interaction This is the extra table produced by the bolded line in the syntax for testing the interaction effect in terms of simple main effects. For each of the within-factor groups (before, warm-up, and after), the mean difference in pulse between the two dietary groups is compared and tested.

26. 26 If the Interaction Effect is Non-significant When the interaction effect is non-significant, one can interpret the results separately for each of the between- and within-factor if they are significant. One can test the planned contrasts or conduct post hoc t tests, correcting for type-one error rate and correcting for violation of equal variances assumptions (for both each between- and within- effects). See Power Points slides course handout #8 and #9.