1. Handout Eight: Two-Way Between- Subjects Design with Interaction- Assumptions, & Analyses EPSE 592 Experimental Designs and Analysis in Educational Research Instructor: Dr. Amery Wu Handout Eight: Two-Way Between- Subjects Design with Interaction- Assumptions, & Analyses EPSE 592 Experimental Designs and Analysis in Educational Research Instructor: Dr. Amery Wu 1

2. Today, we will introduce the analysis and assumptions of a two-way between-Subjects designs. This design explains/predicts continuous data using two independent variables, each with two or more levels (i.e., groups) where subjects are independent across the groups. Where We Are Today Two-way Between-Subjects Factorial Design with Interaction Measurement of Data ContinuousCategorical Type of the Inference Descriptive AB Inferential CD 2

3. 3 The Design  A two-way between-subjects ANOVA is an inferential statistical method  includes two factors (independent variables),  each factor has two or more levels (groups or categories)  subjects are independent across the levels for each factor. Data Assumptions  Data assumptions are the same as those of one-way ANOVA, except that assumptions are made about the RxC sub-samples.  R is the # of levels of the first factor the C is the # of levels of the second factor. Review of Two-way Between-Subjects Factorial Design

4. 4 Two-way Between-Subjects Factorial Design Partitioning the Total Sum of Squares by the 2 Factors SS b-gA SS b-gB SS b-gA*B 2 Factors

5. 5 Statistical Model of Two-way Factorial ANOVA In a two-way ANOVA, two independent variables, A and B, are involved, which are termed “factors” in ANOVA However, the data (DV) is being modeled (explained or predicted) by three terms: A, B, and their interaction effect A*B Data = Model + residual Y = intercept + aA + bB + cA*B + residual Questions: What is the “model” for a one-way ANOVA? Write the equation.

6. 6  Variable A is treated as the primary cause, of which the effect on the Y (DV) is the major research interest. An analogy is the effect of a switch on the light.  Variable B is interpreted as a moderator that adjusts the effect of A on Y. An analogy is a dimmer adjusts the effect of the switch on the light. Namely, the effect of A on Y is dependent on the level (or amount) of B  The choice of which is the primary cause should be guided by the research question and the substantive theory. Interaction Effect Interpreting the Results- Randomized Experimental Data Two-way Between-Subjects Factorial ANOVA

7. 7  The relationship between A and Y (DV) is dependent on the level (or amount) of B.  Variable B moderates the relationship between A and Y.  The choice of whether A or B should be considered as the moderator depends on the research question and the substantive theory. Interpreting the Results - Observational Data Two-way Between-Subjects Factorial ANOVA Interaction Relationship Question: what are the differences between this figure and the one on the last slide?

8. 8 One Way ANOVA (One Independent Variable A) SS tot = SS bgA + SS wg, or simply SS tot = SS A + SS r Two Way ANOVA (Two independent variables A & B) SS tot = SS bgA + SS bgB + Ss bgA*bgB + SS wg or simply SS tot = SS A + SS B + SS AB + SS r Comparing One-Way and Two-Way Factorial ANOVA Using Data = Model + Residual

9. Interpreting a Two-way Factorial ANOVA- - When the Interaction Effect is Non-significant When the interaction effect is not significant, run the post hoc paired-comparisons for the main effect(s) that are significant If both main effects are significant; hence, more multiple paired comparisons are involved, a more conservative post- hoc t-test should be considered to protect from Type-I error. (see the Word document: GLM post hoc comparison.doc)

10. Female Male An example Plot for No Interaction Effect When the two separate lines (here representing the moderator) are parallel, it signals that there is no interaction effect. However, one must NOT use such a plot to infer about the population interaction effect as many textbooks suggest because it is descriptive and does not test whether the parallelism occurred only by chance.

11. Lab Activity: Post-hoc Comparison for Main effects

12. SPSS Output: Post-hoc Comparison for Main effects

13. Using the same research context (IV: instruction and gender; DV: performance), all the means of the main effect and interaction cells are the same as before. However, number of subjects increased from 5 to 15 for each cell.

14. 14 Students’ performance was rated by their clinical supervisors on a scale of 0-10. Mean ratings of the groups are tabulated below. An Example to Contextualize Learning Two-way Between-Subjects Factorial ANOVA Research Questions: 1.Does the instruction method have an effect on students’ performance? 2.Is there a sex difference in students’ performance? 3.Does the effect of instruction method on the students’ performance depend on the sex of the students? Means Instruction Method ABCMar sex SexMale6787 Female6998 Mar instruction 688.57.5

15. 15 Computing The F Statistic and the F Test Two-way Between-Subjects Factorial ANOVA Lab Activity: See Excel File “Two Way ANOVA with Interaction.xls.”

16. 16 Calculating the F Statistic - SS of Total Calculating Total Sum of Squares Subject ID SexInstructionperformanceDeviation 2 1 female1412.25 2 female156.25 3 female162.25 4 female170.25 87 male370.25 88 male380.25 89 male392.25 90 male3106.25 SS Total 274.5 df Total N-1 89

17. 17 Calculating the F Statistic - SS for the Main effects of Instruction Method Means Instruction Method ABCMar sex SexMale6787 Female6998 Mar instruction 688.57.5 SS instruction30*(6-7.5)(6-7.5)+30*(8-7.5)(8-7.5)+30*(8.5-7.5)(8.5-7.5)105 df = C-12

18. 18 Calculating the F Statistic - SS for the Main effects of Sex Means Instruction Method ABCMar sex SexMale6787 Female6998 Mar instructio n 688.57.5 SS Sex45*(7-7.5)*(7-7.5)+45*(8-7.5)*(8-7.5) 22.5 df = R-11

19. 19 Testing the interaction effect is, in essence, testing whether there is a true difference in the difference. Calculating the F Statistic - SS for “instruction by sex” interaction effect Instruction Method ABC SexMale678 Female699 sex difference0-2 Sex diff Instruction A0 Instruction B-2 Instruction C Mean SS difference=15*(0-(-1))*(0-(-1))+15*(-2-(-1))*(-2-(-1))+15*(-1-(-1))*(-1-(- 1))30 SS interaction = Half SS difference15 df = (R-1)*(C-1)2

20. 20 Calculating the F Statistic - SS for within Groups Calculating SS Within Groups Subject IDSexInstructionPerformanceDeviation 2 1female144 2 151 3 160 4 171 5 184 6 144 7 151 8 160 9 171 10female184 11female144 12female151 13female160 14female171 15female184 m f1 630SS f1 16male144 17male151 18male160 19male171 20male184 21male144 22male151 23male160 24male171 25male184 26male144 27male151 28male160 29male171 30male184 m m1 630m m1 31female281 32female290 33female290 34female290 35female2101 36female281 37female290 38female290 39female290 40female2101 41female281 42female290 43female290 44female290 45female2101 m f2 96SS f2 46male254 47male261 48male270 49male281 50male294 51male254 52male261 53male270 54male281 55male294 56male254 57male261 58male270 59male281 60male294 m m2 730SS m2 61female381 62female390 63female390 64female390 65female3101 66female381 67female390 68female390 69female390 70female3101 71female381 72female390 73female390 74female390 75female3101 m f3 96SS f3 76male364 77male371 78male380 79male391 80male3104 81male364 82male371 83male380 84male391 85male3104 86male364 87male371 88male380 89male391 90male3104 m m3 830SS m3 SS Within 132 df Total N- # of cell groups 84

21. 21 The ANOVA Summary Table Two-way Between-Subjects Factorial ANOVA ANOVA Summary F Table SSdfMSFp Instruction105.0252.5033.41<0.001 Sex22.5122.5014.32<0.001 Instruction*Sex15.027.504.770.011 Error (within)132.0841.57 Total274.529

22. 22 Lab Activity: Two-Way Between-Subjects Factorial ANOVA in SPSS

23. 23 Lab Activity: SPSS Outputs for Two-Way Between-Subjects Factorial ANOVA in SPSS

24. Interpreting a Two-way Factorial ANOVA- - When the Interaction Effect is Significant Increasing the sample size increased the power, the interaction effect is now significant even though all the cell means were the same.

25. Interpreting a Two-way Factorial ANOVA - When the Interaction Effect is Significant When SPSS “General Linear Model” (GLM) menu is used:  If the interaction effect is significant, interpret the interaction effect ONLY (even if the main effects are significant).  One should then conduct the post hoc paired comparisons of the cell mean differences - simple main effects. The simple main effect is the mean difference between the groups of factor B (i.e., sex) for each given group of factor A (i.e., instruction method).  Using SPSS GLM, interpreting the main effects is unnecessary because, the main effects are included in the estimate of cell means. Test of the simple main effects actually have incorporated the test of main effects.

26. Lab Activity: SPSS Syntax for Testing Simple Main Effects UNIANOVA performance BY instruction sex /METHOD = SSTYPE(3) /INTERCEPT = INCLUDE /CRITERIA = ALPHA(.05) /DESIGN = instruction sex instruction*sex /EMMEANS = TABLES (instruction*sex) compare (sex) adj (Bonferroni). Note that for post hoc simple main effect t-tests, only the methods of BONFERRONI, LSD, SIDAK tests are available in SPSS. SPSS drop-down menu dose not provide function of testing simple main effects. Instead, you should use the following syntax of EMMEANS to test them.

27.  For females, there was significant difference between instruction methods A and B, methods A and C (p= 0.005). However there is no difference between methods B and C, p= 1.000.  For males, there were not significant differences between any instruction methods. Lab Activity: SPSS Output for Testing Simple Main Effects Protecting from Type- I error

28. Primary IV Moderator IV Remember to hit “Add” Lab Activity: Plotting Interaction Effect in SPSS

29. Lab Activity: SPSS Plot for Interaction Effect The difference between female and male students’ performance is different for each instruction method group. Namely, the gender difference in performance is dependent on (moderated by) which instruction method they receive. The effect of instruction method on students’ performance is different for males and females. Namely, the effect of the instruction method is dependent on ( moderated by) sex.

30. 30 Interpreting Models with More Than Three Factors  In general, for any ANOVA models, if the higher-order interaction effect is statistically significant, we do not interpret the lower-order effects directly, including the main effects or the interaction effects.  Lower-order effects will be interpreted in the context of the higher level interaction effect.  For example, if the 3-way interaction of an ANOVA including three factors is statistically significant, the 2- way interaction effects are interpreted in the context of the 3-way interaction. If a 2-way interaction effect is significant, its corresponding one-way main effects are interpreted in the context of the 2-way interaction effect.

31. 31 Observed Effect Size Eta-Squared vs. Partial Eta-Squared Eta-squared and partial eta-squared are effect size measures for the effect/difference of the factors included in a ANOVA. Lab Activity: Calculate the eta-squared and partial-eta squared using the ANOVA table in the Excel file: Two Way ANOVA with Interaction.xls. Questions: Is there a difference between eta-squared and partial eta-squared for one-way ANOVA? Will partial eta-squared always be smaller or greater than the eta-squared? Two common used effect size measures for ANOVA: h 2 (eta-squared) = SS effect / SS total Partial h 2 = SS effect / (SS effect + SS error ) or = SS effect / (Ss total – SS other effects )

32.  Given that the numerators of the two effect size measures are the same, Partial h 2 will yield a bigger effect size than does the h 2 because the denominator for partial h 2 is smaller (by removing the SS other than the given effect.) However, h 2 is conceptually easier to interpret; indicating the percentage of the total sum of square that is attributable to a given effect. Also, h 2 can be used to compare the relative effects because all the effects have a common denominator. Observed Effect Size Eta-Squared vs. Partial Eta-Squared

33. Lab Activity: Effect Sizes for Two-way ANOVA in SPSS