1. Handout Five: Between-Subjects Design of Analysis of Variance- Planned vs. Post Hoc Comparisons EPSE 592 Experimental Designs and Analysis in Educational Research Instructor: Dr. Amery Wu Handout Five: Between-Subjects Design of Analysis of Variance- Planned vs. Post Hoc Comparisons EPSE 592 Experimental Designs and Analysis in Educational Research Instructor: Dr. Amery Wu 1

2. Multiple Comparisons  When comparing means of two groups (levels) of the IV (factor), there is only one single comparison, e.g., treatment vs. control. When the mean difference of this comparison is hypothesis tested, it is referred to as the two-independent samples t-test. This is why no omnibus F test is needed for a two-group comparison.  When there are more than two groups, e.g., treatment A to C and control, there are multiple ways of comparing two groups. The groups could be 1)the original groups observed naturally (observational design) or formed manipulatively (experimental design) for data collection, or 2)composite groups formed by statistically weighting the original groups into combined or mixed groups. 2

3. 3 The goals for today’s class To learn about planned and post hoc methods for comparing group means. To learn to choose between the two methods and learn about the advantages and disadvantage of each method. To learn to code the contrasts for planned comparisons To learn to use SPSS to output results for both methods of comparisons, and interpret the results.

4. There are two types of multiple comparisons post hoc or planned 4

5.  If the researcher does not hypothesize which group means he would like to compare prior to the data collection and simply let the data show which two groups differ, then the comparison is referred to as a post hoc comparison.  In this case, all the possible mean differences between any pairs of groups, say Treatment A, B, C, and control, will be tested. There are g*(g-1)/2 possible pairs, where g is the # of groups. For this example, there are (4 x 3)/2 = 6 possible paired comparisons. Post Hoc Comparisons 5

6.  Planed comparisons focus on a few scientifically or theoretically sensible comparisons. When a researcher specifies which two group means (original or combined) he would like to compare before he collects the data and test his data, the comparisons are referred to as planned comparisons.  For example, a researcher may only be interested in testing whether each of the treatment groups, say A, B, or C differs to the control (no-treatment) group. Planned Comparisons 6

7. Like the one-tailed hypothesis test, planned comparisons are often used to answer specific questions. It is used when the researchers have a firm belief about the direction and the sizes of the means for the two groups; when there is a specific theory (hypothesis) he would like to test; or when earlier literature has informed the relative size of the means. If the researchers simply would like to explore group differences because the research is still at an early stage of research, lacks theories to build hypotheses, or has no literature to back up their reasoning, then post hoc comparisons are suggested. Post hoc comparisons are mostly seen in the literature. Planned or Post Hoc? 7

8. Post Hoc Paired-Comparisons The F-test is an omnibus test to detect the possibility of group mean differences in the population. If the omnibus F-test is significant, the next task is to detect which pair(s) of means are different using independent-samples t-tests. For independent variable with “g” groups, there are g(g- 1)/2 non-redundant paired comparisons. To protect from Type-I error rate, the usual t-test is modified (typically by upwardly adjusting the standard error of the mean difference or the critical value according to the number of paired comparisons involved). The most common post-hoc t-test (protected for Type-I error) is Tukey’s test 8

9. This data was originally contrived by Dr. Karl L. Wuensch and was modified by the instructor for pedagogical reasons. The independent variable is the daily dose of the new drug with 3 levels (control, 10mg, & 20mg) that were randomly prescribed to 3 groups of patients (20 in each group). The dependent variable is the patients’ depression level in quantity measured after two months of the new treatment. Research Question: Does the dose of the drug treatment have an effect on the patients’ depression level? SPSS Activity: Run the descriptive statistics separately for each group and report what you’ve observe. 9 Post Hoc Comparisons - An Example to Contextualize the Learning

10. 10 Design Experimental Observational Data Continuous Categorical Model Descriptive/Summative Explanatory/Predictive Inference Descriptive vs. Inferential Relational vs. Causal Research Question Quantitative Methodology Network

11. 11 Lab Activity: Post Hoc Comparisons of ANOVA in SPSS

12. Lab Activity: SPSS Output for Post Hoc Comparisons of ANOVA 12

13. Planned Comparisons - An Example to Contextualize the Learning The Dean of Faculty of Science has a mandate to improve the writing proficiency of academic English of the undergraduate students in the Faculty. She would like to know which groups of students, if any, are in need of some assistance with their academic writing. She sampled 60 students and collected a 500 words writing sample from each student. Students’ performances were rated on the scale of 0-100 by expert raters in the filed of language assessment. Twenty of the recruited students are English first speakers, the other 20 speak a fist language that has a Latin origin, and the rest speak a non-Latin origin language as their first language. Specially, the Dean would like to know that (1)whether English first speakers write better or worse than the Latin-origin speakers, and (2)whether the English first speakers write better or worse than the non-English first speakers. 13

14. 14 Design Experimental Observational Data Continuous Categorical Model Descriptive/Summative Explanatory/Predictive Inference Descriptive vs. Inferential Relational vs. Causal Research Question Quantitative Methodology Network

15. Question 1: Whether English-first speakers write better or worse than the Latin-origin speakers? Ho: µ English =µ Latin Origin Ho: µ English ≠ µ Latin Origin Question 2: Whether the English first speakers write better or worse than the non-English first speakers? Ho: µ English =µ non-English Ho: µ E ≠ µ non-English Planned Comparisons - An Example to Contextualize the Learning 15 Contrasts for the Two Hypotheses

16. In SPSS, Analyze > Compare Means > One-Way ANOVA > Contrasts 16 Lab Activity: Planned Contrasts Comparisons in SPSS

17. Note that it could be argued that the Dean does not have to correct the Type I error (lowering the alpha) because she only planned 2 contrasts to be tested, which is not greater than the degrees of freedom for between-groups df =3-1=2; thus, the Type I errors are still controlled under 0.05. If more contrasts are planned and tested, then correction of Type I error may be necessary. 17 Lab Activity: SPSS Output for Post Hoc Comparisons of ANOVA

18. Planned comparisons could be more powerful (more likely to reject a false hypothesis) than the post hoc comparisons for two major reasons: (1) Typically, the differences are more likely to be true because they are specified based on a theory or previous findings rather than the shot-gun approach of the post hoc comparisons; (2) There could be more tolerance for Type I error for each planned comparison because the overall alpha is shared by fewer comparisons than the number of all possible pairs in the post hoc comparisons. Planned or Post Hoc? 18