Handout Twelve: Design & Analysis of Covariance EPSE 592 Experimental Designs and Analysis in Educational Research Instructor: Dr. Amery Wu

Design and Analysis of ANOVA with Covariate(s) Where We Are Today Design and Analysis of ANOVA with Covariate(s) Measurement of Data Quantitative Categorical Type of the Inference Descriptive A B Inferential C D Today, we will introduce the design, analysis, and assumptions of ANCOVA- ANOVA with covariate(s). This design explains/predicts quantitative data using one or more independent variables (categorical with 2 or more levels) and one or more covariates (categorical or quantitative). 2

An Example to Contextualize Learning Design and Analysis of ACOVA A study investigated the effect of Omega-3 fatty acids on reducing the LDL cholesterol. Ten of the 20 participants were randomly selected to take Omega-3 supplement daily (treatment group); the other 10 participants abstained from taking any supplements containing Omega-3 (control group). Cholesterol levels [mg/ml] were measured six months later. Questions: In this random experimental design, what are the possible threats to internal validity?

Identifying Potential Covariates with the design (a priori) The researcher had foreseen some of the possible threats to the internal validity. For example, the random assignment may not have achieved random (equivalent) groups because of the small sample size. Furthermore, individual changes that may have some influence on participants’ cholesterol level, may occurs during the 6-month study period. To circumvent these problems ahead of time, the researcher hypothesized three possible, health, biological, and life style variables as “covariates”; they are (1) number of chronic diseases, (2) age, and (3) exercise time (hrs/wk), respectively.

What is Analysis of Covariance (ANCOVA)? ANCOVA is an inferential statistical technique used to analyze data obtained from experimental design, where the differences in the DV among the IV groups (treatment effects) are compared and tested. The purpose of an ANCOVA is to increase the power of F-test for the treatment effect. ANCOVA is used when the DV is continuous, the treatment IV is categorical, and typically the covariate(s) are continuous (although categorical covariates are also allowed). Mathematically, ANCOVA with one covariate can be written as Yi = b0 + b1Treatment + b2Covariate + ei. Note that this is equivalent to an OLS linear multiple regression.

What is Analysis of Covariance (ANCOVA)? Unlike two-way ANOVA where the effects of both the IVs are investigated, or how the effect of one IV interacts with that of the other IV is investigated, the covariate(s) in an ANCOVA are viewed as nuisance variables and is to be controlled for. They are nuisance variables because they are associated with the DV but their associations with the DV is of no interest to the researchers.

How ANCOVA Works? The function of the covariate(s) in an ANCOVA is to reduce error variance (within-groups variance) by removing some extra error variance left over from that accounted for by the treatment IV). In an ANOVA, In an ANCOVA, SStot = SSw-g + SSb-g = SStot = SSw-g + SSb-g + SScov SSw-g = Sstot - SSb-g > SSw-g = SStot - SSb-g - SScov F= Varbg/Varwg. When within-groups variance is reduced, and the between-groups variance remains the same, the F statistic of the treatment IV is increased, hence increases the power of the hypothesis testing of the effect of the treatment IV. For the above condition to be met, it requires that the covariate is related to the DV, but unrelated to the IV.

Lab Activity- Identifying a Potential Covariate statistically (post hoc) Conduct bivariate correlations using SPSS to examine which of the three variables are potential covariates for this study.

Data Assumptions about an ANCOVA In addition to the typical ANOVA assumptions made about the data, three more assumptions are required for an ANCOVA to work well and appropriately. 1. The covariate and the DV are linearly related. The first requirement is to make sure that the covariate and the DV share some common variance; adding the covariate would help reduce the error variance. 2. The covariate and the treatment IV are independent (uncorrelated). This requirement is borne upon the assumption that adding the covariate will reduce the error variance but not the variance contributable to the treatment (between-groups variance), i.e., the effect of the treatment IV. If the covariate is not only correlated with the DV but also with the treatment IV, the covariate will reduce not only the within-groups variance but also the between-groups variance of the treatment IV, hence may decrease the F-statistic of the treatment IV and reduce the power of the F test.

Data Assumptions about an ANCOVA 3. There is no interaction between the IV and the covariate (homogeneity of regression slopes, i.e., equal slopes). This assumption requires that the simple correlation between the covariate and the DV does not depend on the group membership of the individuals (treatment IV). That is, if the DV is regressed on the covariate separately for individuals in each IV group, the regression weights for the covariate will be the same across the groups. This is the same as assuming that there is no interaction effect between the covariate and the IV on the DV, and the interaction plot will display parallel lines. If the assumption is violated, the custom is to include and interpret the interaction effect. This method is referred to as the “attribute by treatment interaction (ATI)” design, where the covariate is not viewed as a nuisance but a moderator of the treatment. The interaction between the IV (treatment) and the continuous attribute (covariate) is to be investigated.

Modeling Steps for An ACNOVA 1. Check all the assumptions for a basic ANOVA with regard to the IVs of which the effects/differences are of interpretation interests. 2. Check whether there is a significant correlation between the covariate and the DV. 3. Check whether the correlation between the treatment IV and the covariate is non-significant. 4. Check whether the “treatment x covariate” interaction effect is non-significant. 5. If the four conditions stated above meet the requirements, conduct the ANCOVA,

Lab Activity: ANCOVA Using SPSS Step 1: Check all the assumptions for a basic ANOVA for the IVs (factors) for which the effects/differences are of interpretational interests. In this study, we are interested in the treatment effect of Omega-3 fatty acid on the cholesterol level. Therefore, we need to check the data (DV) assumptions with regard to the factor “treatment.” Lab Activity: Check whether the one-way (Omega-3 treatment) ANOVA assumptions are met. Answer: 1. The independence of observations is met by design. 2. The normal distribution (for each group) is met by the skewness values for both groups being less than1.5. 3. The equal variances assumption is met by the SDs of the two groups being very close. Question: If the assumptions are met, move to the next step. If the assumptions are not met, what should you do?

Lab Activity: ANCOVA Using SPSS Step 2. Check whether the covariate and the DV are linearly related. First, use a descriptive graph to examine the linear relationship. Double click the output picture Define

Lab Activity: ANCOVA Using SPSS Step 2. Check whether the covariate and the DV are linearly related. Second, use hypothesis test to examine the linear relationship statistically. If the assumption is not met, stop the ANCOVA analysis or choose other possible covariate(s) if they were planned to be recorded during the design phase. If the assumption is met. Move to the next step.

Lab Activity: ANCOVA Using SPSS Step 3. Check whether the treatment IV and the covariate is statistically unrelated. If the assumption is not met, stop the ANCOVA analysis or choose other possible covariate(s) if they were planned to be recorded during the design phase. If the assumption is met. Move to the next step.

Lab Activity: ANCOVA Using SPSS Step 4. Check whether the “treatment x Covariate” interaction effect is non-significant (equal slopes assumption). First, use descriptive graphs to examine whether the regression coefficients (slopes) are close, i.e., the lines are parallel. Note that if the covariate is categorical, we can use interaction plot to examine the interaction effect. Define Double click the output picture Separate graph for each group

Lab Activity: ANCOVA Using SPSS Step 4. Check whether the “treatment x Covariate” interaction effect is statistically non-significant. Second, use hypothesis test to check whether there is a statistically significant interaction effect. Using General Linear model in SPSS, fit the model including the two main effects (the treatment IV and the covariate) as well as the interaction term (treatment IV*covariate). Check whether the F for the interaction term is non-significant.

Lab Activity: ANCOVA Using SPSS Step 4. Check whether the “IV x Covariate” interaction effect is non-significant. If the non-significant “treatment x covariate” assumption is not met, stop the ANCOVA analysis, or Choose other possible covariate(s) if they were planned to be recorded during the design phase, or If the interaction term is significant and theoretical or practically meaningful, interpret the interaction effect as an ATI (attribute by treatment) design by concluding that the effect of the treatment depends on the value of the covariate. If the assumption is met. Move to the next step.

Lab Activity: ANCOVA Using SPSS Step 5: Conduct the ANCOVA, if conditions for steps 1-4 meet the requirements. In General Linear Model, enter the treatment IV as the fixed factor, and the covariate variable as a covariate, and interpret the results as an ANCOVA (see the next slide). Note that if the covariate is categorical, enter it as another fixed factor. Lab Activity: Compare the result for the one-way ANOVA of Omega-3 effect to that for the ANCOVA.

Lab Activity: Compare the Results One-Way ANOVA vs. ANCOVA Results for One-Way ANOVA Results for ANCOVA The use of the covariate “# of chronic disease” successfully reduced the error variance, increased the F statistic, hence, increased the power of the F test for the treatment effect.

Interpretation of an ANCOVA Results Because the covariate’s association with the DV is viewed as a nuisance, typically it is not interpreted. The interpretation of the effect of the treatment IV is the same as that of a one way ANOVA. The F statistic tests whether the omnibus test is statistically significant. That is, testing whether there is at least one pair of mean difference that is significantly different from zero in the population. If the F test is significant, post-hoc or planned (contrast) independent sample t-tests will be conducted for all possible pairs of group mean differences.