Example of Models for the Study of Change David A. Kenny December 15, 2013

2 Example Data An Honors Thesis done by Allison Gillum of Skidmore College supervised by John Berman on the effects of an semester-long class on the Environment on Environmental Responsible Behaviors. Pretest-Posttest Design 41 Treated and 199 Controls 2 Treated classes and 8 Control classes. No clustering effect due to class. Outcome: Environmentally Responsible Behaviors (ERB), a 12 item scale ranging for 1 to 7. For latent variable analyses, 3 parcels of 4 items were created.

3 Models –Controlling for Baseline Simple Allowing for Unreliability at Time 1 –Change Score Analysis Raykov LCS Kenny-Judd –Standardized Change Analysis Types –Univariate (average of 12 items) –Latent Variable (3 parcels of 4 items)

4 Latent Variable Measurement Models Unconstrained –  2 (9) = 13.22, p =.153 – RMSEA = 0.044; TLI =.992 Equal Loadings –  2 (11) = 18.63, p =.068 – RMSEA = 0.054; TLI =.988 The equal loading model has reasonable fit.

5 Pretest Difference Mean for Controls: 4.79 Mean for Treateds: 5.21 A mean difference of 0.42 t(238) = 3.191, p =.002 d = 0.64, a moderate effect size There is a difference at the pretest! The mean difference on latent variable at time 1 is 0.45.

6 More on the Pretest Difference Likely more environmentally conscious students more likely to take an environmental course. Would you expect the difference to persist (CSA) or narrow (CfB)?

7 Controlling for Baseline: Univariate A beneficial effect of the course on the outcome: Z = 3.992, p <.001  = (expect a narrowing of the gap) We shall see that this is the largest estimate of the treatment effect.

8

9 CfB: Measurement Error in the Pretest Coefficient alpha of.872 for pretest Lord-Porter Correction Convert ( )/( ) =.866 Adjusted pretest score (M T is the mean for the Treated and M C is the mean for the controls):  (X 1 – M T ) + M T  (X 1 – M C ) + M C b = , Z = , p <.001

10

11 Williams & Hazer Method Set X 1 = X 1T + E 1 Fix the variance of E 1 to (1 -  )s Y1 2 or ( )(0.614) = b = , Z = 3.233, p =.001 (with  =.897) Same estimates of b and  as Lord- Porter (standard error a bit different)!

12

13 Controlling for Baseline: Latent Variables –b = , Z = 4.124, p <.001 –  = (surprisingly relatively low) –Cannot directly compare estimates to the univariate analysis.

14

15 Change Score Analysis: Univariate –All 3 methods (see next 3 slides) show a beneficial effect: , Z = 2.619, p =.009 –Smallest effect of any analysis, –Note that the Treateds improve (0.0874), and the Controls decline ( ).

16 Raykov

17 LCS

18 Kenny-Judd

19 Change Score Analysis: Latent Variables –All 3 methods show a beneficial effect , Z = 2.964, p <.003 –Again, you cannot directly compare the univariate and latent variable results. –Smallest effect of any latent variable analysis.

20 Raykov

21 LCS

22 KJ

23 Standardized Change Score Analysis: Univariate Analysis Residual variance decreases slightly over time (but not significantly, p =.31) Time 1: 0.59 Time 2: 0.52 Effect:.3078, Z = 2.775, p =.006 Recalibrated to units of Time 2: , Z = 2.826, p =.005.

24 SCSA

25 SCSA-Y 2

26 Standardized Change Score Analysis: Latent Variables Residual variance decreases slightly over time (but not significantly, p =.30) Time 1: 0.56 Time 2: 0.52 b = , Z = 3.116, p =.002 Units of Time 2: b = , Z = 3.194, p =.001

27 SCSA

28 SCSA-Y 2

29 Summary of Univariate Effects CfB: CfB with Reliability Correction: CSA: SCSA:

30 Summary of Latent Variable Effects CfB: CSA: SCSA:

31 What Estimate Would I Report? CSA Latent Variable: (Z = 2.964, p <.003) No reason to think that the factors that created the Time 1 difference to change. Note too the variance does not change. Others might respectfully disagree.