Measuring Change in Two-Wave Studies David A. Kenny

The Focus Here in Measuring Change To understand the causes of change, e.g., intervention analysis. Not: To understand the variable of interest and how it changes. To measure who changes more or less in the sample.

A Bit of Notation Two times X a potential cause of change in Y Time 1 measure: Y1 Time 2 measure: Y2 X a potential cause of change in Y easier to think of as a dichotomy (0 and 1) but not required X may be randomized or not

Change Score Analysis (CSA) Time 2 minus Time 1 Seen by some as a naïve model. Can compute the difference score more indirectly (keep Time 1 in the model).

Control for Baseline (CfB) Adjust out for the effect of Time 1, i.e. empirically weight Time 1 and not fix its coefficient to one. Generally preferred method.

Equations Change Score Analysis (CSA) Y2 – Y1 = a + bX + e Control for Baseline (CfB) Y2 = a + bY1 + bX + e (will refer to b later)

The Coefficient for Y1 Note that we can rewrite CSA as Y2 = a + 1Y1 + bX + e Thus, CSA fixes the coefficient for Y1 to one, whereas CfB empirically estimates it. For CfB usually, though not always, b is less than 1. In fact, if the variances of Y1 and Y2 are equal, b cannot be bigger than 1 (given that b = 0).

Lord’s Paradox CSA and CfB yield different results when b is different from 1 and there is correlation between X and Y1.

Why Different? X=1 Y2 X=0 Time 1 Time 2

How Different? X=1 Y2 gap X=0 Time 1 Time 2

CSA: The Gap Persists X=1 Y2 X=0 Time 1 Time 2

CfB: The Gap Narrows (b < 1) X=1 Y2 X=0 Time 1 Time 2

CSA vs. CfB X=1 CfB CSA Y2 X=0 Time 1 Time 2

Randomized Studies Lord’s paradox happens only when there is some association between X and Y1, i.e. a gap. If there is randomization, there is no gap in the population. By chance, there is some gap in the sample.

Estimate of the Treatment Effect (b) MD1: The mean difference between treatment groups at time 1. MD2: The mean difference between treatment groups at time 2. CSA: b = MD2 – MD1 CfB: b = MD2 - bMD1 If b < 1, then CfB adjusts less than CSA.

and so if b < 1… If MD1 is positive (1s start ahead of 0s), more gets subtracted using CSA, resulting in a smaller b than CfB. If MD1 is negative (1s start ahead of 0s), more gets added using CSA, resulting in a larger b than CfB.

Strategy In randomized studies and in non-randomized studies in which the gap is small, controlling for the Y1 should be done as it almost always has more power than change score analysis.

What Is Meant by “Small” Suggestion If the effect of X on Y1 is not statistically significant and the r between X with Y1 is less than .075 in absolute value or d of less than 0.15 in absolute value, then the gap is small.

But what if there is a gap at Time 1? The bigger the gap, the greater the divergence between the two methods. To choose the right analysis, we need to know why there is that gap: What causes that gap and how does it change?

Report the Size of the Gap! In non-randomized studies, we need to know the size of the gap. If X is a dichotomy, report d. If X is continuous, report the correlation of X with Y1.

The Assignment Variable See Judd & Kenny book (1981) Z is an unmeasured variable that explains the gap at time 1. It does so because it determines X (selection), but it also associated with the Y1. If Z were measured and controlled for, there would be no gap.

Model for CSA Think of Z as a dichotomy, with levels 1 and 2. We measure the mean of Y1 and Y2 for different values of Z. We compute the gap at each time (controlling for X). The gap is the same at both times.

Continuous Z Z need not be a dichotomy, but the general points still hold. The effect of Z on Y1 or g is the same as the effect of Z on Y2, controlling for X.

Example Z is gender. Proportionally more men in the treated group than the control group. Gender differences in Y are same at the two times (controlling for X).

Model for CfB Z causes or is correlated with causes of Y1. However, Z has no effect on Y2 once Y1 is controlled. Y1 serves as a complete mediator of the Z-Y2 relationship.

b

Example Schools try to select low performing students into a the treatment group. Z is child’s ability at time 1. As Y1 is a measure of ability, it explains any relationship between Z and Y2 (path from Z to Y2 is zero).

Other Presentations Change Score Analysis Standardized Change Score Analysis Controlling for the Baseline Analysis Power in the Study of Change