1. Longitudinal Experiments Larry V. Hedges Northwestern University Prepared for the IES Summer Research Training Institute July 28, 2010

2. What Are Longitudinal Experiments? Longitudinal experiments are experiments with repeated measurements of an outcome on the same people Examples Experiments with immediate and delayed posttests Experiments that track individuals over many performance periods (e.g., school years) Experiments that intend to impact growth rate Experiments that make repeated measurements of behavior (e.g., teacher behavior) to increase precision of measurement

3. Why Do Longitudinal Experiments? Three reasons for doing longitudinal experiments 1.More than one discrete endpoint is of interest (e.g., immediate vs delayed outcome) 2.Several measures of the outcome are necessary to increase precision or reduce variation (e.g., teacher behavior is averaged over many occasions) 3.The time course of treatment effect (growth trajectory) is of interest (e. g., an intervention is intended to increase the rate of vocabulary acquisition in preschool children) In all three cases, linear combinations of outcomes may be of interest

4. Why Do Longitudinal Experiments? Unless different outcomes are being compared, there is no need to use longitudinal methods! But, if different outcomes are being compared, outcomes are not independent Thus, longitudinal methods must be used

5. Modeling Longitudinal Experiments We can describe models for longitudinal experiments via ANOVA or HLM notation We can analyze longitudinal experiments via either ANOVA or HLM There are big advantages to using HLM notation for these models There are even bigger advantages to using HLM for analyses of these models Hence we will primarily use HLM notation in our discussion of longitudinal experiments

6. Discrete Endpoints The design will typically have at least three levels Measures are nested (clustered) within individuals, individuals are nested (clustered) within schools Level 1 (measures within individuals) Level 2 (individuals within schools) Level 3 (schools) Let Y ijk, the observation on the k th measure for the j th person in the i th school

7. Discrete Endpoints, Schools Assigned (No Covariates) Level 1 (measure level) Y ijk = β 0ij + ε ijk ε ~ N(0, σ W 2 ) Level 2 (individual level) β 0ij = γ 00i + η 0ij η ~ N(0, σ I 2 ) Level 3 (school level) γ 00i = π 00 + π 01 T i + ξ 0i ξ ~ N(0, σ S 2 ) Where we code the (centered) treatment T j = ½ or - ½, so that π 01 is the ANOVA treatment effect

8. Discrete Endpoints (Unconditional Model) Note that the ε ijk ’ s are not just measurement errors but also contain differences between outcomes for each individual Similarly the η 0ij ‘s are between individual differences in these quantities Then the ξ 0i ‘s are between-school differences on these quantities That makes the unconditional model difficult to interpret

9. Discrete Endpoints, Schools Assigned (Comparing Early and Delayed Outcome) Level 1 (measure level) Y ijk = β 0ij + β 1ij D ijk + ε ijk ε ~ N(0, σ W 2 ) Level 2 (individual level) β 0ij = γ 00i + η 0ij η ~ N(0, Σ I ) β 1ij = γ 10i + η 1ij Level 3 (school level) γ 00i = π 00 + π 01 T i + ξ 0i ξ ~ N(0, Σ S ) γ 10i = π 10 + π 11 T i + ξ 1i Note that the η 0ij ’s and η 1ij ’s can be correlated as can the ξ 0i ’s and ξ 1i ’s

10. Discrete Endpoints, Individuals Assigned (Comparing Early and Delayed Outcome) Level 1 (measure level) Y ijk = β 0ij + β 1ij D ijk + ε ijk ε ~ N(0, σ W 2 ) Level 2 (individual level) β 0ij = γ 00i + γ 01i T i + η 0ij η ~ N(0, Σ I ) β 1ij = γ 10i + γ 11i T i + η 1ij Level 3 (school level) γ 00i = π 000 + ξ 00i γ 01i = π 010 + ξ 01i ξ ~ N(0, Σ S ) γ 10i = π 100 + ξ 10i γ 11i = π 110 + ξ 11i Note that the η 0ij ’s and η 1ij ’s can be correlated as can the ξ ’‘s and ξ ’s

11. Discrete Endpoints (Comparing Early and Delayed Outcome) Note that, in this model, the ε ijk ’ s can be interpreted as measurement errors Similarly the η 0ij ‘s are between individual differences in these quantities and the intraclass correlation ρ I = σ I 2 /(σ s 2 + σ I 2 + σ W 2 ) is a true (individual level) reliability coefficient Then the ξ 0i ‘s are between-school differences on these quantities and the intraclass correlation ρ S = σ S 2 /(σ s 2 + σ I 2 + σ W 2 ) is a true (school level) reliability coefficient

12. Discrete Endpoints, Schools Assigned (Comparing Early and Delayed Outcome) Covariates can be added at any level of the design But remember that covariates must be variables that cannot have been impacted by treatment assignment Thus time varying covariates (at level 1) are particularly suspect since they may be measured after treatment assignment

13. Average of Several Measures The design will typically have at least three levels Measures are nested (clustered) within individuals, individuals are nested (clustered) within schools Level 1 (measures within individuals) Level 2 (individuals within schools) Level 3 (schools) Let Y ijk, the observation on the k th measure for the j th person in the i th school with p measures per individual

14. Average of Several Measures (Treatment Assigned at the School Level) Level 1 (measure level) Y ijk = β 0ij + ε ijk ε ~ N(0, σ W 2 ) Level 2 (individual level) β 0ij = γ 00i + η 0ij η ~ N(0, σ I 2 ) Level 3 (school level) γ 0i = π 00 + π 01 T i + ξ 0i ξ ~ N(0, σ S 2 ) Where we code the (centered) treatment T j = ½ or - ½, so that π 01 is the treatment effect

15. Average of Several Measures (Treatment Assigned at the Individual Level) Level 1 (measure level) Y ijk = β 0ij + ε ijk ε ~ N(0, σ W 2 ) Level 2 (individual level) β 0ij = γ 00i + γ 01i T ij + η 0ij η ~ N(0, σ I 2 ) Level 3 (school level) γ 00i = π 00 + ξ 0i ξ ~ N(0, Σ S ) γ 01i = π 01 + ξ 1i Where we code the (centered) treatment T j = ½ or - ½, so that π 01 is the treatment effect

16. Average of Several Measures Note that, in this model, the ε ijk ’ s can be interpreted as like (item level) measurement errors Then the β 0ij ‘s can be interpreted as individual level measures (for the j th person in the i th school) Thus the η 0ij ‘s are between individual differences in these quantities and the quantity ρ I = σ I 2 /(σ s 2 + σ I 2 + σ W 2 /p) is a true (individual level) reliability coefficient Then the ξ 0i ‘s are between-school differences on these quantities and the quantity ρ S = σ S 2 /(σ s 2 + σ I 2 + σ W 2 /p) is a true (school level) reliability coefficient

17. Growth Trajectories The problem of fitting growth trajectories is more complicated It requires choosing a form for the growth trajectories It also requires choosing a form for the model of individual differences in these growth trajectories Many forms are possible, but polynomials are conventional for two reasons: Any smooth function is approximately a polynomial (Taylor’s Theorem) Polynomials are simple

18. What is a Polynomial Model? Y ijk = β 0ij + β 1ij t ijk + β 2ij t ijk 2 + β 3ij t ijk 3 + ε ijk Typically, t ijk is a measure of time at the measurement for the j th person in the i th school at the t th measurement We typically center the measurements at some point for convenience (often the middle of the time span) Centering strategy determines the interpretation of the coefficients of the growth model Note that the measurements do not have to be at exactly the same time for each person

19. Understanding a Polynomial Model Y ijk = β 0ij + β 1ij t ijk + β 2ij t ijk 2 + β 3ij t ijk 3 + ε ijk How do we interpret the coefficients? β 0ij is the intercept at the centering point β 1ij is the linear rate of growth at the centering point β 2ij is the acceleration (rate of change of linear growth) at the centering point β 3ij is the rate of change of the acceleration (often negative leading to flattening out of growth curves at the extremes)

20. Understanding a Polynomial Model Consider the quadratic growth model to understand acceleration with mean centering Thus you can see that the linear growth rate at time t is In other words, the linear growth rate increases with t and the only place where the linear growth rate is β 1ij is the middle

21. Understanding a Polynomial Model Thus β 1ij is the linear rate of growth at the centered value (here, the middle) If β 2ij > 0, the linear growth rate will be larger above the centered value and smaller below the centered value Centering at other values than the middle can make sense if that is where growth trajectory is of interest and if the model fits the data For example, centering at the end gives coefficients with interpretable rates at the end of the growth period

22. Understanding a Polynomial Model Consider the quadratic growth model to understand acceleration with mean centering Thus you can see that the acceleration at time t is In other words, the acceleration increases with t and the only place where the acceleration is β 2ij is the middle

23. Understanding a Polynomial Model Thus β 2ij is the acceleration of growth at the centered value (here the middle) If β 3ij > 0, the acceleration will be larger above the centered value and smaller below the centered value Centering at other values than the middle can make sense if that is where growth trajectory is of interest and if the model fits the data For example, centering at the end gives coefficients with interpretable rates at the end of the growth period

24. No Growth (Centered) β 0 = 5, β 1 = 0.00, β 2 = 0.00, β 3 = 0.00

25. Linear Growth (Centered) β 0 = 5, β 1 = 1, β 2 = 0.00, β 3 = 0.00

26. Quadratic Growth (Centered) β 0 = 5, β 1 = 1, β 2 = 0.05, β 3 = 0.00

27. Cubic Growth (Centered) β 0 = 5, β 1 = 1, β 2 = 0.05, β 3 = -0.01

28. Linear, Quadratic, and Cubic Growth (Centered) β 0 = 5, β 1 = 1, β 2 = 0.05, β 3 = -0.01,

29. Selecting Growth Models Several considerations are relevant in selecting a growth model First is how many repeated measures there are: The maximum degree is one less than the number of measures (linear needs 2, quadratic needs 3, etc.) However the estimates of growth parameters are much better if there are a few additional degrees of freedom But the most important consideration is whether the model fits the data! Unfortunately, this is not always completely unambiguous

30. Selecting Growth Models Individual growth trajectories are usually poorly estimated HLM models estimate average growth trajectories (via average parameters) and variation around that average: These are much more stable Estimates of individual growth curves can be greatly improved by using empirical Bayes methods to borrow strength from the average This may make sense if there all the individuals in the groups are sampled from a common population It can be problematic if some individuals are dramatically different

31. Selecting Analysis Models One issue is selecting the growth model to characterize growth A different, but related, issue is selecting how treatment should impact growth Should it impact linear growth term? Should it impact the acceleration? Which impact is primary? How does looking at multiple impacts weaken the design? What if impacts are in opposite directions?

32. Longitudinal Experiments Assigning Treatment to Schools In the language of experimental design, adding repeated measures adds another factor to the design: A measures factor The measures factor is crossed with individuals, treatments, and clusters Schools are nested within the treatment factor and individuals are nested within school by treatments Repeated measures analysis of variance can be used to analyze these designs, but we will not pursue that point of view Instead we will use the HLM notation

33. Longitudinal Experiments Assigning Treatment To Schools Level 1 (measures) Y ijk = β 0ij + β 1ij t ijk + β 2ij t ijk 2 + ε ijk Level 2 (individuals) β 0ij = γ 00j + η 0ij η ~ N(0, Σ I ) β 1ij = γ 10j + η 1ij β 2ij = γ 20j + η 2ij Level 3 (schools) γ 00j = π 00 + π 01 T i + ξ 0j ξ ~ N(0, Σ S ) γ 01j = π 10 + π 11 T i + ξ 1j γ 20j = π 20 + π 21 T i + ξ 2j

34. Longitudinal Experiments Assigning Treatment To Schools This model has three trend coefficients in each growth trajectory Note that there are 3 random effects at the second and third level This means that 6 variances and covariances must be estimated at each level This may require more information to do accurately than is available at the school level It is often prudent to fix some of these effects because they cannot all be estimated accurately

35. Longitudinal Experiments Assigning Treatment Within Schools In the language of experimental design, adding repeated measures adds another factor to the design: A measures factor The measures factor is crossed with individuals, treatments, and clusters The treatment factor is crossed with schools and individuals are nested within school by treatments Repeated measures analysis of variance can be used to analyze these designs, but we will not pursue that point of view

36. Longitudinal Experiments Assigning Treatment Within Schools Level 1 (measures level) Y ijk = β 0ij + β 1ij t + β 2ij t 2 + ε ijk ε ~ N(0, σ W 2 ) Level 2 (individual level) β 0ij = γ 00j + γ 01j T j + η 0ij η ~ N(0, Σ C ) β 1ij = γ 10j + γ 11j T j + η 1ij β 2ij = γ 20j + γ 21j T j + η 2ij Level 3 (school level) γ 00j = π 00 + ξ 00j ξ a0 ~ N(0, Σ S ) γ 01j = π 10 + ξ 01j ξ a1 ~ N(0, Σ T x S ) γ 10j = π 00 + ξ 10j γ 11j = π 00 + ξ 11j γ 20j = π 00 + ξ 20j γ 21j = π 00 + ξ 21j

37. Longitudinal Experiments Assigning Treatment Within Schools This model has three trend coefficients in each growth trajectory Note that there are 6 random effects at the third level This means that 15 variances and covariances must be estimated at the third level This requires a great deal of information to do accurately It is often prudent to fix some of these effects because they cannot all be estimated accurately However there is some art in this, and sensitivity analysis is a good precaution

38. Longitudinal Experiments Covariates can be added at any level of the design But remember that covariates must be variables that cannot have been impacted by treatment assignment Thus time varying covariates (at level 1) are particularly suspect since they may be measured after treatment assignment

39. Power Analysis Power computations for longitudinal experiments are doable, but depend on parameters that may not be well known For example reliability of trend coefficients When parameters such as these are known, the computations are straightforward, but there is relatively little information about them that can be used for planning To make matters worse, the values of some parameters (such as reliability) depend on the number of measures Thus it is often necessary to rely on values of variance components

40. Power Analysis Still some generalizations are possible Power increases with the number of measures Power increases with the length of time over which measures are made (except for β 0ij ) Power increases with the precision of each individual measure These factors impact different trend coefficients differently Clustering increases the complexity of computations

41. Power Analysis Pilot data (or data from related studies, perhaps non-experimental ones) is more important in planning longitudinal experiments Longitudinal experiments to look at growth trajectories are attractive, but this is an area at the frontier of practical experience Research is ongoing to produce better methods for power analysis of longitudinal experiments that will be practically useful

42. Good Luck!