1. UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Moving Beyond Status: What We Gain from Growth Models Pete Goldschmidt, Ph.D. Washington Educational Research Association Seminar on Growth Modeling Renton, WA – June 2, 2006 If you choose to use this title slide, simply delete the previous slide (the one-line title version). This will be slide 1 of your presentation.

2. 2/52 Beyond Status - Overview Purpose What’s wrong with status? The reality of the schooling process Taking the data structure into account Gains – the next best thing to be there Longitudinal models – the journey is the reward

3. 3/52 Purpose What are we trying to evaluate? Schools Programs Teachers Students Want to be able to examine outcomes and determine whether is performing “adequately” or not. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

4. 4/52 Schools – Monitoring Performance We expect that schools demonstrating poor performance – as indicated by student outcomes – will/can change/improve those processes that are not effectively generating the desired outcomes.

5. 5/52 Status as Indicator of Performance

6. 6/52 Schools and Static Aggregate Student Outcomes Internal Factors + External Factors Aggregating individual student variables inflates their importance – correlations between aggregate performance and school enrollment characteristics about.75 Simple aggregate static measures of student performance judge schools based on both internal (C&I, etc) and external factors (student demographics) – but are overly influenced by external factors. School Performance should only be based on results that reflect the effects of internal factors. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

7. 7/52 Relationship of annual “Performance Indicator” with annual school mean Mathematics performance

8. 8/52 Relationship of annual “Performance Indicator” with annual school mean Mathematics performance “Performance indicator”… [ .. – .j]; where  equals percent free lunch and j indexes school.

9. 9/52 Directly Comparing the Relationships Among Indicators Reveals To adjust the slide numbering, do the following: 1.Go to the VIEW menu, MASTER, and select SLIDE MASTER 2.In the lower right, change the number 28 to your number of slides 3.Do not change the character. It generates the auto-numbers.

10. 10/52 Realties of Schooling Students bring varying amounts of human capital Students attend different schools (and different classrooms) Students acquire skills and knowledge over time Current exam results reflect accumulation of effects resulting from all of the above

11. 11/52 Effect of neglecting data structure I Where: j = school i = student y= test score x = hrs studying

12. 12/52 Relationship between hrs of studying and test scores Use individual observations on students

13. 13/52 Relationship between hrs of studying and test scores Use school aggregates (averages) of hours of studying and test scores

14. 14/52 Relationship between hrs of studying and test scores Use individual observations at each school

15. 15/52 Relationship between hrs of studying and test scores

16. 16/52 Data Structure I Even in cross–sectional analyses, natural data structure is important. Cross-sectional studies also heavily dependent on counter-factuals Potential confounding factors (PCFs) that may account for the observed relationships PCFs are often external factors beyond schools’ control.

17. 17/52 Reality of Schooling – a process Student results at any one time are a function: Ait = f(Bit, Pit, Sit, Iit, Eit), (1) where for student i at time t Achievement A, is some function of: Student Background (B) Peer and other influences (P) School inputs (S) Innate ability (I) And luck (E). Model is cumulative and past inputs may affect current Achievement. Also would need independent measure of innate ability, gathered before any S has occurred. These are tremendous data requirements, and generally infeasible.

18. 18/52 Reality of Schooling – a process If we assume that (1) holds for any time t, then we can consider change in achievement from t to t`. Ait` - Ait = f(.) Then by simply adding Ait to both sides, we get a familiar model: Ait` = f(Bit`-t, Pit`-t, Sit`-t, Iit, Ait, Eit )(2) Still lack measure of I, and omitting variables will increase the effect of included variables if there is a correlation between the omitted variable and the included variables. However: once student B is included in the model the effect of omitting I is small; and, effect lessoned because include Ait. Also, remaining variables measured contemporaneously, but this is generally not too problematic since only going back from t` to t.

19. 19/52 A Process -> Gains By moving away from cross-sectional analyses begin to address process of schooling and pitfalls of PCFs. Gains Ait’ – Ait = f(.) Covariance adjustment Ait’ = f{Ait + (.)}

20. 20/52 Gains VS. Covariance Adjustment models Some prefer CAM to gain scores because Regression to the mean Bias Reliability ----- Regression to the mean only an issue if selection based on pre-test. Neither gains nor CAM biased under random assignment Gains scores not inherently unreliable Reliability Precision

21. 21/52 Example – Information Gain from Gains: unmatched data

22. 22/52 Gains: t -3 to t 4 – a linear change

23. 23/52 Gains Not Always Linear Multiple occasions allow for a more accurate portrayal of change over time.

24. 24/52 Benefits of Longitudinal Modeling Cross sectional vs. Longitudinal Modeling Cross sectional provides contemporaneous relationships Issues with experimental design Longitudinal Modeling vs. traditional Repeated Measures ANOVA Don’t need balanced data Can have missing data Model growth directly (not group by time interaction) Treat time continuous rather than categorical More power because not running multiple t-tests Generate individual trajectories EB estimates good because provide trajectories for subjects that may not have enough data to estimate using traditional methods.

25. 25/52 Longitudinal Models Can be: Trend analysis Sequential cohorts Panel models Growth models Value Added Models Layered models Latent growth models Growth models incorporating panel and cohort growth Binomial growth models Survival Analysis - dichotomous outcomes

26. 26/52 Requirements for Longitudinal Modeling Intent ( Quasi) Experimental design issues Structure Data Time If using a panel model need to have constant subject ID over time. Missing data and attrition. Time Every subject does not need data at every time point. Subjects need not be measured at the same time points. Time need not be in equal intervals. Outcomes Meaning of outcome over time needs to be constant. Change in outcome needs to make sense. Number of occasions –minimally 2, but really 3.

27. 27/52 Longitudinal models can address many aspects of the schooling process Move beyond pre-post evaluation which gives a measure of effectiveness at a single point in time relative to baseline Individual trajectories Sub-groups Evaluate programs School effects Value Added Can address the process of change Longevity of effects Effect of variation in length of treatment

28. 28/52 The Basic model and the information it provides Y ti =  0i +  1i T ti + e ti {within student model}  0i =  00 + u oi {between student model}  1i =  10 + u 1i Where Y ti = outcome for subject i at time t.  0i = status of subject i when T =0.  1i = average change per unit of T for subject i.  00 = grand mean of status when T =0.  10 = grand mean change per unit of T. e ti = within person error. u oi = unique increment to grand mean of status for subject i u 1i = unique increment to grand mean of change for subject i

29. 29/52 Basic Model = Longitudinal Growth Panel Model (LGPM) Longitudinal Panel Design Keep track of students’ achievement form one grade to the next E.g., collect achievement scores at Grades 2, 3, 4, and 5 for students in a school Focus on students’ developmental processes What do students’ growth trajectories look like?

30. 30/52 Choice of Metric and LGPM IRT-based scale scores Vertically equated scores across grades and years Theoretically represent growth on a continuum that can measure academic progress over time Change from year to year is an absolute measure of academic progress Change represents a relative position from year to year not absolute growth in achievement Relative standing compared to a norming population Scale Scores : Normal Curve Equivalents : Won’t even consider: - Percentile Ranks - Grade Equivalents

31. 31/52 Metrics and Purpose (LGPM) The metric matters depending on the intended purpose More critical for absolute decisions Less critical for relative decisions

32. 32/52 Another View – Longitudinal School Productivity Model (LSPM) Multiple-cohorts design (Willms & Raudenbush, 1989; Bryk et. al., 1998) Monitor student performance at a school for a particular grade over years E.g., collect achievement scores for 3rd grade students attending a school in 1999, 2000, and 2001 Focus on schools’ improvement over subsequent years

33. 33/52 LSPM Unconditional School Improvement Model Level-1 (within-cohort) model: Yijt = βjt0 + rijt * βjt0 : estimates of performance for school j (j = 1,.., J) at cohort t (t = 0,1,2,3,4) Level-2 (between-cohort, within-school) model: βjt0 = j0 + j1Timetj + ujt * j0 : status at the first year (i.e., Timetj = 0) or initial status for school j * j1 : yearly improvement / growth rate during the span of time for school j Level-3 (between-school) model: j0 = 00 + Vj0 * 00 : grand mean initial status j1 = 10 + Vj1 * 10 : grand mean growth rate

34. 34/52 Choice of Metric and LSPM Metric choice less critical in LSPM because scale is modeled out (grade is held constant while cohorts pass through

35. 35/52 Realty of Schooling – Data Structure II YearGrade Cohort 20012 200223 2003234 200423451 200523452 200623453 4 5 6

36. 36/52 Connected Cohorts - 3 years

37. 37/52 Modeling Individual Student Growth Implicitly Taking Cohort membership into Account

38. 38/52 Longitudinal models Explicitly taking both Panel and Cohorts into Account LGPM models decompose growth into: Test occasions within students Students within schools and Schools What might the effect of changing cohorts be on growth? If the cohort a student is in affects student performance then need to consider students Within cohorts and schools

39. 39/52 LSPM – Cohorts Improving Over Time

40. 40/52 LGPM – Individual Student Growth Trajectories do not Demonstrate Improvement Over Time

41. 41/52 LGPM Vs. LSPM Address different aspects of student and school progress For the individual student the former is key, while for schools, both are important (but the latter may be more important in terms of school improvement). School ranking based on cohort models and panel models are only moderately correlated. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

42. 42/52 A 4 Level Model Combining Panel and Cohort Growth Reveals:

43. 43/52 Relationship Between Status Panel and Cohort Growth

44. 44/52 Model subgroups directly using Longitudinal Binomial Panel Model (LBPM) Focusing on achievement gaps and the likelihood of meeting the target in 2013- 2014 and utilizing an accountability model not intended for evaluation. Can use a longitudinal binomial growth model that simply models the probability over time that a subgroup will be proficient. Does not require a vertically equated metric. Provides a clear picture of current status. Provides a direct estimate of progress over time. Demonstrates where subgroups are and are going.

45. 45/52 Data Structure for LBPM Year 1GirlBoy Low SES215/245234/257 Not Low SES300/345300/330 Year 2GirlBoy Low SES220/249230/260 Not Low SES304/351304/326 Year 3GirlBoy Low SES215/232243/260 Not Low SES306/3347301/330 Cells by year

46. 46/52 LBPM Results for Schools

47. 47/52 LBPM: Will schools meet 2013-2014 goal? Results indicate that despite “good” progress only 26% of schools will meet the 100% proficiency target by 2013-2014 When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

48. 48/52 Relationship Between School Quality Indicators and the Probability of Proficiency To adjust the slide numbering, do the following: 1.Go to the VIEW menu, MASTER, and select SLIDE MASTER 2.In the lower right, change the number 28 to your number of slides 3.Do not change the character. It generates the auto-numbers.

49. 49/52 Growth Vs. Value Added Models All Value Added Models are Growth Models Growth models - interest lies in “explained” portion of the model Value Added models – interest lies in “unexplained” portion of the model

50. 50/52 Conclusions Simple static aggregates of student performance restate what district staff and school principals already know (at best) But likely suffer from ecological fallacy Suffer from effects of PCFs (external non- school controllable factors) And do not address school processes. Whether growth (change) is taking place Who is demonstrating growth What is related to growth Is it enough growth When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

51. 51/52 Conclusions Need to be conscientious of the type of change being estimated and the type that might be desired Cohort vs. Panel Cohort and Panel Binomial Pass/fail models As evaluation/monitoring of schools, programs or students use more complex models need more care in interpreting and presenting results When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

52. 52/52 Pete Goldschmidt voice email 310.794.4395 Goldschmidt@cse.ucla.edu If you choose to use this end slide, simply delete the previous slide (with no contact information). ©2006 Regents of the University of California