UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing United States Growth Models: what are they and how do they measure accountability? Pete Goldschmidt, Ph.D. 46 th Annual Learning Disability Association Conference February 25-28, 2009 If you choose to use this title slide, simply delete the previous slide (the one-line title version). This will be slide 1 of your presentation.

2/28 Dual Purpose of growth models Monitor School performance Accountability (NCLB/State). Evaluation. Monitor individual student performance Focus on developmental process. Use individual student growth trajectories to provide interventions sooner.

3/28 Accountability - A Tautological Statement Accountability models should hold schools accountable for those things that schools are responsible. Generally we consider those things to be student outcomes. Outcomes can be multi-faceted, but emphasis is on academic performance. Academic performance is usually limited to a few subjects. Academic performance is usually measured by a large scale assessment. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

4/28 All student success is attributable to the current school (in the current year). By extension all student success is attributable to current teacher. Also assumes that students do not bring any “human capital” inputs with them to the school. There are no selection effects – the students in this school are like any other students in any other school in the district/state. One could bring in any other students from any other school and they would perform equally well. There are no compositional effects. Status Accountability Model Irrespective of everything else going on – how is this school performing right now? Assumes that:

5/28 Status Accountability Model Static Aggregate Measures of Performance Year – to – year performance is very consistent, but this consistency is due to factors other than simply school processes.

6/28 Status Accountability Model Aggregating individual student variables inflates their importance – correlations between aggregate performance and school enrollment characteristics are about.75. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

7/28 No Child Left Behind Accountability Model is a Simple Aggregate Static Measure Adequate Yearly Progress (AYP), by construction penalizes large (and) heterogeneous schools. AYP as an accountability model can categorize schools, but does so very imprecisely. Policy makers want the accountability model to provide more than simply a categorization (and at least the correct categorization) but potentially also use the results to inform school improvement. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

8/28 AYP is an Extremely Poor Evaluation Model It reduces achievement information into 4 or 5 broad categories. It ignores the fact that the accumulation of both external and internal factors over time affect current student performance. AYP dichotomous indicator Distributive information within categories lost Change within categories lost Assumes that low performing students demonstrate faster growth than higher performing students – but recognized as equal performance on AYP measure Ecological fallacy – can misguide interventions When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

9/28 Percent Minority by AYP Status

10/28 Status Accountability & Tautology II Internal Factors + External Factors An accountability model should only be based on results that reflect the effects of internal factors. Simple aggregate static measures of student performance judge schools based on both internal and external factors – but are overly influenced by external factors. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

11/28 Aggregate Status Models Evidence suggests that simple means or performance, that aim to capture variation in school indices performance, capture exactly that portion of school performance not controllable by the schools. Example: Raters completed a school quality survey and these survey results were compared with existing measures of school quality. The survey results indicated that school quality can generally be summarized into two factors: External factors – factors outside of school control Internal factors – factors controllable by schools

12/28 Directly Comparing the Relationships Among Indicators Reveals To adjust the slide numbering, do the following: 1.Go to the VIEW menu, MASTER, and select SLIDE MASTER 2.In the lower right, change the number 28 to your number of slides 3.Do not change the character. It generates the auto-numbers. Growth and value added estimates are more highly correlated with internal than external measures.

13/28 Moving Beyond Status Considerations Begin with questions: What do we consider a “good” school to look like? Is there an appropriate assessment system in place? What additional data requirements are there? Is there capacity to utilize various model choices?

14/28 Moving Beyond Status Gain Model School Improvement Model Growth Model Value Added Model

15/28 Gain scores provide a direct estimate of student growth Some argue that gain scores are biased and inherently unreliable, but this is not necessarily true. In fact gain scores can be more reliable than the underlying individual scores. Often the unreliability associated with gain scores is due to small sample sizes and lack of variability in gains among students. It is important to distinguish between true gains and observed gains. Often observed gains are used and these tend to be spuriously (negatively) related to year one scores. Year to year fluctuations may be to great too provide accurate indicators of school performance.

16/28 Gains: t -3 to t 4 – a linear change

17/28 Gains Not Always Linear Multiple occasions allow for a more accurate portrayal of change over time.

18/28 School Improvement (following cohorts) Longitudinal analyses, not panel data Follow schools over time as students pass through: How do this year’s 3 rd graders compare to last year’s third graders? Or how does the 2004 cohort compare to the 2005 cohort?

19/28 Following Cohorts

20/28 Following Students

21/28 Benefits of Longitudinal Modeling Cross sectional vs. Longitudinal Modeling Cross sectional provides contemporaneous relationships Longitudinal Modeling vs. Traditional Repeated Measures ANOVA Don’t need balanced data Can have missing data Model growth directly (not group by time interaction) Treat time continuous rather than categorical More power because not running multiple t-tests Generate individual trajectories EB estimates good because provide trajectories for subjects that may not have enough data to estimate using traditional methods. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

22/28 Benefits and form Advantages of LGPM: Direct measure of growth Do not need complete outcome data for each student Previously demonstrated that growth model estimates are robust to sample sizes down to 30 per school (grade) Growth model estimates are robust to missingness (MCAR, MAR) Test occasion nested within students (level 1) At level 2 we add student characteristics for both the slope and intercept. At Level 3 we add school characteristics for everything assumed to vary across schools. Caveats: Extensive data requirements Scaling issues

23/28 Static results by AYP status

24/28 Achievement growth by AYP status

25/28 Gap closing by AYP status

26/28 SWD Non-SWD Achievement Gap

27/28 Presidents Commission on Excellence in Special Education Report of 2002 Findings:  Of those with “specific learning disabilities,” 80% are there simply because they haven’t learned how to read. Thus, many children identified for special education—up to 40%— are there because they weren’t taught to read. The reading difficulties may not be their only area of difficulty, but it’s the area that resulted in special education placement. Sadly, few children placed in special education close the achievement gap to a point where they can read and learn like their peers. (Presidents Commission Report, 2002, p.2) When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

28/28 Presidents Commission on Excellence in Special Education Report of 2002 Findings:  The ultimate test of the value of special education is that, once identified, children close the gap with their peers. That’s what accountability for results is about. (Presidents Commission Report, 2002, p.3) When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

29/28 Students with Disabilities (SWD) vs. Non-Disabled Students in LAUSD, 2005 STAR Testing To adjust the slide numbering, do the following: 1.Go to the VIEW menu, MASTER, and select SLIDE MASTER 2.In the lower right, change the number 28 to your number of slides 3.Do not change the character. It generates the auto-numbers.

30/28 SWD vs. Non-SWD Achievement Gap Federal law requires states and local districts to improve the performance of students with disabilities on standardized assessments The current measure typically being used to calculate an achievement gap uses the percentage of nondisabled students performing at the proficient and above level vs. the percentage of students with disabilities performing at proficient and above When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

31/28 SWD vs. Non-SWD Achievement Gap This measure tends to obscure the demographic factors both within and between school districts that can affect the performance of students with disabilities In addition, since a performance gap is often the defining characteristic of a disability on the individual level, particularly for disabilities such as SLD, it may not appear particularly innovative or useful to establish an overall achievement gap between population of students with disabilities and nondisabled students in a districts When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

32/28 SWD vs. Non-SWD Achievement Gap Indeed, a few researchers have asserted that that the gap is not closable and that a focus on the gap is counterproductive This assertion ignores the possibility that if there is significant variation between the performance of similar groups of students with disabilities among schools within a school district, it is possible that some portion of this variation is associated with non-cognitive factors such as school quality or instructional programs When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

33/28 SWD vs. Non-SWD Achievement Gap  We could not identify any research attempting to establish an SWD vs. non-SWD achievement gap controlling for other variables that can affect for performance, tracking changes in the gap over time and attempting to identify any variations in the gap between schools within a specific school district. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

34/28

35/28

36/28

37/28 Results Summary Average Achievement in Non-SWD329 SLI322.6 SLD271.7 Average Achievement Growth '02-'03 to '04- '05 per year Non-SWD-1.9 SLI-1.9 SLD2.7 Variance among schools Status Non-SWD18.9 SLI17.0 SLD17.3 Growth Non-SWD5.8 SLI SLD2.5

38/28 Distribution of SWD NON-SWD Gap Closing in Reading

39/28 Gap closing in a single school

40/28 Individual Trajectories Non SWD SWD

41/28 Student Z: Domain Scores

42/28 Student Z Growth

43/28 Student Z

44/28 Student Z

45/28 Grade 4 Student Z

46/28 Grade 5 Student Z

47/28 next presentation Pete Goldschmidt voice fax If you choose to use this end slide, simply delete the previous slide (with no contact information). ©2006 Regents of the University of California