1. Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables March 2012 Presentation to the Association of Education Finance and Policy Conference Matt Johnson Stephen Lipscomb Brian Gill

2. VAMs Used Today Differ in Their Specifications 2 Value-Added Model Student Characteristics Classroom Characteristics Multiple Years of Prior Scores Colorado Growth ModelNo Yes DC IMPACTYesNo FloridaYes New York CityYes No SAS EVAASNo Yes

3.  How sensitive are teacher value-added model (VAM) estimates to changes in the model specification? –Student characteristics –Classroom characteristics –Multiple years of prior scores  How sensitive are estimates to loss of students from sample due to missing prior scores? Research Questions 3

4.  Teacher value-added estimates are not highly sensitive to inclusion of: –Student characteristics (correlation ≥ 0.990) –Multiple years of prior scores (correlation ≥ 0.987)  Estimates are more sensitive to inclusion of classroom characteristics (correlation = 0.915 to.955)  Estimates are not very sensitive to loss of students with missing prior test scores from sample (correlation = 0.992) –Precision increases when two prior scores are used but fewer teacher VAM estimates are produced Preview of Main Results 4

5.  Explore sensitivity to several specifications: –Exclude score from two prior years (Y i,t-2 ) –Exclude student characteristics (X i,t ) –Include class average characteristics  Student data from a medium-sized urban district for 2008–2009 to 2010–2011 school years  All models run using the same set of student observations  Instrument using opposite subject prior score to control for measurement error Baseline Model 5

6. Student LevelClass Level Free or Reduced-Price Meals Disability Gifted Program Participation Lagged Rate of Attendance Lagged Fraction of Year Suspended Race/Ethnicity Gender Age/Behind Grade Level Average Prior Achievement in Same Subject Standard Deviation of Lagged Achievement Number of Students in Classroom Student and Class Characteristics 6

7. Correlation of 6th-Grade Teacher Estimates Relative to Baseline VAM Specification 7 Math (n = 87) Reading (n = 99) Exclude Student Characteristics0.9900.996 Exclude Prior Score from t-20.9930.987 Exclude Student Characteristics and Prior Score from t-2 0.9780.970 Add class average variables0.9550.915 Baseline: Student Characteristics and Prior Scores from t-1 and t-2 Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

8. Exclude Student Characteristics 1st (Lowest)2nd3rd4th 5th (Highest) Baseline Model 1st (Lowest)955000 2nd590500 3rd0575200 4th00207010 5th (Highest)0001090 Percentage of 6th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification 8 Findings are based on VAM estimates for 99 reading teachers in grade 6 from 2008–2009 to 2010–2011 for a medium-sized, urban district. Correlation with baseline = 0.996.

9. Baseline + Class Average Characteristics 1st (Lowest)2nd3rd4th 5th (Highest) Baseline Model 1st (Lowest)8020000 2nd5653000 3rd1510501510 4th05106520 5th (Highest)00102070 Percentage of 6th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification 9 Findings are based on VAM estimates for 99 reading teachers in grade 6 from 2008–2009 to 2010–2011 for a medium-sized, urban district. Correlation with baseline = 0.915.

10.  Benefits of including two prior years: –More accurate measure of student ability –Increase in precision of estimates  Costs of using two prior years: –Students with missing prior scores dropped –Some teachers dropped from sample  Relative magnitude of costs/benefits? One or Two Years of Prior Scores? 10

11.  Estimate two VAMs using one year of prior scores –First VAM includes all students –Second VAM restricts sample to students with nonmissing second prior year of scores  Correlation between teacher estimates: 0.992  Percentage of students dropped: 6.2  Percentage of teachers dropped: 3.9  Net increase in precision from using two prior years –Increase in average standard error of estimates: 2.3% when students with missing scores are dropped –Decrease in average standard error of estimates: 7.6% when second year of prior scores added One or Two Years of Prior Scores? 11

12. Mathematica ® is a registered trademark of Mathematica Policy Research.  Please contact –Matt Johnson MJohnson@mathematica-mpr.com –Stephen Lipscomb SLipscomb@mathematica-mpr.com –Brian Gill BGill@mathematica-mpr.com For More Information 12