1. The Impact of School Improvement Grants (SIG) on Student Outcomes:
Findings from a National Evaluation Using a Regression Discontinuity Design
SREE Spring Conference
March 3, 2017
Lisa Dragoset • Jaime Thomas • Mariesa Herrmann • John Deke • Susanne James-Burdumy • Dara Lee Luca
Based on the Institute of Education Sciences (IES) report by Dragoset et al. (2017). Funding from IES is gratefully acknowledged. Many thanks to Tom Wei (our IES project officer) for his valuable input throughout the study.

2. The SIG Program
Aimed to increase achievement in low-performing schools
Fiscal year 2009 appropriations were roughly $3.5 billion
Participating schools implemented one of four school intervention models and the practices it prescribed
Each participating school received $50,000 to $2 million per year for three years

3. School Intervention Models
Key activities
Transformation
Replace the principal
Implement evaluation system that accounts for student achievement growth
Implement comprehensive instructional reforms, increase learning time, create community-oriented schools, have operational flexibility
Turnaround
Replace at least 50 percent of school staff
Restart
Convert to a charter school or close and reopen under the management of a charter or education management organization
Closure
Close and enroll its students in higher-achieving schools in the district

4. Study Provides Large-Scale, Rigorous Evidence on Impacts of SIG
Previous impact studies of SIG focused on individual states or cities
Dee (2012); LiCalsi et al. (2015); Gold et al. (2012)
This study used:
Regression discontinuity design (RDD) to examine impact of SIG-funded models on student achievement
460 schools from 50 districts in 21 states
Student-level administrative data from states and districts from school years 2009–2010 through 2012–2013

5. SIG Eligibility Rules Created RDD Opportunities
Treatment schools were in eligibility Tiers I and II, defined by cutoffs on continuous variables
Achievement (bottom 5%)
Graduation rate (60% or lower)
Comparison schools were in Tier III or were ineligible
Main analysis focused on the achievement assignment variable
Fuzzy RDD
85% of schools below cutoff implemented a SIG-funded model
10% of schools above cutoff implemented a SIG-funded model

6. RDD Methods Local linear impact estimation
Estimated impacts within a bandwidth, adjusting for assignment variable using linear functional form
Imbens-Kalyanaraman bandwidth selection method
Estimated separate impacts for each grade
Overall impacts are a sample-size weighted average
Included covariates
Baseline test scores, demographics, state and district indicators
Bootstrapped standard errors
Estimated local average treatment effect

7. Schools implementing a SIG-funded intervention model in 2010–2011
Intervention Schools Were More Disadvantaged and More Likely to Be Urban Than SIG Schools Nationwide
School characteristics (percentages)
Schools implementing a SIG-funded intervention model in 2010–2011
Study sample
Entire U.S.
Students eligible for free or reduced-price lunch
84* 78
Eligible for Title I
94* 89
Located in urban area
88* 58
School intervention model
Transformation
59* 74
Turnaround
28* 20
Restart
10* 4
Closure 3 2
Sources: Common Core of Data, 2009–2010; IES database of SIG grantees; state and district administrative records.
* Significantly different from schools in the United States implementing a SIG-funded intervention model in 2010–2011.

8. Validity of the RDD Was Established
Intervention and comparison schools had similar average baseline characteristics after adjusting for the assignment variable
No statistical or graphical evidence that the achievement assignment variable was manipulated
Baseline characteristics in 2009–2010
Intervention schools
Comparison schools
p-value of difference
Math achievementa
-0.78
-0.72
0.57
Reading achievementa
-0.77
0.55
Percentage of students who are:
Eligible for free or reduced-price lunch 77 80
0.39
English language learners 15 18
0.15
White 16 12
0.16
Sources: State and district administrative records.
a We standardized test scores by subtracting the state-grade-level mean and dividing by the state-grade-level standard deviation. Negative values indicate that the schools in our sample were lower-achieving than the average school in the state.

9. SIG-Funded Models Had No Impacts on Student Achievement Three Years After Grants Were Received
95% confidence interval
Sources: State and district administrative records.
Note: Units for test scores are effect sizes. Units for high school graduation are percentage points/100. For example, an impact of 0.1 indicates an increase of 10 percentage points out of 100 total.
None of the impacts were statistically significant at the 0.05 level.

10. Findings Were Robust to Alternative Analysis Methods
Alternative bandwidths
Include graduation rate assignment variable
Account for student mobility
Students who were slated to attend each school (instead of those who actually attended)
Alternative aggregation weights
Alternative functional form (cubic polynomial)
Exclude covariates

11. Why Did We Find No Impacts of SIG-Funded Models on Student Achievement?
Several potential explanations; we focus on two:
SIG-funded models did not substantially increase the use of SIG-promoted practices
Practices might be poorly implemented or ineffective
No evidence on quality of implementation
Previous literature provides mixed evidence on effectiveness of some practices

12. For More Information Lisa Dragoset IES report:
IES report:
Dragoset, L., Thomas, J., Herrmann, M., Deke, J., James-Burdumy, S., Graczewski, C., Boyle, A., Upton, R., Tanenbaum, C., & Giffin, J. (2017). School Improvement Grants: Implementation and Effectiveness (NCEE ). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

13. References
Dee, T. (2012) School Turnarounds: Evidence from the 2009 Stimulus. Cambridge, MA: National Bureau of Economic Research, Available at
Gold, E., Norton, M.H., Good, D., & Levin, S. (2012). Philadelphia’s Renaissance Schools Initiative: 18 month interim report.” Philadelphia, PA: Research for Action.
LiCalsi, C., Citkowicz, M., Friedman, L.B., & Brown, M. (2015). “Evaluation of Massachusetts Office of District and School Turnaround Assistance to commissioner’s districts and schools: impact of School Redesign Grants.” Washington, DC: American Institutes for Research.

14. Supplemental Slides

15. SIG-Funded Models Had No Impacts on Student Achievement One and Two Years After Grants Were Received
95% confidence interval
Sources: State and district administrative records.
Note: Units for test scores are effect sizes. Units for high school graduation and college enrollment are percentage points/100. For example, an impact of 0.1 indicates an increase of 10 percentage points out of 100 total.
None of the impacts were statistically significant at the 0.05 level.

16. SIG-Funded Models Had No Impact on Math and Reading Achievement for Any Grades
95% confidence interval
Sources: State and district administrative records.
Note: Units are effect sizes.
None of the impacts were statistically significant at the 0.05 level.

17. High school graduation
SIG-Funded Models Had No Impact on Math, Reading, or High School Graduation Within School/Student Subgroups
Subgroup
Impact in 2012–2013 on:
Math scores
Reading scores
High school graduation
Elementary schools
0.11
0.18 NA
Secondary schools
0.10
0.07
-0.05
Title I receiving schools
-0.12
0.03
0.05
Schools in early Race to the Top (RTT) states
-0.31
Schools in later RTT states
0.72
Schools in non-RTT states
0.09
0.15
0.00
English language learner (ELL) students
-0.01
-1.19
0.66
Non-ELL students
0.02
-0.60
0.79
Sources: State and district administrative records.
Note: Units for test scores are effect sizes. Units for high school graduation are percentage points/100. For example, an impact of 0.1 indicates an increase of 10 percentage points out of 100 total. NA indicates cases for which we could not calculate impacts due to insufficient sample sizes. None of the impacts were statistically significant at the 0.05 level.

18. In Elementary Schools, Improvements in Achievement Were Similar for All Models
Math test scores in elementary schools
Sources: State and district administrative records.
Note: Units are normal curve equivalents (a method of standardizing test scores onto a 1–100 scale).
None of the differences between the models were statistically significant at the 0.05 level.

19. In Secondary Schools, Achievement Improved More with Turnaround Model Than with Transformation Model
Math Test Scores in Secondary Schools
Sources: State and district administrative records.
Note: Units are normal curve equivalents (a method of standardizing test scores onto a 1–100 scale).
* Significantly different from transformation model.