1. KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education Conference Christina Clark Tuttle Philip Gleason Brian Gill Ira Nichols-Barrer

2.  Network of 125 charter schools serving over 39,000 disadvantaged students in 20 states and D.C.  Model involves high expectations and more time in school to prepare students for college  Early KIPP model served grades 5-8 (age 10-14) –Represent a majority of schools in operation (n=70) –First elementary and high schools opened in 2004  Mixed effects for charter schools generally  Positive pattern of findings for KIPP specifically Background 2

3.  What are the impacts of KIPP middle schools on student achievement and other student outcomes?  Do KIPP schools engage in selective entry or exit?  Does the performance of KIPP students suggest they are on a path toward college? Research Questions 3

4.  Quasi-experimental analysis of “all” 70 KIPP middle schools –De-identified data from states on student selection, test scores, and attainment  Experimental (or lottery-based) analysis in 13 schools (1,000+ students) –School records –Parent and student surveys –Study-administered test  Validation of observational methods using experimental results Evaluation Design 4

5. Mathematica ® is a registered trademark of Mathematica Policy Research.  22 KIPP middle schools –20 still operating –2 “closed” by KIPP  Opened by SY2005-06 –Allows for more than one cohort to be analyzed across multiple years after KIPP entry  Located in jurisdictions with available data –Three consecutive years of longitudinally-linked student-level data, typically through 2007-08 –For both traditional public and charter schools –Between 3 and 8 cohorts per school Pilot QED Sample Selection 5

6. Mathematica ® is a registered trademark of Mathematica Policy Research.  Treatment group comprised students entering KIPP in 5 th or 6 th grade (n=5,993)  Defined three comparison groups: –District: all students within the district –Feeder: students in ES also attended by KIPP students at baseline (and their MS) –Matched comparison: propensity-score matched comparison group using baseline characteristics  Analyses –Student characteristics –Attrition and replacement –Impacts on achievement Analytic Approach 6

7. Mathematica ® is a registered trademark of Mathematica Policy Research. Demographic Characteristics 7 Difference from KIPP is statistically significant at the 5% level

8. Mathematica ® is a registered trademark of Mathematica Policy Research. Baseline Achievement 8 Difference from KIPP is statistically significant at the 5% level

9. Mathematica ® is a registered trademark of Mathematica Policy Research. Attrition Rates, by Grade 9 Difference from KIPP is statistically significant at the 5% level

10. Mathematica ® is a registered trademark of Mathematica Policy Research. Average Baseline Achievement in Math, Stayers vs. Transfers 10 Difference from stayers is statistically significant at the 1% level  At both KIPP and district schools, early leavers are lower-achieving than students who stay

11. Mathematica ® is a registered trademark of Mathematica Policy Research. Incidence of Late Arrivals 11  KIPP schools replace more students than they lose in grade 6, but fewer in grades 7 and 8  District comparison schools replace more students than they lose in both grades 7 and 8 Replacement Ratio: Ratio of New Arrivals to Prior Attrition Proportion of total enrollment Grade 5-6 Transition Grade 6-7 Transition Grade 7-8 Transition KIPP1.180.780.600.15 FeederNA1.321.350.14

12. Mathematica ® is a registered trademark of Mathematica Policy Research. Average Baseline Achievement in Math, On-Time vs. Late Arrivals 12 Difference from on-time arrivals is statistically significant at the 1% level  At KIPP schools, late arrivals are higher- achieving than on-time arrivals; at district schools, they are lower-achieving

13. Mathematica ® is a registered trademark of Mathematica Policy Research. Baseline reading and math scores, by grade 13  By 8 th grade, KIPP classrooms comprise students higher-achieving at baseline Baseline readingBaseline math KIPPFeederDistrictKIPPFeederDistrict 5 -0.10 -0.09 0.03** -0.10 -0.09 0.04** 6 -0.03 -0.08** 0.05** -0.01 -0.07** 0.05** 7 0.06 -0.07** 0.05 0.07 -0.07** 0.05 8 0.07 -0.07** 0.05 0.13 -0.05** 0.06* All -0.09 0.03** -0.08 -0.09 0.03** *Difference from KIPP is statistically significant at the 0.05 level **Difference from KIPP is statistically significant at the 0.01 level

14.  Model specification:  Retain students who leave KIPP in the treatment group  “Freeze” scores for grade repeaters (more common in KIPP: 11% vs. 2% of 5 th graders) Estimating Impacts 14

15. Year 1Year 2Year 3Year 4 Reading 0.09** (0.011) 0.16** (0.013) 0.24** (0.018) 0.16** (0.027) Math 0.26** (0.011) 0.35** (0.014) 0.42** (0.020) 0.25** (0.027) Number of KIPP schools 22 17 N (math)11,2428,0195,4392,576 N (reading)11,2188,0415,4472,570 Estimated Impact of Potential Exposure to KIPP 15 *Difference is statistically significant at the 0.05 level **Difference is statistically significant at the 0.01 level

16. Percentage of KIPP Schools with Positive and Negative Impacts in Reading, by Years after KIPP Entry 16

17. Percentage of KIPP Schools with Positive and Negative Impacts in Math, by Years after KIPP Entry 17

18. Mathematica ® is a registered trademark of Mathematica Policy Research.  KIPP students are: –More likely to be a racial minority, eligible for FRPL –Less likely to be limited English proficiency or special education –Lower-achieving at baseline than the district overall but equivalent to other students at the same ES  Rates of attrition are similar in KIPP and district schools Conclusions 18

19. Mathematica ® is a registered trademark of Mathematica Policy Research.  Late arrivals present a mixed picture –Proportion of late arrivals relative to enrollment is similar at KIPP and comparison schools –KIPP schools are less likely to replace in later grades –KIPP late arrivals are higher-achieving  Patterns of attrition and late arrivals mean later grades at KIPP comprise higher-performing students, but “peer effects” can explain no more than about a quarter of cumulative impacts  Estimated impacts on reading and math scores are positive, statistically significant, and of substantial magnitude Conclusions 19

20. Mathematica ® is a registered trademark of Mathematica Policy Research.  Please contact: –Christina Clark Tuttle ctuttle@mathematica-mpr.com  View reports online at: –Impacts: http://www.mathematica-mpr.com/publications/PDFs/ education/KIPP_fnlrpt.pdf –Selection and Attrition: http://www.mathematica-mpr.com/publications/PDFs/ education/KIPP_middle_schools_wp.pdf For More Information 20

21. Supplemental Slides 21

22. Mathematica ® is a registered trademark of Mathematica Policy Research. Location of KIPP Schools in Sample 22 KIPP state in study Other KIPP state (as of 2005) Recent KIPP state (as of 2012)

23. Mathematica ® is a registered trademark of Mathematica Policy Research. KIPPFeederDistrict Black0.370.44**0.45** male0.380.420.45** Hispanic0.240.29**0.30** male0.270.30 FRPL0.34 0.37** Overall0.34 0.36** Average Cumulative Attrition by Subgroup 23 *Difference is statistically significant at the 0.05 level **Difference is statistically significant at the 0.01 level

24. Size of Impacts in Reading after Three Years 24 KIPP Schools

25. Size of Impacts in Math after Three Years 25 KIPP Schools