1. Martin F. Lueken Anna M. Jacob Jennifer Ash Prepared for the Campbell Collaboration Colloquium Copenhagen 2012 Thursday, May 31 2012 The Effects of Charter Competition on Academic Outcomes: A Review of U.S. Evidence

2.  What is a charter school in the United States?  Considered a public school  Subject to laws that govern public schools  More autonomous than traditional public schools (TPS) – usually not subject to other controls (i.e. collective bargaining agreements)  E.g. can set own academic calendar, less restricted in hiring decisions INTRODUCTION

3.  1991, Minnesota passed first charter law in United States  Political compromise in response to push for education vouchers  Today, 41 states with charter school laws  Charter schools serve over 1.5 million students INTRODUCTION

4. BACKGROUND  Context: charter schools part of school choice movement  Increase school options  Threat to traditional public schools (TPS) to lose students, hence funding  incentive to improve  TPS options:  Improve teaching, how they use resources, etc. (constructive response)  Exert efforts to block reform, barriers to entry (non-constructive response)

5. BACKGROUND

6.  Two effects of charter schools  Direct effect: how well do charter school students achieve relative to TPS students?  Indirect effect: how do other schools behave in face of charter competition? CHARTER EFFECTS

7. What is the effect of charter school competition on student achievement in other traditional public schools? RESEARCH QUESTION

8.  Analytic Challenges  Endogeneity must be addressed in charter school studies (e.g. charter school location not random)  Outcome measures (student level vs. school level)  Variation in charter environments  Charter laws vary significantly by state  Some laws encourage competition, some laws impede competition  Funding levels, caps on # of schools or students, restriction on locations CHALLENGES TO SYSTEMATIC REVIEW

9.  How wide the net?  Definition of charter competition  Include studies with any measure of competition  Grades  Focus on grades K-12  Geographic level  Include studies addressing competition up to state level INCLUSION CRITERIA

10.  Sample period: 2002 and later  Geographic/language: United States/English only  Types of studies:  only quantitative studies that attempt to account for endogeneity problem (e.g. regressions with instrumental variables or fixed effects)  must include statistical control for pre-test  Must include comparison group  Outcomes: student scholastic achievement in math and reading measured by standardized exams

11. Phase 1: Identify Databases Phase 2: Title Review Phase 3: Abstract Review Phase 4: Methods Review Phase 5: Coding Phase 6: Final Inclusion Decision Phase 7: Synthesis SEARCH STRATEGY

12. 1.Searched electronic databases  Google Scholar, PsycINFO, ProQuest, EconLit 2.Searched grey literature 1.NBER working papers, dissertations and theses 3.Hand-searched relevant journals  Journal of School Choice,  Education Next 4.Reviewed introduction and literature reviews of included studies SEARCH STRATEGY

13. Search results Database Titles retrieved Abstracts reviewed Methods reviewed Studies coded Studies kept EconLit366 Google Scholar788 NBER627 ProQuest9403 PsycINFO730 Handsearched74 Total11988

14. Search results Database Titles retrieved Abstracts reviewed Methods reviewed Studies coded Studies kept EconLit36688 Google Scholar78827 NBER62723 ProQuest940362 PsycINFO73061 Handsearched7421 Total11988282

15. Search results Database Titles retrieved Abstracts reviewed Methods reviewed Studies coded Studies kept EconLit3668858 Google Scholar7882724 NBER627236 ProQuest94036235 PsycINFO7306127 Handsearched742118 Total11988282168

16. Search results Database Titles retrieved Abstracts reviewed Methods reviewed Studies coded Studies kept EconLit3668858 Google Scholar7882724 NBER627236 ProQuest94036235 PsycINFO7306127 Handsearched742118 Total119882821682215

17. Search results Database Titles retrieved Abstracts reviewed Methods reviewed Studies coded Studies kept EconLit3668858 Google Scholar7882724 NBER627236 ProQuest94036235 PsycINFO7306127 Handsearched742118 Total119882821682215

18. Table: Locations studied in included articles StatesSchool Districts Arizona (1)ChicagoChula Vista, CA Florida (1)DenverFresno, CA Michigan (3)MilwaukeeLos Angeles, CA North Carolina (2)New York CityNapa Valley, CA Ohio (3)PhiladelphiaSan Diego, CA Texas (4)San DiegoWest Covina, CA "large urban school district in SW" LOCATIONS UNDER STUDY

19.  Number of charter schools within a district or within some specified distance (8)  Enrollment shares of charter schools by district (7)  Distance from TPS to nearest charter school (4)  Student transfer rates from TPS to charter schools (4)  Whether charter school is present in district (2) MEASURES OF CHARTER COMPETITION

20.  Analytic Methods  Fixed effects = 9  Difference-in-differences = 3  Instrumental variables = 3  Level of data  Student = 8  School = 7  Sources  Peer-reviewed = 8  Dissertations = 3  Working papers = 2  Reports = 2 CHARACTERISTICS OF 15 STUDIES

21. SIMPLE VOTE COUNTING Table: Simple vote count of studies included in systematic review MathReadingOverall* Positive656 Mixed / no effect577 Negative212 *overall counts include two studies that used composite measures (positive for Holmes et al., 2003; negative for Kamienski, 2008) -- math and reading effects could not be dissected from these measures

22. ANTICIPATED CHALLENGES Shocks to life-as-usual

23.  Challenges in gathering data from studies  Which estimates to include?  Numerous models and robustness checks run  Some studies (i.e. Zimmer & Buddin, 2009) estimate effects separately for elementary, middle, and high schools; others (i.e. Sass, 2006) produce an aggregate estimate for all grades  Outcome measures?  Most studies use individual student test scores  Some studies (school-level data) use schools’ proficiency rates as outcomes  How to compute effect size? Two separate ones? ANTICIPATED CHALLENGES

24.  Currently planning how to best meta-analyze data  Potential moderator analyses  Effect sizes by states  Effect sizes by district level  Effect sizes by racial background CONCLUSIONS

25. Martin F. Lueken University of Arkansas mlueken@uark.edu Anna M. Jacob University of Arkansas ajacob@uark.edu Jennifer Ash University of Arkansas jash@uark.edu CONTACT