1 A Randomized Evaluation of a School Scholarship Program in Rural India Karthik Muralidharan (with Michael Kremer, Venkatesh Sundararaman) Conference.

Slides:



Advertisements
Similar presentations
AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J.
Advertisements

YOU CANT RECYCLE WASTED TIME Victoria Hinkson. EXPERIMENT #1 :
The World Bank Human Development Network Spanish Impact Evaluation Fund.
Advantages and limitations of non- and quasi-experimental methods Module 2.2.
Girls’ scholarship program.  Often small/no impacts on actual learning in education research ◦ Inputs (textbooks, flipcharts) little impact on learning.
Location of Uganda Proposed Impact Evaluation for Uganda.
With D.D. Karopady M. Sreenivasa Rao The Azim Premji Foundation Karthik Muralidharan Department of Economics, UCSD Abhilash Mudaliar Venkatesh Sundararaman.
M SREENIVASA RAO AZIM PREMJI FOUNDATION 29 April, 2011 Achieving universal quality primary education in India Lessons from the Andhra Pradesh Randomized.
A Guide to Education Research in the Era of NCLB Brian Jacob University of Michigan December 5, 2007.
Baseline for school surveys - Young Lives longitudinal survey of children, households & communities every 3 years since ,000 children Ethiopia,
Benefits and limits of randomization 2.4. Tailoring the evaluation to the question Advantage: answer the specific question well – We design our evaluation.
Social Welfare gains from Community Forests In Orissa, India By, Jon Barnes.
Performance Based Incentives for Learning in the Mexican Classroom Brian Fuller, MPA, Foundation Escalera Victor Steenbergen, MPA Candidate, London School.
Pre-analysis plans Module 8.3. Recap on statistics If we find a result is significant at the 5% level, what does this mean? – there is a 5% or less probability.
Equality of Opportunities from a Fiscal Perspective: Education in Liberia January 26, 2010 Jose Cuesta and Ana Abras PRM PR April 25, 2011.
Pricing the right to education The cost of reaching new targets by 2030 Aaron Benavot Director, EFA Global Monitoring Report Launch Event, Results for.
CYCLING TO SCHOOL: INCREASING SECONDARY SCHOOL ENROLLMENT FOR GIRLS IN INDIA Karthik Muralidharan (UC San Diego, NBER, J-PAL, BREAD) Nishith Prakash (University.
How do Vouchers Work? Evidence from Colombia Eric Bettinger, Case Western U Michael Kremer, Harvard Juan Saavedra, Harvard 7 June 2007 World Bank.
Performance-Based Funding in Higher Education Presentation by Arthur M. Hauptman Financing Reforms for Tertiary Education in the Knowledge Economy Seoul,
Marc Duff, Chief Financial Officer Racine Unified School District Worked in RUSD Finance Department for over 7 years Worked in RUSD Finance Department.
Public Economics Katarzyna Głuch. Definition School voucher (education voucher) is a certificate issued by the government which parents can apply toward.
Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank 1 October 2010.
History of Comparative Education Classification of the field today
Supported by the National Science Foundation under Grant No. DUE Welcome! Washington Science Teachers Association Spring Conference Moses Lake,
Types of Evaluation.
School meals and child outcomes in India Farzana Afridi, Delhi School of Economics IGC-ISI Conference, 20 th – 21 st December, 2010.
WORKING DRAFT Last Modified 3/4/2014 1:23 AM India Standard Time Printed 26/01/ :36 India Standard Time Opportunity for PPP in India McKinsey study.
Comments on Teacher Incentive Papers: Duflo, Dupas and Kremer (Kenya) Muralidharan and Sundararaman (India) Bruns, Ferraz and Rangel (Brazil) By Paul Glewwe.
Teacher-Designed Incentive Pay in Texas A Presentation to the IES Research Conference by Lori L. Taylor.
Pascaline Dupas, UCLA Pupil-Teacher Ratios, Locally-Hired Contract Teachers, and School-Based Management: Evidence from Kenya Making Schools Accountable:
Lessons for Education Policy in Africa Evidence from Randomized Evaluations in developing countries James Habyarimana Georgetown University.
Incentives James Habyarimana Impact Evaluation Initiative World Bank.
Cross-Country Workshop for Impact Evaluations in Agriculture and Community Driven Development Addis Ababa, April 13-16, 2009 AIM-CDD Using Randomized Evaluations.
Quasi-experimental Design CRJS 4466EA. Introduction Quasi-experiment Describes non-randomly assigned participants and controls subject to impact assessment.
THE EFFECT OF INCOME SHOCKS ON CHILD LABOR AND CCTs AS AN INSURANCE MECHANISM FOR SCHOOLING Monica Ospina Universidad EAFIT, Medellin Colombia.
Vouchers in Milwaukee: What Have We Learned From the Nation’s Oldest and Largest Program? Deven Carlson University of Oklahoma.
School-Based Management: Harry Anthony Patrinos
1 Comments on: “New Research on Training, Growing and Evaluating Teachers” 6 th Annual CALDER Conference February 21, 2013.
Evaluating Charter Schools. Harlem Children’s Zone Started by Geoffrey Canada in a 97 block area of Harlem in NYC. Combines high energy “no excuses” type.
Designing an Evaluation of the Effectiveness of NIH’s Extramural Loan Repayment Programs.
Discussion of Private Schooling in India Abhijit V. Banerjee.
Community Participation in Public Schools: Impact of Information Campaigns in three Indian States Priyanka Pandey, Sangeeta Goyal & Venkatesh Sundararaman.
AADAPT Workshop Latin America Brasilia, November 16-20, 2009 Non-Experimental Methods Florence Kondylis.
Lessons for Education in Africa Evidence from Randomized Evaluations in Kenya Esther Duflo J-PAL A B D U L L A T I F J A M E E L P O V E R T Y A C T I.
Quasi Experimental Methods I Nethra Palaniswamy Development Strategy and Governance International Food Policy Research Institute.
Evaluating a test-based subsidy program for low-cost private schools: Regression-discontinuity evidence from Pakistan Felipe Barrera-Osorio Dhushyanth.
Assessing the Distributional Impact of Social Programs The World Bank Public Expenditure Analysis and Manage Core Course Presented by: Dominique van de.
Learning about learning The GEC approach to M&E UKFIET Conference Joseph Holden & Jason Calvert 15 th September 2015 © PEAS.
Private involvement in education: Measuring Impacts Felipe Barrera-Osorio HDN Education Public-Private Partnerships in Education, Washington, DC, March.
Vibha is a 501 (c) (3) organization, registered in the State of New Jersey. Vibha's Tax ID is Sikshana a better foundation.
Second Cross-country Workshop of the Africa Programs for Education and AIDS Impact Evaluation Dakar, December 2008 What do we know about school bursaries?
Research on teacher pay-for-performance Patrick McEwan Wellesley College (Also see Victor Lavy, “Using performance-based pay to improve.
Chapter 10: Arguments for and against Protection.
Evaluation of the DC Opportunity Scholarship Program: Final Report Conducted by Westat, University of Arkansas, Chesapeake Research Associates Presented.
Applying impact evaluation tools A hypothetical fertilizer project.
Measuring Impact 1 Non-experimental methods 2 Experiments
Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative.
Comments on: The Evaluation of an Early Intervention Policy in Poor Schools Germano Mwabu June 9-10, 2008 Quebec City, Canada.
State of the Field: The Need to Understand and Incorporate Variation in Impact in Seeking to Influence Outcomes for Women and Children Kate Schwartz &
1 Teacher Performance Pay: Evidence from the Andhra Pradesh Randomized Evaluation Study (AP RESt) Government of Andhra Pradesh Azim Premji Foundation The.
TRANSLATING RESEARCH INTO ACTION Randomized Evaluation Start-to-finish Abdul Latif Jameel Poverty Action Lab povertyactionlab.org.
Cross-Country Workshop for Impact Evaluations in Agriculture and Community Driven Development Addis Ababa, April 13-16, Causal Inference Nandini.
Department of Economics, University of Stellenbosch
Measuring Results and Impact Evaluation: From Promises into Evidence
NWEA Measures of Academic Progress (MAP)
School Choice: Private Choices in Public Education
Impact Evaluation Methods
Young Lives, University of Oxford
Evaluating Impacts: An Overview of Quantitative Methods
Presentation transcript:

1 A Randomized Evaluation of a School Scholarship Program in Rural India Karthik Muralidharan (with Michael Kremer, Venkatesh Sundararaman) Conference on Public Private Partnerships in Education World Bank, Washington DC 7-8 June, 2007

2  Background  Research Questions & Experimental Design  Questions for Discussion/Feedback Outline

3 Private Schools in Rural India  Private fee-charging schools are widespread in rural India -Especially in areas with poorly performing public schools -28% of villages had private schools in 2003 (probably much higher now) -18% of school enrollment in rural India is in private schools (2006)  Measures of private school performance are superior to public schools – especially with village fixed effects -Teacher attendance, teaching activity, student attendance, test scores  Children attending private schools come from advantaged backgrounds relative to those attending public schools -Difficult to infer causal effect of private schools on performance -Omitted variables and also additional years of schooling -High levels of elite exit from public schooling  Several important questions need to be answered to assess the suitability of a voucher/scholarship/PPP model for improving school education outcomes in India

4 Andhra Pradesh Randomized Evaluation Study  3-way partnership between Government of Andhra Pradesh, Azim Premji Foundation, and the World Bank  5 year MoU with GoAP to systematically study the most promising policy options to improve developing country education via randomized allocation of programs  4 major interventions have been implemented and evaluated so far (Muralidharan & Sundararaman 2006) -Group and Individual-level performance pay for teachers -Use of contract teachers as opposed to civil-service teachers -Cash block grants to schools  Starting a long-term study on the impact of school choice/scholarships/vouchers for disadvantaged children to attend private schools under AP RESt

5 Existing Research on School Vouchers  Several studies consider the impact of randomly offering students a chance to attend private/charter schools -Rouse in Milwaukee -Peterson et. al in New York City, Washington DC, and Dayton, Ohio -Angrist et. al in Colombia -Cullen et. al in Chicago -Hoxby & Rockoff in Chicago  These report varying extent of benefits to voucher recipients -No one finds voucher recipients doing worse off  But more evidence is needed on the overall impact of school vouchers on all the students in a system -Hoxby (2003), Lavy (2006), Hsieh and Urquiola (forthcoming)

6 Decomposing Peer Effects  Randomly allocated voucher programs typically create 4 distinct categories of students -In public school, don’t apply for voucher -Apply for voucher, but don’t win (and typically go to public school, but sometimes go to private school anyway) -Apply for voucher, and win (and likely to go to private school) -In private school, independent of the voucher program  Best studies typically compare the 2 nd and 3 rd groups -ITT and ToT measures  But we don’t know if the first and fourth groups are worse off as a result of peer effects -The public school potentially loses its most motivated students -The private school gets students below its existing average  Ideally, we would want to randomize entire communities into voucher programs and compare the result for all students with the results of all students in similar “control” communities without vouchers

7 Proposed Research Design  An Indian village is pretty close to a “closed economy” in terms of school choice – especially for primary schooling  Identify ~200 villages that all have an existing private school  Two stage randomization -Randomly select half the villages to receive school vouchers -Randomly select children in “voucher villages” to receive them  Track learning outcomes of all children in all schools in both treatment and control villages -Child-level comparison gives the impact on participants -Village-level comparison gives the overall effect of the voucher program  Phase 1 (Pilot) with ~32 villages starting now, with a planned expansion to ~200 villages next year

8 Scholarship Program Design (1 of 2)  The universe of children eligible for the scholarship will be those currently in government schools (grades 1-3) -Provides a sample of disadvantaged children without having to do means testing  The number of scholarships to be offered in a village will be capped at ~30% of the enrollment of all the government schools -Don’t want to empty out the government school -Parents will be required to apply for the scholarship to be eligible (should provide higher first stage but we will find out)  Once a child receives a scholarship, he/she will continue to receive one till the end of primary school subject to meeting attendance requirements and taking the end of year tests -Differential exposure to program across cohorts of recipients

9 Scholarship Program Design (2 of 2)  No topping up – scholarship amount will be set at around the 80th percentile of the private school fee distribution across all villages -Private schools can determine the number of such places at this scholarship rate (but must accept all students who are allocated to these places by a lottery – limits cream skimming)  All expenses for books, uniforms, and school supplies are being covered by the scholarship. -A transport subsidy may be provided in some cases, but may not be required if the choice is being exercised within the village. -The total scholarship spending per child (all inclusive) is expected to be around Rs. 3,200/child per year (USD 80/year). -This is significantly less than the spending per child in the government schooling system  At least Rs. 4,000/year counting only pure variable costs  Over Rs. 5,000/year including various overhead costs

10 Resource Equalization  Even if the “scholarship villages” do better it could be a reflection of additional resources in these villages -Resource equalization is a problem for many other studies as well  Possible solutions: -Make public school lose money for every student who leaves  Not easy since the main expense is very lumpy (teacher salaries)  Also politically much more difficult -Provide matching resources to the public schools in control villages  Doubles “scholarship cost” of program  Operationally more difficult because it becomes another “treatment” -Use estimates from other studies to net out the effect of additional resources in public schools  Most likely course of action – especially since this in Andhra Pradesh  Existing studies in AP will provide estimates of the returns to the 2 main categories of inputs to public schools – additional teachers, and cash grants with school-level flexibility on spending (no ‘infrastructure’ though)

11 Questions We Are Getting At  Do private schools perform better even after accounting for the unobserved variables that might determine private school enrollment? -Answer by comparing scholarship winners to losers in voucher villages -Can think of the question as whether marginal spending on education is best routed via the private sector  What is the aggregate impact of the program (and is any group worse off)? -Answer by comparing voucher villages to non-voucher villages -Unit of analysis is the grade-level average score  What is the functional form of peer effects?  How do parents exercise choice?

12 Questions We Are Not (Yet) Getting At  Effects of competition on productivity of public schools? -Not at this point – negative incentives for public schools are quite difficult practically (lumpy inputs), politically, and potentially ethically -Positive incentives for retention could be considered (your thoughts?)  Benefits of better matching? -Unlikely to be the focus at the primary level  Adequacy and nature of supply response? -Not at this point -Scholarships will only be redeemable at schools existing prior to study  Combining choice with the effects of information to parents on school performance? -Not in Phase 1, but could be an orthogonal/additional treatment

13 Issues for Discussion (1 of 2)  Take up -Aspiration gap, uncertainty of funding/unanticipated expenses -Larger distances to travel -Have required an expression of interest in applying (want a good first stage), and the number appears high (~90%)  Medium of Instruction/SR Adjustments -Most private schools are “English Medium” -Need to allow enough time for SR adjustments to be made  What grades to target? -Had initially intended grades 1-4 (moving to 2-5) -Now only looking at KG, 1, 2 (moving to 1, 2, 3)  Scholarship amount? -Don’t want top ups, 80 th percentile makes it attractive for most schools and covers marginal costs for almost all schools

14 Questions for Discussion (2 of 2)  Transport - to provide or not? -More competition versus greater complexity -Very hard to provide a “standard” option for transport. -Best option may be to provide a subsidy to parents -Piloting both options  Thoughts on ways to provide incentives for the public school to retain children without explicit penalties  What expectations should the public school teachers be operating under?  How much and what kind of information should we provide to parents?  Thoughts on verification/payment/fraud prevention?