Evaluating a test-based subsidy program for low-cost private schools: Regression-discontinuity evidence from Pakistan Felipe Barrera-Osorio Dhushyanth Raju HDN Learning Week November 10, 2008

Presentation plan Program context and motivation Program design Research questions Regression-discontinuity research design Data and sample Findings Caveats and potential threats to internal validity 2

Foundation-Assisted Schools (FAS) program: Overview Administration: Designed and administered by semi-autonomous organization: the Punjab Education Foundation Objectives: Increase school participation among children from disadvantaged backgrounds as well as their achievement levels. Timeline: FAS program established in 2005 and expanded in phases (4 phases completed; applicant screening for phase 5 underway). Coverage: – Districts: 18 out of the 35 districts in Punjab; 87% of schools in 7 districts; – Schools: 1,082 low-cost private schools (grades 1-10); – Students: 474,000. – Largest PPP program in Pakistan. Program school characteristics: – Mean school size: 351 students. – Majority of schools are middle level (59%), coeducational (83%), registered (87%) and in rural areas (55%). 3

FAS program: Benefits Subsidy: Rs. 300 (US$4.3) per student per month transferred every month for 12 months in the year. Subsidy level set at upper-end of price range for low-cost sector. Use of subsidy largely unfettered. Teacher bonus: Rs. 10,000 (US$143) per teacher for a maximum of 5 teachers in each program school that attains a minimum pass rate of 90% in the Quality Assurance Test (QAT). Offered once a year. Bonus size: 370% of mean maximum monthly teacher salary at baseline. School bonus: Rs. 50,000 (US$714) to the school in each district with the highest pass rate in the QAT. Offered once a year. Bonus size: 76% of mean monthly subsidy payment to schools given mean enrollment size at baseline. 4

FAS program: Continued benefit eligibility conditions Maintain minimum enrollment size of 100 students. Eliminate all tuition and fees for all students. Place a PEF-issued signboard outside school gate which announces tuition- free schooling and provides PEF contact information. Report enrollment figures to PEF every month. Attain a minimum student pass rate of 67% in Quality Assurance Test (QAT) First violation: Penalties (e.g., frozen enrollment). Second violation: Permanent program disqualification. 5

FAS program: Continued benefit eligibility conditions Maximum student-teacher and student-classroom ratios of 35:1. One class per classroom in any period. Registration of school with local authorities within one year of program entry. Maintenance or upgradation of school infrastructure quality per instructions of PEF. Adequate furniture and teaching and learning materials and tools as determined by PEF. Less stringently applied. Schools which violate these conditions given grace period within which to comply (grace period determined on a case-by-case basis). 6

FAS program: Quality Assurance Test (QAT) Used for assessing eligibility for continued benefit receipt. Developed in-house by PEF; administered by contracted organizations. Based on common syllabi and textbooks used in low-cost private schools. Selected subjects: English, Urdu, Mathematics, and Science. Tests almost full range of cognitive learning levels (from knowledge to synthesis). Offered twice a year (in Oct/Nov and Mar/Apr). To date, 5 QATs offered. Test offered to two to three grades (# depends on school level). At least 67% of students in tested grades have to score 40% or higher on the QAT. 7

FAS program: Shortlisting Quality Assurance Test (SLQAT) Used by PEF for assessing initial eligibility for FAS program benefits. Pared-down version of QAT (less questions, less subjects, lower levels of learning only). Developed and administered by PEF. Offered to two to three grades (# depends on school level). At least 67% of students tested have to score 33% or higher on the test. Used as the final screening device for phase-3 and phase-4 applicants. 8

Research questions What is the causal effect of the FAS program on – Number of students – Average student learning – Inputs: teachers, classrooms, blackboards, toilets – Student-input ratios. Student learning data unavailable presently. 9

Identification Entry process: – Schools apply when a call for applications is issued. – Schools with qualifying applications are subject to a physical inspection by PEF. – In phase-3 and phase-4, schools that pass the physical inspection are offered the SLQAT. Assignment to FAS program (treatment) based ultimately on school’s pass rate on SLQAT. If school attains at least 67% (the cutoff), school is eligible; otherwise not. Virtually all schools that attained at least the cutoff, accepted treatment (take-up rate approx. 100%). No treatment dropouts to date. 10

Identification: Phase-4 SLQAT takers Applicant schools with pass rates at or above cutoff, eligible for treatment and took up treatment. Applicant schools with pass rates below cutoff, ineligible and not offered treatment. Data neatly fit a sharp RD design. Probability of treatment is zero below the cutoff, and one above. Treatment parameter: ATT at the cutoff. 11

Identification: Phase-3 SLQAT takers Phase-3 SLQAT failers had the opportunity to reapply and take the phase-4 SLQAT. Phase-3 SLQAT takers can be disaggregated into three subgroups, namely those who: (1) did not enter the program in either phase 3 or 4 (2) entered the program in phase 3 (3) entered the program in phase 4. Given that (3) were failers in the phase-3 SLQAT, these schools can be treated as crossovers. Data fit a fuzzy RD design with one-way noncompliance (crossovers). Probability of treatment is positive but less than one below cutoff, and one above. Treatment parameter: LATE at the cutoff. 12

13

Estimation Both data designs Estimator: ATT and LATE at cutoff estimated nonparametrically using local linear regression (LLR). Kernel (weighting function): Triangular. Bandwidth (window width): Optimal size heuristically-set such that kernel is applied to at least 30 observations on either side of the cutoff. Sensitivity analysis: Fixed kernel; selected increases in bandwidth size. 14

Baseline data Source: Administrative data. Data on school characteristics and non-learning outcomes for all phase-3 and phase-4 applicants obtained from applications maintained electronically by PEF. Data on SLQAT pass rates and average school test performance for phase-3 and phase-4 SLQAT takers obtained from individual student test score data maintained electronically by PEF. Test score and application databases linked together by visually matching on school name and location. 94% and 97% of schools in test database linked to application database in phase 3 and 4, respectively. Full SLQAT-taking samples: Phase 3: 747. Phase 4:

Endline data Source: Phone interviews (schools provided phone numbers in applications). Sample: Phase-3 and phase-4 testtakers with pass rates +/- 15% pts of the cutoff (sample obtained from application-test score linked database). Referred to as neighborhood samples. Original neighborhood sample sizes: Phase 3: 268 (36% of SLQAT taking sample). Phase 4: 319 (38%). Treatment period: Data collected 14 and 10 months after first subsidy payment to phase-3 and phase-4 entrants (partially spans two academic years). Unit nonresponse rate: Phase 3: 28% Phase 4: 26%. Nonreponse bias analysis: No significant correlation with treatment assignment; no strong evidence of correlations with school characteristics measured using baseline data. Neighborhood sample sizes with endline data: Phase 3: 192. Phase 4:

Impact findings Application of partially fuzzy RD design to phase-3 SLQAT neighborhood sample: – No evidence of significant program impacts. – RD LATE estimates sensitive to bandwidth choice. – Empirical standard errors inordinately large, suggesting weak identification. Application of sharp RD design to phase-4 SLQAT neighborhood sample: – Large positive effects on enrollment, teachers, classrooms, and blackboards; within short treatment period of 10 months. – RD ATT estimates somewhat sensitive to bandwidth choice. – Conservative estimates: +85 children (37% relative to baseline mean); +3.4 teachers (37%); +4 classrooms (47%); and +2.8 blackboards (27%). 17

LLR estimates of post-treatment discontinuities 18

Cost-effectiveness findings Annual cost of one additional student induced by the FAS program: – Underlying data: 3,600 rupees per student per year; mean baseline enrollment in phase-4 neighborhood schools: 232; impact of 85 students. – 13,426 rupees (US$189) to induce an additional student per year. Annual cost per student of increasing enrollment by 1%. – Underlying data: 3,600 rupees per student; impact: 37%. – 97 rupees (US$1.4) – Among the lowest cost-effectiveness ratios estimated. 19