1. Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets Jishnu Das (World Bank) With: Tahir Andrabi (Pomona College) Asim Ijaz Khwaja (HKS, Harvard)

2. The Context  Private Schools have expanded dramatically since the 1990s in South Asia PakistanRural India

3. The Context: Its not what you think!  Although much discussion about madrassas, this is not where the action is!

4. The Context: The “New” Village Environment  In our sample of 112 villages, there were 812 schools  50% of rural Pakistan’s most populous province—Punjab— live in villages like one of these two A village in Central PunjabA village in North Punjab

5. The Questions  Can better information about school performance take advantage of the market structure to improve educational outcomes?  What is the equilibrium impact of information on educational markets?  Quality  Price  Quantity  Is there a strong case for the provision of better information?  Address these issues using an experimental design in Punjab province, Pakistan

6. Precursors  Information can lead to a number of different types of results!  Positive: Information leads to greater accountability/verifiability, competition  (Bjorkman & Svenson (2008) – community-based health reporting/monitoring in Uganda;  Jin & Leslie (2003) – Restaurant Hygiene cards in LA  Rokoff and Turner (2008); Chiang (2008) – public school accountability in US; Hastings (2007)  Nothing: Information may be known, not understood/credible/believed;  Banerjee et al. (2007) - no learning improvements from information dissemination (Indian state)  Negative: Information may lead to greater cream-skimming/sorting (winner takes all)  Education : Chile (Urquiola and Mizala 2007)  Health: Dranove et al. (2003) – hospital outcomes in NY  Direct Manipulation: US – cheating teachers (Jacob & Levitt, 2003)  Gaps  All market-level studies are observational (Dranove and others, Urquiola and Mizala, Jin and Leslie.  Experimental work thus far either in cases where markets are sparse or market reactions not examined  First experimental equilibrium results on impact of information in education

7. Remainder of talk  A note on private schools  The data  The experiment  The Results  A note on the results

8. A note on private Schools  All unaided, very sparsely regulated, co-educational, mostly small “mom & pop” operations  Better learning than public schools Learning Probably causal differences

9. A note on Private Schools (II)  And cheaper, too!

10. Data (I)  Sample:  112 villages from 3 districts in the Punjab, Pakistan  Randomly selected from list of villages with at least one private school in 2000 (3 rd of villages & 50% of pop); somewhat bigger/richer than average village  Defining Schooling Markets:  Goal – capture parents/children complete choice set & schools’ potential market  Feasible:  92 % of children attend village school (HH census)  Large distance effect – most primary-school children go within 15 minutes  Create 15 minute (30m for RYK) boundaries around village HHs (include some schools right outside village boundary) (Figure 3)

11. LEAPs Project - www.leapsproject.org :www.leapsproject.org  Baseline HH census (80,000)  Four Rounds (2004-2007):  School (823) Questionnaires:  General School Questionnaire  Class Teacher Questionnaire  Head Teacher Questionnaire  Educational Performance:  Child-Tests (Follow 12,000 plus children over 4 years) in English, Urdu and Mathematics  Household-Level Information  Detailed Household Interviews for randomly selected HHs (1,800)  Short school-based Child questionnaire (randomly select 10 in each school) Attainment FamilySchoolTeacherAdministrationChild Data (II)

12. Data (III)  Survey Instruments & Timeline:  HH census (80,000 hhs) – 2003  Round 1 (Baseline):  School-Based (Jan-Feb 04): (i) 823 primary schools + class 3 teachers; (ii) 800+ Class 3 teachers; (iii) 6,000 class 3 kids (brief info)  HH-Based (March-Apr 04): Detailed HH surveys (1,800); part matched on class 3 children  Child-Tests (Jan-Feb 04): 12,000+ Class 3 children – Norm- referenced test to maximize variation – Use Item Response Theory to get at underlying child knowledge; we administer (minimize cheating etc.)  Report Card Intervention – Sept/Oct  Round 2 (2005): Report all of Round 1 Surveys/Tests (96% children tracked)

13. The Experiment  After baseline, villages within each of 3 districts divided with equal probability into treatment and control  Report Card provided to each Class 3 kid parent in school-meeting – explain scores Parent Card 1: Child Info  In all 3 subjects (Maths, Urdu & English):  Child score and quintile  Child’s School score & quintile  Child’s village score and quintile  Quintile described as “needing a lot of work” to “very good”

14. The Report Card Intervention Parent Card 2: Village Schools Info  For all Primary schools in villages give :  School Name  Tested Children  School scores and quintiles in all 3 subjects  “Bundled-Impact”:  Information (child, schools)  Increase precision, verifiability  Meeting effect?

15. A note on the information  This is not value-added information  Why?  Feasible intervention  Theoretical considerations (who can back out VA better?)  Empirically doesn’t look too bad  Nevertheless, combination of selection and measurement error may lead to erroneous inference by parents

16. A further note on the information  Reliability vs. Measurement Error (Kane & Staiger)  Information is fairly reliable  Low measurement error of test  Large variation across schools - see Figure  Selection (into schools)  Value-added estimates?  Selection Not as severe (see learning gaps Figure)  Need to have Information be clear and understandable  Policy feasible/relevant  Households may be better able to back out value-added  village w/ 15 schools; test-score & (2) standard- error bands (computed using IRT)

17. What should we expect?  3 Broad Classes of models  Symmetric information  Some unobservable components of quality for both schools and households  Asymmetric information: Price signals quality  Asymmetric information: Price does not signal quality  In Model 1 price declines for all schools; depending on structure of demand can get heterogeneous declines by initial quality; quality weakly improves  In Model 2 price declines more for initially higher performing schools; quality weakly improves  In Model 3 price/quality movements are ambiguous

18. Results: Quality Village- level Average Child- Level Average Child- Level Average (No switcher s) Report Card Villages 0.114 (0.045) 0.095 (0.038) 0.102 (0.038) Good Private School 0.039 (0.516) Bad Private School 0.347 (0.019) Governm ent Schools 0.088 (0.053) Notes Learning:  Similar across subjects; holds 2 yrs after Attrition:  Unlikely concern: no difference in baseline scores for attritors between treatment and control samples Switching/Dropouts:  Results entirely driven by children who stayed in same school:  Few Switch Schools (5%); Gains similar if restrict to non- switching children  For gains to be attributable to switchers, need switchers to have gained 1.7sd, given numbers--- highly unlikely!

19. Results: Quantity Notes Enrolment  Large increase in RC villages (almost 5 percent)  Entirely from Government schools, entry into Grade i Switching  No evidence of increased churning  But evidence of differential churning School Closures  Significant among initially low performing private schools Enrolme nt Change Probabilit y that child switches Probabilit y that child drops- out School Closure Report Card Villages 33.78 (13.75)** 0.012 (0.009) 0.006 (0.004) -- Good Private School -2.079 (0.592) 0.039 (0.516) 0.037 (0.200) Bad Private School -4.738 (0.248) 0.347 (0.019) 0.117 (0.030) Governm ent Schools 6.536 (0.006) 0.088 (0.053) NA

20. Results: Price (Private Schools Only) Notes Fees  Large Declines—24 percent across the board  Larger in initially higher performing private schools  These are reported by the school; we obtain identical results using reports from households In Rs. ($1=Rs.60 in 2003) In Log fees Report Card Villages -217.96 (65.090)*** -0.24 (0.087)*** Good Private School -241.841 (0.000) -0.257 (0.001) Bad Private School -139.171 (0.237) -0.126 (0.286)

21. Results: Schools or Households? Notes Household  Little evidence of any big changes (consistent with Das and others 2009)  Children in bad private schools are now playing less  Sleeping more  Spending more time in school Schools  Private schools increase teacher eduation, textbooks  More time on task— fewer breaks

22. Conclusion  RC increase learning and/or fees drop – equity and efficiency both increase?  Results depend on pre-existing market conditions (parental demand; eductaional production function - school type vs. effort)  Cost of Intervention ~ fee drop  RC exercise cost $1 per child (testing, grading & dissemination)  Cost savings ~ $3/child in private schools (1/3rd of all children enrolled in private schools)  Welfare calculations?  Tricky: Typical Cost-Benefit calculations in LIC ignore welfare costs for providers → learning gains free of cost  BUT: complete welfare analysis has to factor in provider welfare loss – transfer from school to parents & decline in rents – need more structural approach  Policy Questions & Caveats:  Public vs (socially cheaper) Private sector  Intervention simultaneously improve private sector (equity, efficiency) and public sector – State’s role as information provider (rather direct regulator)

23. Further Notes or how I began to worry that this may actually lead to policy  Theory: Outlined 3 classes of theory; there are others  Question: Why changes in provider behavior, but no increased churning?  Alternative Question: (Hastings): Why switching but no change in provider behavior?  Answer: We don’t know the dynamic equilibrium process in control villages (ratcheting?)  Empirics: Is this a big effect?  CANNOT compare SD increases across tests  Can simulate changes from 0.05sd to 1.3sd by changing the test!  Have to link to some cardinal change (see Heckman)  Trying to calibrate to TIMMS using identical questions  Can answer: how big is this change relative to the world distribution  Longer-term effects (up to 5 years later)  How do we treat utility of providers in welfare computations?  Feasibility  This is proof of concept; mainstreaming is a different issue