Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets Jishnu Das (World Bank) With: Tahir Andrabi (Pomona College) Asim Ijaz Khwaja (HKS, Harvard)

The Context  Private Schools have expanded dramatically since the 1990s in South Asia PakistanRural India

The Context: Its not what you think!  Although much discussion about madrassas, this is not where the action is!

The Context: The “New” Village Environment  In our sample of 112 villages, there were 812 schools  50% of rural Pakistan’s most populous province—Punjab— live in villages like one of these two A village in Central PunjabA village in North Punjab

The Questions  Can better information about school performance take advantage of the market structure to improve educational outcomes?  What is the equilibrium impact of information on educational markets?  Quality  Price  Quantity  Is there a strong case for the provision of better information?  Address these issues using an experimental design in Punjab province, Pakistan

Precursors  Information can lead to a number of different types of results!  Positive: Information leads to greater accountability/verifiability, competition  (Bjorkman & Svenson (2008) – community-based health reporting/monitoring in Uganda;  Jin & Leslie (2003) – Restaurant Hygiene cards in LA  Rokoff and Turner (2008); Chiang (2008) – public school accountability in US; Hastings (2007)  Nothing: Information may be known, not understood/credible/believed;  Banerjee et al. (2007) - no learning improvements from information dissemination (Indian state)  Negative: Information may lead to greater cream-skimming/sorting (winner takes all)  Education : Chile (Urquiola and Mizala 2007)  Health: Dranove et al. (2003) – hospital outcomes in NY  Direct Manipulation: US – cheating teachers (Jacob & Levitt, 2003)  Gaps  All market-level studies are observational (Dranove and others, Urquiola and Mizala, Jin and Leslie.  Experimental work thus far either in cases where markets are sparse or market reactions not examined  First experimental equilibrium results on impact of information in education

Remainder of talk  A note on private schools  The data  The experiment  The Results  A note on the results

A note on private Schools  All unaided, very sparsely regulated, co-educational, mostly small “mom & pop” operations  Better learning than public schools Learning Probably causal differences

A note on Private Schools (II)  And cheaper, too!

Data (I)  Sample:  112 villages from 3 districts in the Punjab, Pakistan  Randomly selected from list of villages with at least one private school in 2000 (3 rd of villages & 50% of pop); somewhat bigger/richer than average village  Defining Schooling Markets:  Goal – capture parents/children complete choice set & schools’ potential market  Feasible:  92 % of children attend village school (HH census)  Large distance effect – most primary-school children go within 15 minutes  Create 15 minute (30m for RYK) boundaries around village HHs (include some schools right outside village boundary) (Figure 3)

LEAPs Project - :  Baseline HH census (80,000)  Four Rounds ( ):  School (823) Questionnaires:  General School Questionnaire  Class Teacher Questionnaire  Head Teacher Questionnaire  Educational Performance:  Child-Tests (Follow 12,000 plus children over 4 years) in English, Urdu and Mathematics  Household-Level Information  Detailed Household Interviews for randomly selected HHs (1,800)  Short school-based Child questionnaire (randomly select 10 in each school) Attainment FamilySchoolTeacherAdministrationChild Data (II)

Data (III)  Survey Instruments & Timeline:  HH census (80,000 hhs) – 2003  Round 1 (Baseline):  School-Based (Jan-Feb 04): (i) 823 primary schools + class 3 teachers; (ii) 800+ Class 3 teachers; (iii) 6,000 class 3 kids (brief info)  HH-Based (March-Apr 04): Detailed HH surveys (1,800); part matched on class 3 children  Child-Tests (Jan-Feb 04): 12,000+ Class 3 children – Norm- referenced test to maximize variation – Use Item Response Theory to get at underlying child knowledge; we administer (minimize cheating etc.)  Report Card Intervention – Sept/Oct  Round 2 (2005): Report all of Round 1 Surveys/Tests (96% children tracked)

The Experiment  After baseline, villages within each of 3 districts divided with equal probability into treatment and control  Report Card provided to each Class 3 kid parent in school-meeting – explain scores Parent Card 1: Child Info  In all 3 subjects (Maths, Urdu & English):  Child score and quintile  Child’s School score & quintile  Child’s village score and quintile  Quintile described as “needing a lot of work” to “very good”

The Report Card Intervention Parent Card 2: Village Schools Info  For all Primary schools in villages give :  School Name  Tested Children  School scores and quintiles in all 3 subjects  “Bundled-Impact”:  Information (child, schools)  Increase precision, verifiability  Meeting effect?

A note on the information  This is not value-added information  Why?  Feasible intervention  Theoretical considerations (who can back out VA better?)  Empirically doesn’t look too bad  Nevertheless, combination of selection and measurement error may lead to erroneous inference by parents

A further note on the information  Reliability vs. Measurement Error (Kane & Staiger)  Information is fairly reliable  Low measurement error of test  Large variation across schools - see Figure  Selection (into schools)  Value-added estimates?  Selection Not as severe (see learning gaps Figure)  Need to have Information be clear and understandable  Policy feasible/relevant  Households may be better able to back out value-added  village w/ 15 schools; test-score & (2) standard- error bands (computed using IRT)

What should we expect?  3 Broad Classes of models  Symmetric information  Some unobservable components of quality for both schools and households  Asymmetric information: Price signals quality  Asymmetric information: Price does not signal quality  In Model 1 price declines for all schools; depending on structure of demand can get heterogeneous declines by initial quality; quality weakly improves  In Model 2 price declines more for initially higher performing schools; quality weakly improves  In Model 3 price/quality movements are ambiguous

Results: Quality Village- level Average Child- Level Average Child- Level Average (No switcher s) Report Card Villages (0.045) (0.038) (0.038) Good Private School (0.516) Bad Private School (0.019) Governm ent Schools (0.053) Notes Learning:  Similar across subjects; holds 2 yrs after Attrition:  Unlikely concern: no difference in baseline scores for attritors between treatment and control samples Switching/Dropouts:  Results entirely driven by children who stayed in same school:  Few Switch Schools (5%); Gains similar if restrict to non- switching children  For gains to be attributable to switchers, need switchers to have gained 1.7sd, given numbers--- highly unlikely!

Results: Quantity Notes Enrolment  Large increase in RC villages (almost 5 percent)  Entirely from Government schools, entry into Grade i Switching  No evidence of increased churning  But evidence of differential churning School Closures  Significant among initially low performing private schools Enrolme nt Change Probabilit y that child switches Probabilit y that child drops- out School Closure Report Card Villages (13.75)** (0.009) (0.004) -- Good Private School (0.592) (0.516) (0.200) Bad Private School (0.248) (0.019) (0.030) Governm ent Schools (0.006) (0.053) NA

Results: Price (Private Schools Only) Notes Fees  Large Declines—24 percent across the board  Larger in initially higher performing private schools  These are reported by the school; we obtain identical results using reports from households In Rs. ($1=Rs.60 in 2003) In Log fees Report Card Villages (65.090)*** (0.087)*** Good Private School (0.000) (0.001) Bad Private School (0.237) (0.286)

Results: Schools or Households? Notes Household  Little evidence of any big changes (consistent with Das and others 2009)  Children in bad private schools are now playing less  Sleeping more  Spending more time in school Schools  Private schools increase teacher eduation, textbooks  More time on task— fewer breaks

Conclusion  RC increase learning and/or fees drop – equity and efficiency both increase?  Results depend on pre-existing market conditions (parental demand; eductaional production function - school type vs. effort)  Cost of Intervention ~ fee drop  RC exercise cost $1 per child (testing, grading & dissemination)  Cost savings ~ $3/child in private schools (1/3rd of all children enrolled in private schools)  Welfare calculations?  Tricky: Typical Cost-Benefit calculations in LIC ignore welfare costs for providers → learning gains free of cost  BUT: complete welfare analysis has to factor in provider welfare loss – transfer from school to parents & decline in rents – need more structural approach  Policy Questions & Caveats:  Public vs (socially cheaper) Private sector  Intervention simultaneously improve private sector (equity, efficiency) and public sector – State’s role as information provider (rather direct regulator)

Further Notes or how I began to worry that this may actually lead to policy  Theory: Outlined 3 classes of theory; there are others  Question: Why changes in provider behavior, but no increased churning?  Alternative Question: (Hastings): Why switching but no change in provider behavior?  Answer: We don’t know the dynamic equilibrium process in control villages (ratcheting?)  Empirics: Is this a big effect?  CANNOT compare SD increases across tests  Can simulate changes from 0.05sd to 1.3sd by changing the test!  Have to link to some cardinal change (see Heckman)  Trying to calibrate to TIMMS using identical questions  Can answer: how big is this change relative to the world distribution  Longer-term effects (up to 5 years later)  How do we treat utility of providers in welfare computations?  Feasibility  This is proof of concept; mainstreaming is a different issue