1. Performance Based Incentives for Learning in the Mexican Classroom Brian Fuller, MPA, Foundation Escalera Victor Steenbergen, MPA Candidate, London School of Economics

2. Agenda 1.Star Program in Context 2.Theoretical Overview 3.Evaluation Strategy 4.General Treatment Effects on Test Scores 5.Indirect Treatment Effects (Teachers and Parents)

3. The Star Program Incentives program aimed at marginalized, rural middle school students (ages 12-15) in Chiapas, Mexico. RCT with 147 schools participating (76 treatment, 71 control) In total 7852 students (4011 treatment, 3841 control)

4. Chiapas, Mexico o 37% of workers earn less than monthly minimum wage o Ranks last nationally for computer ownership (7.2%), landlines (12.6%) o 49% rural residents Indigenous Peoples o 39% cannot read or write o 83% work in agriculture o Avg. income is 32% of avg non-indigenous Telesecundaria Middle Schools o Distance learning expands coverage -> many first-time learners o 95% and 89% in lowest two categories in reading and math (21% and 19%, OECD) The Star Program

5. A.Most Improved Student (1 Female, 1 Male) o Increases in general GPA o Targets middle and low-achieving students B. A+ Lottery Winners (1 Female, 1 Male) o Individual subject grades = tickets o Targets high achieving students The Star Program Three times a year, each grade selected:

6. First and second grade students: o material incentives (books, MP3-players, laptops) o increases likelihood students keeping their prices Third grade students: o financial incentives (scholarships up to M$2000). o grants might involve parents in children’s learning. All winners: o Certificates of achievement o Public recognition at ceremony The Star Program Awards

7. 1.Star Program in Context 2.Theoretical Overview 3.Evaluation Strategy 4.General Treatment Effects on Test Scores 5.Indirect Treatment Effects (Teachers and Parents) Agenda

8. Incentive programs work when students lack sufficient motivation to learn (Fryer, 2010) o They do not ‘enjoy’ learning much o Their concerns are short-term driven o They lack knowledge of future benefits to education. Primary impact: financial reward for learning might incentivise student effort to win the prize. Theoretical Overview

9. Student performance requires complementary support from parents and teachers (Fryer, 2010) Secondary impact: incentive program might enhance parent and teacher involvement (Kremer et al, 2004). o Parents: family also benefit from the incentive program (e.g. scholarship) o Teachers: community informally sanctions absent teachers and rewards winners. o Teachers: increased student motivation leads to greater teacher effort. Theoretical Overview

10. Increase student effort to win prize INCENTIVE PROGRAM Student Extrinsic Motivation OVERALL STUDENT ACHIEVEMENT Parental Involvement Teacher Effort Improve teaching and parental support Conceptual Framework

11. 1.Star Program in Context 2.Theoretical Overview 3.Evaluation Strategy 4.General Treatment Effects on Test Scores 5.Indirect Treatment Effects (Teachers and Parents) Agenda

12. Three groups of control variables o Student characteristics o Teacher and director variables o School and regional level characteristics Randomization check using two-tailed t-tests OLS regression with treatment status as dependent variable The trial was well randomized; no significant differences for variables hypothesized most closely associated with student achievement. Evaluation Strategy

13. General treatment effects: Generalized Least Squares regression with random errors clustered at the school level. Effects across student ability: Quantile regression at different points in conditional distribution of test scores, clustered on school level using bootstrapping (Chen, Wei and Parzen, 2003). Indirect treatment effects: Ordered logit regression with clustered standard errors on school level. Evaluation Strategy

14. 1.Star Program in Context 2.Theoretical Overview 3.Evaluation Strategy 4.General Treatment Effects on Test Scores 5.Indirect Treatment Effects (Teachers and Parents) Agenda

15. General impact between 0.217 and 0.259 standard deviations. Impacts are generally significant at the 5% level. Slightly higher impact for males than females. General Treatment Effects

16. Treatment Effects per Grade

17. All impacts are statistically significant at conventional levels. Impact size between 0.200 and 0.346 standard deviations. Larger impact for reading than mathematics. Treatment Effects Grade 3

18. Impact across Student Ability Do incentive awards exclusive benefit previously high-performing students? (Leuven et al, 2010).

19. 1.Star Program in Context 2.Theoretical Overview 3.Evaluation Strategy 4.General Treatment Effects on Test Scores 5.Indirect Treatment Effects (Teachers and Parents) Agenda

20. TreatmentControlImpact Teacher was absent 3 days or less in the last 4 weeks 90.3%75.6%14.68*** TreatmentControlImpact Teacher always or almost always asks students if they understand topic 89.0%78.3%10.65* Teacher attendance Teacher effort Treatment Effects on Teachers

21. TreatmentControlImpact Parents have attended a school meeting this year 93.5%88.0%5.57* Parental Involvement Treatment Effects on Parents

22. 1. Incentive program had a sizeable, significant and robust impact on student performance, raising overall test scores by 0.237 standard deviations. Comparable impact to other education programs. 2. Overall impact for mathematics > reading 3. Overall impact for males > females Conclusion

23. 4. Impact for financial incentives > material incentives Relevance of prize 5. Impact for low, average and high-performing students Successful mix of incentives (‘most improved’ and ‘high grades’) 6. Additional treatment effects for teacher effort and parental support. Conclusion

24. Brian Fuller brian@escalera.org Questions