1. Paying Teachers to Perform: The Effects of Bonus Pay in Pernambuco, Brazil Barbara Bruns Lead Education Economist, Latin America and The Caribbean Region World Bank Claudio Ferraz Professor of Economics Pontifical Catholic University (PUC) Rio de Janeiro, Brasil Jessica Rodriguez, Tassia Cruz, Vitor Pereira OPCS –PSP GET –PREM Knowledge and Learning Series Washington, DC, May 8 2012

2.  Teacher Bonus Pay (Pay for Performance)  Theory of action  Global evidence  Context for Bonus pay program in Pernambuco  Results to date  Next Steps Overview

3. Teacher Pay for performance: theory of action (1) “Accountability strategy”: strengthens incentives/pressures for teachers to produce key results (growing use in OECD, MICs, 15 programs in Brasil since 2007) Theory of action: Bonus pay linked to student learning results can  Create short term stimulus for teachers to exert more effort and focus on learning results  Create long-term effects on teacher selection (by making profession more attractive to high performers) Logical policy in context of: Consensus about the desired outcome: student learning gains are key education result for growth and inequality reduction Better measures of students learning Difficulty monitoring the desired teacher behavior directly (effort and effectiveness in producing learning) Perception that teacher salaries are a problem Low and undifferentiated pay making it difficult to attract talent into teaching No links between salary and performance (91% of salary variance in PE explained by yrs on job)

4. Teacher Pay for performance: theory of action (2) Design  Most common form is school-based bonus (individual bonus hard to implement) rewarding year on year learning gains  Relatively expensive (10-20% of wage bill) compared to other “accountability reforms” such as information and SBM (but not compared to across-the-board wage increases) Highly controversial  Corruptible: Extensive evidence of gaming and cheating on high stakes tests in US context  Debate on whether teacher effort is the binding constraint? Or teacher capacity?  Little evidence yet on longer-term selection effects

5. Teacher Pay for Performance: Global evidence  Seven rigorous evaluations outside US bonus pay raised student learning outcomes 0.13-0.27 SD (highest for individual bonus in India)  Latest US evidence is different: even significant individual bonuses had no impact on student results (Nashville, NYC)  In several developing country cases bonus did not lower teacher absence (puzzling)  Most consistent “pathway” in JPAL studies (very low income settings) was extra teacher effort out of school

6. Teacher Pay for Performance: Global evidence  Research frontier now is to explain these results:  Context (low/high accountability?) Neal, 2011  Design features?  Controllability (noise in performance measure + locality)  Size of bonus  Coverage/predictability of bonus  Deepen understanding of how incentives change teacher behavior inside the classroom  Positive: stimulate teacher effort (higher attendance, more preparation)  Negative: cheating, teaching to test

7. Teacher Pay for Performance: Global evidence CountryPredicted Strength, Core Design (bonus size, predictability, noise in performance measure) Average Bonus Value (% MW) Observed Effect Size (Max.) India: Andhra Pradesh: test scores(RCT) 4.7 (indiv) 4.3 (group) 36.27.16 Rajasthan: attendance (RCT)4.7 (indiv)30.17 Israel: Individual bonus based on student test results (RD) 4.3 (indiv)300 14% higher pass rates, 10% higher test scores Israel : School bonus based on multiple student performance measures (tests, graduation rates, credits taken) (RD) 2.7 (group)40.13 Kenya: school bonus, test scores (RCT) 443.14 Brazil: Pernambuco school bonus (DD) 3 180.44 Chile: SNED school bonus340.12

8. Teacher Pay for Performance: Global evidence PE

9. Bonus pay program in Pernambuco Brazil

10. Context for Bonus Pay in Pernambuco, Brasil 27 States and 5,000+ municipalities deliver basic education But federal govt. has put in place: funding equalization and high quality national “results measurement system” IDEB Test scores Pass rates Pernambuco: 950 State schools 1 million students Large schools (1,000 students and 30-40 teachers/per school) Many teachers work two jobs (state and/or municipal or private school

11.  Weak salary incentives for teachers (unified salary scale – 90% of salary explained by age, exp, and education  Weak sanctions and supervision (no dismissal for poor performance or high absence)  Limited results focus:  Teachers late or absent  Unused school libraries and computer labs Pernambuco Educational Context

12. In 2007, Pernambuco had some of the worst education outcomes in Brazil In 2008, the state government introduced a performance pay system for teachers: the Bonus de Desempenho Escolar (BDE) Reward for achieving 50% or more of annual school targets, modeled on IDEB

13. Targets sent to School Principals

14. June 2008 Bonus Program announced Aug 2008 Targets sent to schools Nov 2008 SAEPE exam June 2009 Bonus paid Aug 2009 New targets Nov 2009 SAEPE/ Prova Brasil exams May 2010 Bonus Paid August 2010 New targets Nov 2010 2 nd round classroom observations Dec 2010 SAEPE exam April 2011 Bonus paid Program Implementation Nov 2009 1 st round classroom observations

15. Question 1: Did the bonus “work”? Did it improve test scores and pass rates? (more than they would have improved w/o the bonus) Question 2: How did it work?  How did schools respond? What strategies? Change in strategies over time?  Did teacher behavior change in response to incentives?  Did effects vary by student, teacher, and school characteristics?  Were there negative effects? (eg. gaming, cheating) Question 3: How do schools respond to non-linear performance targets and threshold? Did not winning the bonus encourage or discourage effort in subsequent year? Research method: Difference in differences analysis plus classroom observation and directors’ interviews Pernambuco Teacher Bonus: Research questions

16. Pernambuco School Bonus—Program Results

17. State IDEPE Index Results, 9 th grade

18. State IDEPE Index Results, 11 th grade (high school)

19. Strategy: Difference-in-differences  Can answer the question: Did the Pernambuco state schools improve more than they otherwise would have after the introduction of the bonus?  Method: exploit rich national and state-level performance data for the period pre- and post bonus to compare trends in:  Pernambuco state schools with other Northeast states Impact Evaluation: Did the bonus “work”?

20. Pre-trend on test scores– Portuguese 5 th grade

21. Pre-trend on test scores– Math 5 th grade

22. Pre-trend on test scores– Portuguese 9 th grade

23. Pre-trend on test scores– Math 9 th grade

24. Impact Evaluation: Did the bonus “work”?

25. Post-bonus: PE state schools vs. other NE states Effects on test scores– Portuguese 5 th grade

26. Post-bonus: PE state schools vs. other NE states Effects on test scores– Math 5th grade

27. Post-bonus: PE state schools vs. other NE states Effects on test scores– Portuguese 9 th grade

28. Prova Brasil: Pernambuco state vs. Northeast 5 th grade 9 th grade

29. Increase 10% in the pass rate Pass rate: Pernambuco vs. Other Northeast States

30.  Increase in test scores for 9 th grade, both for Portuguese (.36 SD) and Math. (0.41 SD) using the difference-in-differences between Pernambuco and other NE states Small effect on 5th grade math (.12 SD) and none for 5 th grade language minor  Increases in approximately 10 to 12% on pass rates for 6 th to 8 th grades  No effects on pass rates for 1 st to 5 th grades (but very few state schools offer these grades) Overall Bonus Impact

31. Why? Secretariat believes reason is less state school influence on 5 th grade student performance  Fewer lower primary (5 th grade) enrollments in state schools (officially, this level is decentralized to municipalities)  State policies have focused more resources on upper primary and secondary school  State policies such as bonus may have cumulative results (although results presented are short-term) Overall Bonus Impact

32. Evaluation design: Classroom observations in a panel of 1800 classrooms in 300 schools  Can answer questions:  Do teachers behave differently in schools that achieve/don’t achieve the bonus?  Did teacher behavior change over time in Pernambuco state schools?  Are changes in teacher classroom practice correlated with results? Surveys/interviews with school directors, teachers and regional supervisors  Can answer questions:  What strategies do schools use to try to reach targets?  How do these strategies correlate with bonus results? Research Question #2: HOW did the bonus “work”?

33. Bonus Pay and Teacher Classroom practice in Pernambuco

34. Analysis of Panel of 220 Schools in Pernambuco – 2009 and 2010 (after introduction of school bonus program)

35. Slight increase in instructional time and decline in time spent on classroom management but increase in time off task (social interaction w/students) and no change in teacher absence from the class

36. Increase in teacher effectiveness keeping ALL students engaged (share of time entire class engaged with teacher) Small Group: 2 to 5 students Large Group: More than 5 students

37. Research Question 3: How do different targeting mechanisms affect bonus incentives?

38. Pernambuco school bonus: First round targets had sharp discontinuities (8 th grade Portuguese)

39. Pernambuco school bonus: Second round targets (eliminated discontinuities)

40. Pernambuco school bonus: Third round targets (linear)

41. ….but maintained non-linearity in the bonus distribution

42. Losing the bonus by small margin seemed to create incentives for improvement compared to winning the bonus by small margin in year 1 But effects heterogeneous based on school size and specific targets Non-linear bonus assignment, at least close to discontinuity, does not seem to induce a discouragement effect Much scope for further analysis of impact of target-setting mechanisms Preliminary (and incomplete) results

43.  Unlike US results, bonus pay can stimulate improvement in key school results within 1-2 years  Pernambuco results to date – even most conservative estimates – show a significant positive impact in the upper grades Summary: what are we learning from Pernambuco?

44. Summary: What are we learning from Pernambuco? PE

45.  Early evidence that it is possible to observe differences in classroom dynamics between improving and stagnant schools, and system-wide progress over time  (Hopefully) also some insights into design questions in setting up target-based bonus programs  Linear or nonlinear targets?  Linear or nonlinear bonus assignment? (everyone gets it or only those that pass a threshold for improvement – Sao Paulo vs PE)  Reward level of performance, value-added, or both? Should schools that remain at the top be rewarded (Sao Paulo vs PE)?  Should targets be multi-year? Renegotiated annually? Summary: what are we learning from Pernambuco?

46.  What strategies do schools most commonly employ in response to bonus incentives? (2012 qualitative survey and study)  How does school size (free-rider potential) and school “social capital” affect bonus attainment? (2012 analysis of existing survey data)  Does classroom-level improvement continue over time? Does it correlate with bonus? (panel of observations continuing in 2012 and 2013)  Do perverse impacts arise over time (as agents become more familiar with program and potential gaming strategies)? (unique set of “high stakes” and low stakes test data)  How to design effective bonus pay programs? (Comparative study of 3 yr. results in different Brazilian programs – MG, PE, SP, RJ – 2012/13)  Size of bonus? Design of targets? Reward level of performance, value-added, or both? Multi-year targets or renegotiated annually? Summary: next phase of Brazil teacher incentives research

47. Muito Obrigados! Barbara Bruns bbruns@worldbank.org bbruns@worldbank.org Claudio Ferrazcferraz@econ.puc-rio.br Claudio Ferraz cferraz@econ.puc-rio.brcferraz@econ.puc-rio.br