1. Implementing Merit Pay, Overcoming the Obstacles: Lessons from the Field Gary W. Ritter Endowed Chair in Education Policy Brent Riffel Nate Jensen Office for Education Policy University of Arkansas Presented to: Oklahoma State Legislature Oklahoma City, OK August 21, 2007

2. Implementing Merit Pay, Overcoming the Obstacles Slide 2 Presentation Overview 1. Why Modify Salary Structure 2. Policy Alternatives 3. Merit Pay Literature 4. Evaluation of ACPP Merit Pay Program in Little Rock 5. Evaluation of Gilchrist County, Florida Performance Pay Program 6. Designing a Performance Pay Program 7. Ten Pitfalls and How to Avoid Them

3. Implementing Merit Pay, Overcoming the Obstacles Slide 3 Policymakers Striving to Increase Student Performance In an effort to increase student performance, where might policymakers look? The research is clear and consistent in acknowledging the important role of teachers. However, the research is not clear or consistent in identifying strategies for recruiting and retaining effective teachers. Teacher salaries may be an appropriate place to exert policy influence.

4. Implementing Merit Pay, Overcoming the Obstacles Slide 4 Teacher Salaries: What’s the Status Quo? Current Single Salary System Based on tenure and degree Lock-step Arguments for single system Fair Simple Arguments against single system Does not address teacher shortages – either by geographic area or subject area Counter-productive reward structure – good teachers encouraged to: Leave field (better salary) Transfer schools (better environment) Move to Administration (only real promotion)

5. Implementing Merit Pay, Overcoming the Obstacles Slide 5 Policy Strategy? Policymakers may want to adjust the salary system to change the incentive structure. To do that, where might policymakers look for change?

6. Implementing Merit Pay, Overcoming the Obstacles Slide 6 Entry Level Teacher Pay … Competitive New Teacher New Business Graduate

7. Implementing Merit Pay, Overcoming the Obstacles Slide 7 Rewards for Teaching Excellence Decline Over Time Teacher

8. Implementing Merit Pay, Overcoming the Obstacles Slide 8 Rewards for Effective Teachers?

9. Implementing Merit Pay, Overcoming the Obstacles Slide 9 Rewards for Effectiveness?

10. Implementing Merit Pay, Overcoming the Obstacles Slide 10 What Should School Leaders Consider? Teachers affect student performance, however… Status Quo does not necessarily encourage innovation or adopting more effective techniques. Therefore, policymakers need a system to recruit and retain talented teachers and to reward high quality instruction. What alternatives do we have?

11. Implementing Merit Pay, Overcoming the Obstacles Slide 11 Three Policy Alternatives to Recruit, Retain, and Reward Effective Teachers “Lump Sums” Does not change incentive structure Legislature employed this alternative Differential Pay Hard-to-staff schools Specific subjects Legislature employed this alternative Merit Pay Teacher characteristics (e.g. National Board, Prof. Dev.) Teacher behavior Student performance

12. Implementing Merit Pay, Overcoming the Obstacles Slide 12 Merit Pay Debate How might merit pay affect teacher performance? Two types of potential effect s: Composition Motivation Supporters believe performance pay leads to: More innovation Increased work ethic Salary satisfaction Opponents believe performance pay leads to: Counter-productive competition Degraded work environment Decreased focus on high-performing students What does the evidence say?

13. Implementing Merit Pay, Overcoming the Obstacles Slide 13 Merit Pay Background Performance pay is not a novel idea: Such programs were quite common in the early twentieth century (Murnane and Cohen, 1986) Present in some form in about 12% of public school districts in the early 1990’s (Ballou, 2001). Some U.S. studies have found that programs providing bonuses to entire schools, rather than changing the pay of individual teachers, have a positive impact on student test scores (Clotfelter and Ladd, 1996; Ladd, 1999).

14. Implementing Merit Pay, Overcoming the Obstacles Slide 14 Merit Pay: Recent Studies Keys and Dee (2005): Evaluated a performance pay program in Tennessee. Students of teachers participating in the performance pay program made exceptional gains in math and reading. Limitation of the study was that the teachers were allowed to choose if they wanted to participate. Figlio and Kenny (2006): Surveyed schools that participated in the National Educational Longitudinal Survey (NELS). They then supplemented the NELS dataset with information on whether schools compensated teachers for their performance. Found that test scores were higher in schools that individually rewarded teachers for their classroom performance.

15. Implementing Merit Pay, Overcoming the Obstacles Slide 15 Merit Pay Review Podgursky and Springer (2006): Conducted a comprehensive review of merit pay systems. Examined merit pay programs in Denver, Texas, Florida, Minnesota, and the Milken Family Foundation’s Teacher Advancement Program (TAP). Five of the seven existing studies had positive results. In summary, merit pay programs have been shown to be effective ways of improving student test scores.

16. Implementing Merit Pay, Overcoming the Obstacles Slide 16 Merit Pay: What do we know? Very few rigorous evaluations Many programs are short-lived Until recently, data limitations Existing research indicates: Teachers often displeased According to recent lit. review, student performance generally improves or stays the same Let’s look at the evaluations of two programs …

17. Implementing Merit Pay, Overcoming the Obstacles Slide 17 Little Rock School District’s Achievement Challenge Pilot Project (ACPP) Program Goals: Increase student performance Reward effective teachers Improve school culture 5 elementary schools Financial rewards to teachers based on annual gains in student performance

18. Implementing Merit Pay, Overcoming the Obstacles Slide 18 ACPP: Understandable, Non-Competitive for Teachers, Significant $, and Focus on Growth of Students Table 1: Payouts for Wakefield for 2006-07 Employee Type / Position 0-4% Growth 5-9% Growth 10-14% Growth 15%+ Growth Maximum Payout Principal $2,500$5,000$7,500$10,000 Teacher (Grades 4 -5) $50$100$200$400$11,200 Teacher (Grades 1-3) $50$100$200$400$10,000 Teacher (Kindergarten) $50$100$200$400$8,000 Coach $1,250$2,500$3,750$5,000 Specialist; Spec. Ed. $1,000$2,000$3,000$4,000 Music Teacher $1,000$2,000$3,000$4,000 Physical Examiner $500$1,000$1,500$2,000 Aide $250$500$750$1,000 Secretary & Custodian $125$250$375$500

19. Implementing Merit Pay, Overcoming the Obstacles Slide 19 Student Effects Question: What is the impact of the ACPP on the math performance of students? Method: Compare the difference in test scores for ACPP students to the difference in test scores for comparison students Comparison based on math scores of 4 th and 5 th grade students due to data availability Tests used : Stanford Achievement Test-9 Iowa Test of Basic Skills Reduces “gaming effect”

20. Implementing Merit Pay, Overcoming the Obstacles Slide 20 Student Effect Results

21. Implementing Merit Pay, Overcoming the Obstacles Slide 21 Teacher Survey Hypotheses Tested Supporters believe merit pay leads to… 1) More innovation 2) Increased work ethic 3) Salary satisfaction Opponents believe merit pay leads to… 4) Counter-productive competition 5) Degraded work environment 6) Decreased focus on high-performing students 7) If merit pay is beneficial, then student achievement should improve. If merit pay is detrimental, then student performance should decrease. 58 ACPP teachers 74 comparison teachers

22. Implementing Merit Pay, Overcoming the Obstacles Slide 22 Teacher Effect Results ConstructsACPP (% Agree) Comparison (% Agree) Result 1) Innovative88%90%Neutral 2) Work Harder86%99%+ Comp 3) Satisfaction with Compensation 53%35%+ ACPP 4) Collaboration95%99%Neutral 5) Positive School Environment 86%56%+ ACPP 6) Openness to Challenges86%66%+ ACPP 7) Effective Teacher90%78%+ ACPP

23. Implementing Merit Pay, Overcoming the Obstacles Slide 23 Policy Implications & Conclusions ACPP improves student performance Student performance in math increased 3.5 NCE points (roughly 6 to 7 percentile points) Teachers support the ACPP Significantly more satisfied with ACPP than single salary system The program did not lead to counterproductive competition The school environment is more positive with ACPP ACPP has positive impacts for students

24. Implementing Merit Pay, Overcoming the Obstacles Slide 24 Gilchrist County, Florida Performance Pay Plan Enacted in the 1998-1999 school year by the Gilchrist County School Board; revised in 2001-2002. Program Goals: Reward & retain high performing teachers Increase learning in the classroom Teachers given monetary bonuses of 5% of their salary if they receive a performance rating of “Outstanding” Improved student test scores Principal evaluation Limitation: Teachers compete in a “zero-sum” game Only the top 25% of teachers can receive a bonus

25. Implementing Merit Pay, Overcoming the Obstacles Slide 25 Gilchrist County, Florida Performance Pay Plan Question: Does performance pay increase student test scores? Method: Gilchrist County compared to a control group comprised of demographically similar school districts FCAT test scores for both groups were used as the comparison tool

26. Implementing Merit Pay, Overcoming the Obstacles Slide 26 Gilchrist County, Florida Results In 1998, students in Gilchrist County scored an average of 11.3 points lower on standardized tests than the comparison group After 2001, students in Gilchrist County surpassed the control group. Since 2001, Gilchrist County students have increased their average FCAT test score by 22.2 points, compared to the 13.9 point gain made by students in the control group

27. Implementing Merit Pay, Overcoming the Obstacles Slide 27 Gilchrist County, Florida Limitations and Conclusions Limitations of this evaluation: Imperfect control group Gilchrist County has a significantly lower % of minority students than the comparison group. Causality Difficult to attribute increase in test scores directly to the performance pay program. However, the program clearly did not have a negative effect on test scores. The evaluation was much less comprehensive than the Little Rock ACPP. However, our evaluation showed significant gains in student test scores since the implementation of the Performance Pay Program.

28. Implementing Merit Pay, Overcoming the Obstacles Slide 28 Implications for Program Development As policymakers consider ways to recruit, retain, and reward effective public school teachers, they would do well to consider the results of this performance pay plan evaluation. Why Have Programs “Failed” in the Past? Characteristics of Merit Pay Programs with a Chance of Succeeding

29. Implementing Merit Pay, Overcoming the Obstacles Slide 29 Performance Pay: Possible Design Teachers rewarded for: Classroom gains in student performance as measured by state CRT and national NRT (30%) School-wide gains in student performance as measured by state CRT and national NRT(60%) Performance evaluation conducted by principal (10%) Ultimately, Performance Pay should increase student performance by: Rewarding effective teachers Exerting a positive influence on school culture Recruiting, rewarding, and retaining effective teachers

30. Implementing Merit Pay, Overcoming the Obstacles Slide 30 Top Ten Obstacles to Merit Pay Programs 10. Only benefits teachers of top students 9. Based on a secret formula 8. Teaching to the test 7. Extra testing 6. All future raises will be based on test scores 5. What about art, PE, and music teachers? 4. Discourages collegiality 3. Test-taking automatons vs. fully-functional human beings 2. Teachers do not teach for money 1. Rewards aren’t worth extra work

31. Implementing Merit Pay, Overcoming the Obstacles Slide 31 Challenge Number 10 Complaint Only benefits teachers of top students Teachers with the easiest students will have an unfair advantage in a scheme based on test scores. Performance Pay Teacher rewards should be based on test score improvement of all students in a classroom for an entire year. It may be easier for students at the low end to experience improvements! Can produce the right incentives for school leaders to get great teachers to choose to work with needy kids.

32. Implementing Merit Pay, Overcoming the Obstacles Slide 32 Challenge Number 9 Complaint Based on a secret formula It is not at all clear to the teachers how the rewards will be decided and whether they are fair. If we don’t understand what will be rewarded, this cannot possibly work. Performance Pay Teacher rewards should be based on a simple calculation of test score improvement. The gain measure is then straightforward: post-score minus the pre-score.

33. Implementing Merit Pay, Overcoming the Obstacles Slide 33 Challenge Number 8 Complaint Teaching to the test A monetary bonus program will force teachers to teach only items on the test and ignore everything else. Performance Pay The benchmark tests are representative of curricular frameworks developed by teachers and leaders. Teachers will be rewarded by teaching skills that educators have decided are important. Testing is not the only component of Performance Pay

34. Implementing Merit Pay, Overcoming the Obstacles Slide 34 Challenge Number 7 Complaint Extra Testing Students will now have to spend more time testing and even less time in the classroom being instructed by effective teachers. Performance Pay Teacher rewards can be based on student scores from tests that are already administered. No new testing would be required.

35. Implementing Merit Pay, Overcoming the Obstacles Slide 35 Challenge Number 6 Complaint All future raises will be based on test scores District leaders will now shift all new funds for salary increases into the bonus program and eliminate other raises. Performance Pay A successful program might focus only on bonuses (additions to standard salary). Performance Pay would likely have no impact on future teacher salary increases.

36. Implementing Merit Pay, Overcoming the Obstacles Slide 36 Challenge Number 5 Complaint What about art, PE, and music teachers? Bonus programs only reward teachers of core subjects and ignore all other important subjects. Other subjects will then be marginalized in the schools. Performance Pay Teacher rewards based on school wide student growth and principal evaluations. Other subjects contribute to school-wide growth. The job of the principal (whose rating is incorporated) is to ensure that teachers of all subjects be rewarded.

37. Implementing Merit Pay, Overcoming the Obstacles Slide 37 Challenge Number 4 Complaint Discourages collegiality Teachers will no longer want to work together because they will begin to view their colleagues as their competitors in a race for bonus money. Performance Pay Teacher rewards will not be treated as a “zero- sum” game. Teacher rewards are not decreased by rewards given to peers. Rewards are based on student improvement and all teachers may be rewarded. Survey data reveals additional collaboration in schools with such programs.

38. Implementing Merit Pay, Overcoming the Obstacles Slide 38 Challenge Number 3 Complaint Test-taking automatons vs. fully functional human beings Teachers will be rewarded for producing students who can successfully fill in bubble sheets instead of producing “fully functioning” human beings. Performance Pay Teacher rewards based on exams that educators believe are important indicators of readiness for self-sufficiency. The ability to be fully functioning is related to learning key skills that the tests measure.

39. Implementing Merit Pay, Overcoming the Obstacles Slide 39 Challenge Number 2 Complaint Teachers do not teach for money. Monetary rewards are not important to teachers. Teachers do not enter this profession to become rich. Performance Pay Teachers are likely similar to most people: they have many preferences, they prefer more money to less, and they prefer to be rewarded for good work. Teachers, like other workers, do seek salary increases.

40. Implementing Merit Pay, Overcoming the Obstacles Slide 40 Challenge Number 1 Complaint Rewards aren’t worth it. A reward of one or two thousand dollars cannot motivate great changes on a daily basis. Many past similar teacher award programs have been viewed as unsuccessful, partially because of low levels of rewards. Performance Pay Depending on the model, effective teachers can earn a maximum reward of … $10,000

41. Implementing Merit Pay, Overcoming the Obstacles Slide 41 Conclusions Theoretically, there are reasonable arguments for and against performance pay Current system does have poor incentives that could be improved by performance pay There are also characteristics of teaching profession that complicate implementation of teacher incentive pay Research evidence is not robust enough to point in either direction We can conclude that it is worth trying this strategy as a policy alternative and then testing it rigorously

42. Implementing Merit Pay, Overcoming the Obstacles Slide 42 Contact Information: Gary Ritter, Associate Professor Office for Education Policy University of Arkansas http://www.uark.edu/ua/oep Email: oep@uark.edu Phone: (479) 575-3773www.uark.edu/ua/oepoep@uark.edu