How Much Value is Added? An Evaluation Plan for the Achievement Challenge Pilot Project

Little Rock Merit Pay Year 1 Evaluation 2 What’s wrong? Teachers affect student performance, however… Policy problem  General & specific teacher shortages  Measuring teacher effectiveness  Providing incentives to teachers Need  System to recruit, retain, and reward high quality individuals in the teaching field

Little Rock Merit Pay Year 1 Evaluation 3 Can current system rectify the problems? Status Quo:  Single Salary System Based on tenure and degree Arguments for single system:  Fair  Simple (critics call it a “breathing bonus”)  Status quo Concerns:  Lacks extrinsic reward for innovation, creativity, hard work  Lacks extrinsic reward for innovation  Does not encourage or reward outcomes  Does not recruit, retain, or reward effective teachers If status quo isn’t working, what alternatives do we have?

Little Rock Merit Pay Year 1 Evaluation 4 Policy Solutions  “Lump Sums” (recruit and retain)  Often in the form of lump increases  Intuitively lacks motivation to work harder  Differential Pay (recruit and retain)  Hard-to-staff schools  Specific subjects  Disadvantaged students  Merit Pay (rewards)  Teacher characteristics  Teacher behavior  Student performance gains Literature: Johnson, 2000; Lazear, 1996; Murnane & Cohen, 1986

Little Rock Merit Pay Year 1 Evaluation 5 Merit Pay Literature Supporters believe performance improves:  Innovation  Work harder  Salary satisfaction Opponents believe performance decreases  Counter-productive competition  Degraded work environment  Focus on high-performing students Evidence: Very few evaluations Policy questions:  Effects of merit pay programs on student performance?  Effects of merit pay programs on teachers?

Little Rock Merit Pay Year 1 Evaluation 6 Possible Policy Implications Possible options:  Improves student achievement, and teachers like program:  Improves student achievement, but teachers dislike program  Does not improve student achievement, but teachers like program;  Does not improve student achievement, and teachers dislike program.

Little Rock Merit Pay Year 1 Evaluation 7 Achievement Challenge Pilot Project (ACPP) Program Goals:  Increase student performance  Reward effective teachers  Make positive influences to school culture  Ultimately, recruits, retains, and rewards effective teachers 5 elementary schools in Little Rock School District Financial rewards based on student performance  payouts computed as NCE gains between fall and spring tests (SAT-9; SAT-10) Meadowcliff payouts per student gain Wakefield payouts based on class average gains

Little Rock Merit Pay Year 1 Evaluation 8 ACPP Addresses Literature Concerns Table 1: Payouts for Wakefield for Employee Type / Position 0-4% Growth 5-9% Growth 10-14% Growth 15%+ Growth Maximum Payout Principal $2,500$5,000$7,500$10,000 Teacher (Grades 4 -5) $50$100$200$400$11,200 Teacher (Grades 1-3) $50$100$200$400$10,000 Teacher (Kindergarten) $50$100$200$400$8,000 Coach $1,250$2,500$3,750$5,000 Specialist; Spec. Ed. $1,000$2,000$3,000$4,000 Music Teacher $1,000$2,000$3,000$4,000 Physical Examiner $500$1,000$1,500$2,000 Aide $250$500$750$1,000 Secretary & Custodian $125$250$375$500

Little Rock Merit Pay Year 1 Evaluation 9 “Observable” School Characteristics School Name Enrollment, % Free/ Reduced Lunch, % Black, Year % Proficient, Math, 2003, Year % Proficient, Literacy, 2003, 2004 Meadowcliff %80% Wakefield %75% Baseline %97% Chicot %75% Franklin %96% Treatment %78% Control %82% Table 2: Demographic Characteristics of Schools in ACPP Evaluation

Little Rock Merit Pay Year 1 Evaluation 10 Research Question 1: Student Effects Question:  What is the impact of the ACPP on the math performance of students? Method:  Student level fixed-effects regression model  Data provided by the Little Rock School District Test scores  Stanford Achievement Test-9 (2003; 2004)  Iowa Test of Basic Skills (2005; 2006)  Reduces “gaming effect” Demographic data  Race, Poverty (FRL), Gender, Age

Little Rock Merit Pay Year 1 Evaluation 11 Methods: Data – Why Math Only? Subject Cohorts, ITBS, 2006 ITBS, 2005 SAT-9, 2004SAT-9, 2003 Reading1VocK: Voc 2Total1: Total 3Total2: Total 4R Comp; Total3: R Comp2: Total 5R Comp; Total4: R Comp 2: Total Math1TotalK: Total 2Total1: Total 3Total2: Total 4Total3: Total 2: Total 5Total4: Total 2: Total Table 3: Summary of Tests by Grade and Year for Fall 2006 Report ITBS 2005, Language subtest not administered to Grade 4 & 5

Little Rock Merit Pay Year 1 Evaluation 12 Methods: Analytic Strategy Regression  Student-level Individual Fixed effects Compares the difference in test scores for treatment students to the difference in test scores for control students This model only applies to 4 th and 5 th grade students because they are the only students who possess  pre-gains ( or to )  post-gains ( to )  Meadowcliff removed – no pre-gain scores

Little Rock Merit Pay Year 1 Evaluation 13 RQ1: Study Sample Characteristics Treatment (n = 132) Comparison (n = 334) Total (n = 466) Schools134 Grade Level, Grade Grade Race % African-American78.8%83.5%83.2% % Caucasian4.5%4.8%4.7% % Hispanic15.2%8.9%10.7% % Other1.5%2.4%2.2% Free and Reduced Lunch Status % Free or Reduced90.8%92.1%91.7% % Full Pay9.2%7.9%8.3%

Little Rock Merit Pay Year 1 Evaluation 14 Teacher Effects What are the attitudes regarding merit pay of ACPP teachers compared to those of teachers in the comparison schools?

Little Rock Merit Pay Year 1 Evaluation 15 Teacher Survey Advantages  Innovation  Work harder  Salary satisfaction Disadvantages  Counter-productive competition  Degraded work environment  Focus on high-performing students Teacher effectiveness

Little Rock Merit Pay Year 1 Evaluation 16 Policy Implications & Conclusions ACPP improves student performance Student performance increased 3.5 NCE points Teachers support the ACPP Significantly more satisfied with ACPP than single salary system Believe the program did not lead to counterproductive competition Believe the school environment is more positive with ACPP Teachers believe ACPP has positive impacts for students Based on student performance increases and teacher opinions, program should be expanded to other elementary schools.

Little Rock Merit Pay Year 1 Evaluation 17 Limitations & Policy Concerns Receptivity is a factor  Teacher support may be vital to program success Limited sample of teachers (58 treatment) Limited sample of students (132 treatment)  All from same school  Only two grades used Funding  $225,000 / school

Little Rock Merit Pay Year 1 Evaluation 18 Survey Practice & Questions