1. Holli G. Bayonas, Ph.D. & Eric S. Howard, M.A.
The SERVE Center at the University of North Carolina at Greensboro
Propensity Score Matching for the Evaluation of a Federal Teacher Incentive Fund Grant
Program Theory
Background
The theory of change for the TIF-funded program is that financial incentives and professional development will result in a larger pool of highly qualified applicants, higher retention rates of teachers, and better school climate. Long-term outcomes include increased student achievement.
Evaluation Questions:
1. To what extent are the most highly qualified teachers and administrators being recruited and retained at TIF Schools?
2. To what extent were the TIF teachers and administrators trained as proposed?
3. To what extent did the training and incentives impact teacher and student outcomes?
The data used in this evaluation includes climate survey results, teacher retention data, teacher data such as certification and years of experience, and various student data such as graduation rates, discipline data, and achievement data. The propensity score matching was implemented as a way to determine if growth in student achievement was a result of the program or a result of other unexplained factors.
Propensity scores were calculated for schools within North Carolina but outside of the district using the following variables: Enrollment numbers, year teacher turnover rate, % free and reduced lunch, % minority, # of teachers with 0-3 years of experience, Performance Composite, and Performance Composite. Essentially, these were the same variable that were examined by administrators when determining which schools would become the TIF schools. A nearest neighbor approach was used when selecting 3 comparison schools to every 1 treatment school.
Presenting Treatment and Comparison School Results
Background PSM
PSM Advantages: (1) it allows the researcher to identify non-comparable covariates early in the analysis; (2) it is less sensitive to model misspecification; and (3) the research is allowed to generate a non-parsimonious model (Hahs-Vaughn & Onwuegbuzie, 2006).
PSM Disadvantages: (1) the model only considers and controls for observed data; (2) differences between treatment and control groups based on their unobservable characteristics are not controlled for; (3) the need for larger data sets to get optimal performance; and (4) the inclusion of irrelevant covariates will reduce model efficiency (Hahs-Vaughn & Onwuegbuzie, 2006).
Abstract
The SERVE Center at UNCG was awarded a 5-year contract to evaluate 30 schools receiving a federal Teacher Incentive Fund (TIF) grant. Select teachers in hard to fill position are given a recruitment incentive and also a performance incentive when their students make gains in achievement (measured by SAS Value-Added). Using propensity score matching (PSM) and publicly available data, comparison schools were selected across North Carolina. In the Year 2-5 evaluation reports, these comparison schools will be essential to see if trends witnessed in the TIF schools are treatment effects or if these trends are shared by the comparison schools. Presenting variables for the treatment school along with its three comparison schools in chart form facilitates evidence-based decision-making by stakeholders.
Summary
1. Without comparison groups, stakeholders could incorrectly conclude results of a program. While stakeholders are concerned about closing the gap on other schools within the county, they need to be able to accurately attribute effects on students. When scores go up in a treatment school and in some comparison schools, then the program cannot take credit for increased student achievement. When there is no comparison school, and variables of interest move in a positive direction, stakeholders can be tempted to attribute this to the program and not take into account other possible influences.
2. When we aggregate for t-tests and other statistics, we loose valuable individual school data which shows the program working better in some schools than in others. Presenting all of the variables in line charts along with the comparison schools allows a more in-depth look at the school, which is essential for decision-making.
3. Identifying control groups through Propensity Score Matching has methodological limitations but it should be considered as an addition to reporting trend data solely on the treatment schools.
Performance Composite Example: In which of these three schools can we say that the program is contributing positively to increasing overall achievement? Should we be looking at only one variable when determining effectiveness?
Only one variable was statistically significant after the matches; 1-year teacher turnover rate. The remaining variables were considered equal across treatment and control groups. The resulting sample consisted of 28 treatment schools and 70 comparison schools. For the main program report, t-tests were calculated to determine differences between control and treatment. For decision-making in Year 2, individual school reports were created so that stakeholders could see a school’s performance in relation to three schools that did not receive the treatment.