Portability of Teacher Effectiveness across School Settings Bill & Melinda Gates Foundation Evaluation of the Intensive Partnership Sites initiative Portability of Teacher Effectiveness across School Settings Zeyu Xu, Umut Ozek, Matthew Corritore November 17, 2018

Motivation › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Redistributing effective teachers at the center of several education policy initiatives Teacher is the most important school input related to student learning The distribution of effective teachers is uneven (recruiting, who moves, and to where) Key assumption: Teachers effectiveness is portable Students face different challenges in learning School culture, environment and working conditions may affect teacher learning, practices, efforts, burnout, etc. Literature Jackson (2010), Jackson & Bruegmann (2009), Goldhaber & Hansen (2010) Sanders, Wright & Langevin (2009) November 17, 2018

Research Questions › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Do teachers retain their effectiveness across schools On average Across schools with similar settings Across schools with different settings (by the direction of the change) Teacher effectiveness measured by Value-added Settings defined by School performance levels School poverty levels Conditional on teachers switching schools November 17, 2018

Preview of Findings › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Among teachers who changed schools, on average their VA was unchanged or slightly improved The same conclusion holds regardless of the similarity/difference between the sending and receiving schools or the direction of the move High-performing teachers’ VA dropped and low-performing teachers’ VA gained in post-move years This pattern is mostly driven by regression to the within-teacher mean and has little to do with school moves Despite this pattern, high VA teachers still performed at a higher level than low VA teachers in post-move years November 17, 2018

Organization Data and samples Methodology Findings › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Data and samples Methodology Findings Summary and discussion November 17, 2018

Data North Carolina 1998-99 through 2008-09 › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion North Carolina 1998-99 through 2008-09 Elementary level (4th and 5th grade math and reading teachers, self- contained classrooms) Secondary level (algebra I and English I teachers, “Algebra I”, “Algebra I-B”, “Integrated Math II”, “English I” classrooms) Florida 2002-03 through 2008-09 Elementary level (4th and 5th grade math and reading teachers, “core courses” in a given subject) Secondary level (9th and 10th grade math and reading teachers, “core courses” in a given subject) November 17, 2018

Sample restrictions Remove charter schools › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Remove charter schools Remove students and teachers who changed schools during a school year (about 2-4% of obs) Remove students with missing values on covariates Keep classrooms with 10~40 students Remove classrooms with >50% special education students November 17, 2018

Number of Unique Teachers in the Analytic Samples Sample sizes › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Number of Unique Teachers in the Analytic Samples North Carolina Florida Elementary Secondary Math 21,119 4,999 29,989 9,101 Reading 3,775 29,354 9,681 November 17, 2018

Two-Stage Analysis Estimate teacher-year value-added › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Estimate teacher-year value-added Difference-in-differences analysis November 17, 2018

Estimate Teacher VA › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Test scores standardized by year, grade and subject (mean=0, sd=1) (X) Covariates include: 1) grade repetition, 2) FRPL, 3) sex, 4) race/ethnicity, 5) gifted, 6) special education, 7) student school mobility and 8) grade level. Bias (no school FE) Noise (EB adjustment) Alternative model specifications (achievement levels model) November 17, 2018

DiD › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Three groups: non-movers, movers to a similar school setting, movers to a different school setting FGLS, se clustered at the teacher level (Y) Year and (T) teacher FEs (X) Teacher experience (0-2, 3-5, 6-12, 13 or more years of exp) (S) School quality (average peer VA) (C) Classroom characteristics (FRL %, mean pretest score, sd of pretest score) (Post) Post-move years indicator (DP, DN) Indicators for school setting differences November 17, 2018

Define School Settings › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion School performance NC: % students performing at or above grade level FL: School performance scores based on both levels and growth Standardized by year and aggregated across all years School poverty % FRPL Aggregated across all years in which a teacher taught in that school Change in school setting measures ∆ = Receiving school – Sending school Similar setting = within half a SD around the mean of the ∆ distribution DP = 1 if ∆ > 0.25 (performance) or ∆ > 0.15 (poverty) DN = 1 if ∆ < -0.25 (performance) or ∆ < -0.15 (poverty) November 17, 2018

Alternative DiD Specs Last pre-move year and first post-move year › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Last pre-move year and first post-move year Between- vs. within-district moves Replace the post-move indicator with individual year dummies (It-1, It-2, It-3…; It+1, It+2, It+3) November 17, 2018

Distribution of Movers › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion By school performance setting change November 17, 2018

Distribution of Movers › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion By school poverty setting change November 17, 2018

Mover Characteristics › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion NC elementary school teachers, by mobility status November 17, 2018

Pre-Post Change in VA (elem) › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion North Carolina Florida Math Reading All 0.004 0.005 -0.001 0.002 By school perf. Higher to lower 0.019 0.011 Similar 0.007 Lower to higher -0.002 0.003 -0.005 By school poverty -0.004 0.000 0.020 0.017 0.009 November 17, 2018

Pre-Post Change in VA (sec) › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion North Carolina Florida Math Reading All 0.056 0.003 0.005 By school perf. Higher to lower 0.067 -0.011 0.013 Similar 0.085 0.014 0.006 0.008 Lower to higher 0.030 0.002 0.000 By school poverty 0.111 -0.006 0.057 0.010 0.004 -0.010 -0.020 -0.003 0.019 November 17, 2018

By Pre-Move VA Actual year of move “Pseudo” move November 17, 2018 › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Actual year of move “Pseudo” move November 17, 2018

By Pre-Move VA Elementary math teachers › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Elementary math teachers Elementary math teachers (pseudo move) November 17, 2018

By Pre-Move VA Elementary reading teachers › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Elementary reading teachers Elementary reading teachers (pseudo move) November 17, 2018

By Pre-Move VA Secondary math teachers › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Secondary math teachers Secondary math teachers (pseudo move) November 17, 2018

By Pre-Move VA Secondary reading teachers › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Secondary reading teachers Secondary reading teachers (pseudo move) November 17, 2018

Adjacent Year Correlations › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Correlation North Carolina Florida Math Reading Yt-2, Yt-1 0.483 0.298 0.380 0.187 (0.426, 0.535) (0.232, 0.362) (0.314, 0.443) (0.111, 0.260) Yt-1, Yt+1 0.341 0.270 0.302 0.138 (0.256, 0.420) (0.182, 0.354) (0.231, 0.369) (0.061, 0.213) Yt-+1 Yt-+2 0.463 0.269 0.427 0.191 (0.381, 0.537) (0.175, 0.358) (0.363, 0.487) (0.115, 0.264) November 17, 2018

Pre-Post Comparisons of VA › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion North Carolina November 17, 2018

Pre-Post Comparisons of VA › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Florida November 17, 2018

Summary › Introduction › Data and Samples › Methodology › Findings › Summary and Discussion Among teachers who changed schools, on average their VA was unchanged or slightly improved The same conclusion holds regardless of the similarity/difference between the sending and receiving schools or the direction of the move High-performing teachers’ VA dropped and low-performing teachers’ VA gained in post-move years This pattern is mostly driven by regression to the within-teacher mean and has little to do with school moves Despite this pattern, high VA teachers still performed at a higher level than low VA teachers in post-move years November 17, 2018