Evaluation of the Wisconsin Educator Effectiveness System Pilot: Results of the Teacher Practice Rating System Pilot Curtis Jones, UW Milwaukee Steve Kimball, UW Madison Presented at the annual meeting of American Education Finance and Policy 2/28/2015

Evaluation Questions How do educators feel about the Framework for Teaching being used to evaluate teachers? Do educators believe they will have the time and resources necessary to complete teacher practice evaluations? How do educators perceive the inclusion of the Framework for Teaching will impact the quality of Wisconsin teaching? How were teachers rated overall? How were teachers rated on components? What factors were related to teacher ratings?

Pilot Participation 192 principals, and 402 teachers across 195 school districts volunteered and were trained to participate in the pilot. Ultimately, 385 schools across 123 school districts participated, with ratings data only available for 135 of the 402 teachers who originally volunteered to pilot the process. Within these districts, far more (449 evaluators and 2,595 teachers) piloted the teacher practice evaluation component of the EE system than were originally planned. Of the 2,595 teachers involved in the pilot, Announced first observation results were recorded for 2,191 teachers and Unannounced for 1,466, but final ratings were only recorded for 507 teachers across 82 schools and 43 districts. Invitations to complete surveys were e-mailed to 329 administrators (192 principals and 139 coaches) and the 402 teachers that had originally agreed to pilot the system. Of these, 190 (58%) administrators and 171 (44%) teachers responded.

How do educators feel about the Framework for Teaching being used to evaluate teachers?

Do educators believe they will have the time and resources necessary to complete practice evaluations?

How do educators perceive the inclusion of the Framework for Teaching will impact the quality of Wisconsin teaching?

How were teachers rated overall? N Min Max Mean Std. Dev. All recorded ratings Final ratings 497 2.05 3.91 3.13 0.25 1st Announced Observations 2186 1 4 3.03 0.37 1st Unannounced Observations 1460 2.99 0.38   Ratings for teachers with all three ratings Final Ratings 308 3.11 2 3.9 3.04 0.26 1.89 3.07 0.27

How were teacher rated overall?

How were teachers rated on components? – Final Ratings

How were teachers rated on components? - Announced

What factors were related to teacher ratings? – Sample Only schools with at least 5 teacher ratings were included in analyses. The resulting sample includes 34 school districts, 129 schools, 2,173 teachers. The 2,173 teachers represented about half of all the teachers in these schools.

What factors were related to teacher ratings? – Descriptive Statistics Descriptive statistics for schools with at least 5 teachers in the pilot Schools Min Max Mean Std. Deviation Announced observation ratings 129 1.9 3.4 3.00 0.25 Unannounced observation ratings 110 2 3.5 2.95 0.28 Final ratings* 28 2.8 3.14 0.15 Use of EE for high stakes decisions 108 1 0.51 0.50 F/R lunch rate 128 7.8% 100% 42.2 27.5 Teacher to student ratio 126 8.1 41.7 16.5 4.6 Percent of teachers rated 6% 0.49 *Final ratings not modeled due to low n.

What factors were related to teacher ratings? - Correlations Announced ratings Unannounced ratings Final ratings Use of EE for high stakes decisions F/R lunch rate Student to teacher ratio Ratio of teachers rated to teachers Number of teachers rated Announced ratings (n= 129) 1 Unannounced ratings (n = 110) 0.544** Final ratings (n = 28) 0.342 0.347 Use of EE for high stakes decisions (n = 108) 0.227* 0.260* 0.058 F/R lunch rate (n = 128) -0.578** -0.321** -0.07 -0.232* Student to teacher ratio (n = 126) -0.242** -0.220* -0.195 -0.300** 0.261** Percent of teachers rated (n = 126) 0.201* 0.135 0.111 -0.064 -0.256** -0.021 Number of teachers rated (n = 384) 0.13 0.09 0.08 0.013 -0.335** 0.008 0.583** ** Correlation is significant at the 0.01 level (2-tailed). * Correlation is significant at the 0.05 level (2-tailed).

What factors were related to teacher ratings What factors were related to teacher ratings? (Generalized Linear Mixed Models) Level 1: Teacher-level model: Practice ratings ijk = π0jk + eijk Level 2: School-level model: π0jk = β00k + β01k F/R lunch ratejk + β02k Student/teacher ratiojk + β03k Percent of teachers ratedjk + r0jk βpjk = γp0k for p = 1 to 3 Level 3: District-level model: β00k = γ000 + γ001 High stakes use of results + u00 Practice ratings ijk = γ000 + β01k F/R lunch ratejk + β02k Student/teacher ratiojk + β03k Percent of teachers ratedjk + γ001 High-stakes use of results + r0jk + u00 + eijk

What factors were related to teacher ratings?- Model Results 12% of Announced and 8% of Unannounced ratings are attributable to district-level factors, while 16% of Announced and 15% of Unannounced ratings are attributable to school-level factors. School F/R Lunch Rate was the only factor that independently explained Teacher Ratings. School F/R Lunch Rate explained 9% (Announced) and 7% (Unannounced) of the school rating variance and 60% (Announced) and 75% (Unannounced) of the district variance. For Every 10% higher school F/R Lunch Rate, Announced teacher ratings are predicted to go down .048 (0.19 standard deviations) and Unannounced ratings .039 (0.14 standard deviations). Estimate Robust Std. Error df t Sig. School F/R Lunch (Unannounced) -0.00394 0.000905 11.395 -4.36 0.001 School F/R Lunch (Announced) -0.0048 33.897 -5.298 <.0001

Scatter Plot of teacher ratings by School F/R Lunch Rates

F/R Lunch Rates as a predictor of component ratings Modeling the individual components shows that F/R lunch predicts teacher ratings across all of the components. This may suggest that the relationship with F/R lunch may be partly due to teacher selection. Averaged Coefficients Domain 1: Planning and Preparation -0.0037 Domain 2: Classroom Environment -0.0043 Domain 3: Instruction -0.0038 Domain 4: Professional Responsibilities -0.0033

F/R Lunch - Probability of Ratings on 2d (Managing Student Behavior) Compared to Distinguished Rating (Multimodeal Generalized Linear Mixed Model Results)

The Relationship of F/R Lunch and Ratings is Difficult to Interpret In schools and classrooms with higher F/R Lunch rates… lower achieving students and students with behavior problems may make it more difficult for teachers to demonstrate the higher order skills necessary to be rated as Distinguished. students are taught by less experienced teachers. More experienced teachers choose to work in more affluent schools. they are less likely to use ratings for high stakes decisions. Using ratings for high- stakes may put pressure on evaluators to inflate ratings. teachers have more crowded classrooms. teachers may be quicker to burn out, be less motivated, and therefore less effective.

Summary and Conclusions Educators reported understanding both the evaluation process and the Framework for Teaching, that it was fair, that the process would likely empower teachers to better understand their instructional skills, and that it will lead to improved teaching in Wisconsin. The largest concern expressed by educators was that the process is time consuming and that implementing it may leave little time for principals to fulfill all their other duties. The relationship of teacher ratings with factors exogenous to the quality of teaching suggest that, until we understand these relationships more fully, schools should not use ratings to compare teachers.

Limitations and Future Work The results presented here were for a small sample of the state. It is also not known how teachers were selected to participate in the pilot. It is not known if these results will hold up when more evaluators are certified and the analyses are based on final ratings for the whole state rather than single observations for a selection of schools. Lack of available teacher data make many of these findings difficult to interpret. This year, we will be able to use more individual teacher data and perhaps classroom information in our analyses. It is not clear exactly what explains the relationship of F/R lunch with teacher practice ratings. We will gather more qualitative data about how contexts may influence ratings. These case studies can provide narratives to help understand the quantitative findings.

Contact information If you have any questions about this presentation or the evaluation please contact: Curtis Jones jones554@uwm.edu