1. Measuring Principals’ Effectiveness:
Results from New Jersey’s Principal Evaluation Study
SREE Spring Conference
March 2, 2017
Christine Ross • Mariesa Herrmann

2. Research partnership with New Jersey Department of Education
NJ developed a principal evaluation system, but little research was available to guide its design
NJ’s principal evaluation system includes measures of professional practice and student achievement growth
Analogous measures as teacher evaluation systems
Problem: little evidence on reliability and validity of these measures
NJDOE requested an assessment of the new principal evaluation system
Pilot year 2012/13
Statewide implementation 2013/14

3. Main questions for the study
Variation in ratings. To what extent did ratings overall and on each of the component measures vary across principals?
Stability of measures. How stable were school median student growth percentiles (SGPs) across years?
Schools with the same principal and those that changed principals
Smaller and larger schools
Correlations with schoolwide student characteristics. What were the correlations between principals’ ratings and the student characteristics of the schools they led?
Correlations among component measures. What were the correlations among component measure ratings?

4. Weights on principal evaluation components in overall ratings, by type of school, 2013-14
Types of schools

5. Percentage of Principals
Nearly all principals received overall ratings of effective or highly effective
Percentage of Principals
Note: 1,656 principals

6. Few principals were rated highly effective on SGPs; Most were rated highly effective on principal goals and teacher SGOs
Note: 1,183 principals

7. School median SGPs were stable across years, even for new principals
Same Principals in and New Principals in
(1,374 principals/schools) (352 principals in 356 schools)
Correlation: Correlation: 0.63

8. School median SGPs were less stable for smaller schools than for larger schools
Change in School Median SGPs 2012/ /14 (Percentile Points)
500
Number of Tested Students in Grades 4–8
Note: 1,267 principals
Findings from 3 years

9. Principals leading schools with larger proportions of economically disadvantaged students tended to receive lower ratings than other principals
Component Measure
Correlation with Schoolwide Percentage of
Economically Disadvantaged Students (Correlation Coefficient)
English Learner Students (Correlation Coefficient)
School Median SGP Rating
–.33*
–.05*
Principal Practice Instrument Rating
–.20*
–.14*
Evaluation Leadership Instrument Rating
–.11*
Principal Goals Rating
–.15*
–.10*
Teachers’ Student Growth Objectives Rating
–.29*
Overall Rating
–.24*
–.12*
* Statistically significant at p < .05, two-tailed test.
Note: 1,450 to 1,781 principals

10. Component measures modestly correlated with each other
Correlation with
School Median SGP Rating
Principal Practice Instrument Rating
Evaluation Leadership Instrument Rating
Principal Goals Rating
Principal Practice Instrument Rating
.16* – a
Evaluation Leadership Instrument Rating
.08*
.61*
Principal Goals Rating
.10*
.32*
Teachers’ Student Growth Objectives Rating
.27*
.23*
.25*
*Statistically significant at p < .05, two-tailed test.
– indicates correlation is for the same measure.
aindicates correlation is shown in a different cell.
Note: 1,183 to 1,752 principals

11. Additional research could further assess measures’ reliability and validity
Topics for further research on principal evaluation:
Implementation quality
Inter-rater reliability of practice instruments, and training needed to attain high rates of inter-rater reliability
Internal consistency of practice instruments
Requires item-level data on principal practice instruments and evaluation leadership instrument
Measures of principals’ contributions to student achievement growth
Requires student achievement data and principal school assignment data over multiple years
Confirming findings with additional year of ratings data

12. For More Information Christine Ross Mariesa Herrmann
Mariesa Herrmann
REL : Herrmann, M., & Ross, C. (2016). Measuring principals’ effectiveness: Results from New Jersey’s first year of statewide principal evaluation
REL : Ross, C., Herrmann, M., & Angus, M. H. (2015). Measuring principals’ effectiveness: Results from New Jersey’s principal evaluation pilot

13. Additional slides

14. References
Branch, G., Hanushek, E., & Rivkin, S. (2012). Estimating the effect of leaders on public sector productivity: The case of school principals. Working paper. Cambridge, MA: National Bureau of Economic Research. Chiang, H., Lipscomb, S., & Gill, B. (2016). Is school value-added indicative of principal quality? Journal of Education Finance and Policy. Coelli, M., & Green, D. (2012). Leadership effects: School principals and student outcomes. Economics of Education Review, 31(1), Dhuey, E., & Smith, J. (2012). How important are school principals in the production of student achievement? Working paper. Toronto, ON: University of Toronto.

15. Most school median SGPs were transformed into a rating of “effective”
Ineffective
Partially effective
Effective
Highly effect- ive
Transforms school median SGP into school median SGP rating
Note: The number of principals with school median SGPs is 1,742.

16. Districts mainly selected commercially-available principal practice instruments
Note: Four other instruments are the New Jersey LoTi Principal Evaluation Instrument, the Rhode Island Model: Building Administrator Evaluation and Support Model, Principal Evaluation and Improvement Instrument, and the Thoughtful Classroom Principal Effectiveness Framework.
Source: New Jersey Department of Education survey of school districts, February 2013 and October 2014

17. Instrument developers provided qualitative information on validity
All seven indicated that the practice instruments are consistent with ISLLC standards for principal leadership
Five indicated that the instruments were developed and informed by research on the relationship between principal practice and school performance or student achievement
None provided information on the statistical relationship between scores on the instrument and student achievement

18. Most instrument developers provided incomplete or no information on reliability
Internal consistency reliability: One developer indicated that analyses of internal consistency reliability confirmed desired constructs
Inter-rater reliability:
One developer provided information on inter-rater reliability standards and training required to meet those standards
Two other developers provide ongoing inter-rater reliability refresher training, but no standards for reliability are indicated

19. Statewide principal SGP ratings were relatively stable across years, among principals in the same school
Ineffective
Partially effective
Effective
Highly effective
Note: The analysis is based on principals who were in the same school in 2012 and 2013
(1,374 principals/schools).

20. School median SGPs were less stable for smaller schools than for larger schools
Change in School Median SGPs 2012/ /14 (Percentile Points)
Increased by More than 5 Percentile Points
500
Decreased by More than 5 Percentile Points
Number of Tested Students in Grades 4–8
Note: 1,267 principals

21. Large changes in school median SGP between years 1 and 2 were less persistent in year 3 for smaller schools than larger ones
Note: 808 principals