Learning Point Associates Measuring Principal Performance: How Rigorous Are Publicly Available Assessment Instruments? Matthew Clifford Learning Point Associates matthew.clifford@learningpt.org Presentation for PEAC July 2010

Learning Point Associates Overview www.learningpt.org www.learningpt.org/expertise/educatorquality/METworks.php

What is Your Perspective on Principal Performance Assessment? What instruments are you or others currently using? How systematically are the instruments being administered? How are results being used? How confident are you in the assessment instruments and process?

What Research Tells Us about Principal Performance Assessment School districts’ principal performance assessments appear to be inconsistently administered and measured (Goldring et al, 2009; Thomas et al, 2000; Murphy et al, 2007). Principal performance assessments are not always aligned with existing state or national professional standards for practice, and lack justification or documentation of psychometric research (Heck & Marcoulides, 1996).

Learning Point Associates: Managing Educator Talent Performance Management Coherent Systematic www.learningpt.org/expertise/educatorquality/METworks.php

Principal Performance Assessment: Recommended Attributes Transparent Trusted Consequential Systematic and consistent Involving multiple measures Validate and reliable (Brown-Sims, 2010)

Our Study: Question & Method Question: How valid and reliable are publicly available principal performance assessment instruments?

Our Study: Review Criteria and Results Criteria for inclusion: Claim of use as a principal performance assessment Recent (within past 15 years or so) Publicly available technical information Results of scan: 2000 articles identified by keyword search & snowball sampling Approx. 400 examined 20 articles reviewed on 8 assessment instruments

Our Study: Instruments Reviewed Approach Validity Reliability Change Facilitator Style Questionnaire (1988) 77-item survey focusing on 6 domains 360-degree of principal as change facilitator Content: Lit review Construct: Factor analysis Alpha Range: .64 to .95 Diagnostic Assessment of School and Principal Effectiveness (1992) 213-item survey 360-degree Content: Expert review Construct: Factor analysis and inter-item correlation .8 to .97 Instructional activity questionnaire (1987) 34-item assessment Focus on instructional leadership Alpha Range: .7 to .9

Our Study: Instruments Reviewed Approach Validity Reliability Instructional activity questionnaire (1987) 34-item assessment Focus on instructional leadership Content: Lit review Construct: Factor analysis Alpha Range: .7 to .9 Leadership Practices Inventory (2002) 30-items Principal and supervisor completes Leadership effectiveness Content: Interviews and surveys of leaders Concurrent: Other management measures Test/re-test: .79 Performance Review Analysis and Improvement System for Education (1985) 81-item with 9 sub-scales Two-dimensional profile with +/- Content: Lit review with Delphi panel Construct: Unknown Alpha Range: .88 to .98 Test/re-test: .59 to .8

Our Study: Instruments Reviewed Approach Validity Reliability Principal Instructional Management Rating Scale (1985) 71-item questionnaire with 11 subscales Content: Lit review and Delphi panel Construct: Correlation within subscale and school document review Alpha: .75 Principal Profile (1986, 1987) Interview-based protocol characterizes principal effectiveness and leadership style Content: Lit review Construct: Factor analysis Inter-rater agreement range from .5 to 1 VAL-ED (2006) 72-items 360-degree, produces profile Alpha= .98 for 12 scales on different forms

Our Study: What did we learn? If there are many principal performance assessment being used in the field, then evidence of their reliability and validity is not publicly available. Of those reviewed, 5 of 8 were developed in tested in the mid-to-late1980s. Survey-based assessment gather self, and others (3), perceptions of principal performance. One used a trained rater. No information on consequential validity, and two examples of concurrent validity.

Matt Clifford E-Mail: matthew.clifford@learningpt.org Phone: 630-689-8017 1120 East Diehl Road, Suite 200 Naperville, IL 60563-1486 General Information: 800-356-2735