1. Copyright © 2010 Learning Point Associates. All rights reserved. Measuring Principal Performance: How Rigorous Are Publicly Available Assessment Instruments? Matthew Clifford Learning Point Associates matthew.clifford@learningpt.org Presentation for PEAC July 2010

2. www.learningpt.org Learning Point Associates Overview www.learningpt.org/expertise/educatorquality/METworks.php www.learningpt.org

3. What is Your Perspective on Principal Performance Assessment? What instruments are you or others currently using? How systematically are the instruments being administered? How are results being used? How confident are you in the assessment instruments and process?

4. www.learningpt.org What Research Tells Us about Principal Performance Assessment School districts principal performance assessments appear to be inconsistently administered and measured (Goldring et al, 2009; Thomas et al, 2000; Murphy et al, 2007). Principal performance assessments are not always aligned with existing state or national professional standards for practice, and lack justification or documentation of psychometric research (Heck & Marcoulides, 1996).

5. www.learningpt.org Learning Point Associates: Managing Educator Talent www.learningpt.org/expertise/educatorquality/METworks.php Performance Management Coherent Systematic

6. www.learningpt.org Principal Performance Assessment: Recommended Attributes Transparent Trusted Consequential Systematic and consistent Involving multiple measures Validate and reliable (Brown-Sims, 2010)

7. www.learningpt.org Our Study: Question & Method Question: How valid and reliable are publicly available principal performance assessment instruments?

8. www.learningpt.org Our Study: Review Criteria and Results Criteria for inclusion: Claim of use as a principal performance assessment Recent (within past 15 years or so) Publicly available technical information Results of scan: 2000 articles identified by keyword search & snowball sampling Approx. 400 examined 20 articles reviewed on 8 assessment instruments

9. www.learningpt.org Our Study: Instruments Reviewed InstrumentApproachValidityReliability Change Facilitator Style Questionnaire (1988) 77-item survey focusing on 6 domains 360-degree of principal as change facilitator Content: Lit review Construct: Factor analysis Alpha Range:.64 to.95 Diagnostic Assessment of School and Principal Effectiveness (1992) 213-item survey 360-degree Content: Expert review Construct: Factor analysis and inter-item correlation Alpha Range:.8 to.97 Instructional activity questionnaire (1987) 34-item assessment Focus on instructional leadership Content: Lit review Construct: Factor analysis Alpha Range:.7 to.9

10. www.learningpt.org Our Study: Instruments Reviewed InstrumentApproachValidityReliability Instructional activity questionnaire (1987) 34-item assessment Focus on instructional leadership Content: Lit review Construct: Factor analysis Alpha Range:.7 to.9 Leadership Practices Inventory (2002) 30-items Principal and supervisor completes Leadership effectiveness Content: Interviews and surveys of leaders Construct: Factor analysis Concurrent: Other management measures Test/re-test:.79 Performance Review Analysis and Improvement System for Education (1985) 81-item with 9 sub-scales Two-dimensional profile with +/- Content: Lit review with Delphi panel Construct: Unknown Alpha Range:.88 to.98 Test/re-test:.59 to.8

11. www.learningpt.org Our Study: Instruments Reviewed InstrumentApproachValidityReliability Principal Instructional Management Rating Scale (1985) 71-item questionnaire with 11 subscales Content: Lit review and Delphi panel Construct: Correlation within subscale and school document review Alpha:.75 Principal Profile (1986, 1987) Interview-based protocol characterizes principal effectiveness and leadership style Content: Lit review Construct: Factor analysis Inter-rater agreement range from.5 to 1 VAL-ED (2006) 72-items 360-degree, produces profile Content: Lit review Construct: Factor analysis Alpha=.98 for 12 scales on different forms

12. www.learningpt.org Our Study: What did we learn? If there are many principal performance assessment being used in the field, then evidence of their reliability and validity is not publicly available. Of those reviewed, 5 of 8 were developed in tested in the mid-to-late1980s. Survey-based assessment gather self, and others (3), perceptions of principal performance. One used a trained rater. No information on consequential validity, and two examples of concurrent validity.

13. www.learningpt.org 13 Matt Clifford E-Mail: matthew.clifford@learningpt.orgmatthew.clifford@learningpt.org Phone: 630-689-8017 1120 East Diehl Road, Suite 200 Naperville, IL 60563-1486 General Information: 800-356-2735