ESSA Implementation: Evidence on Supporting School Leaders REL West, West CC Regional Convening April 16, 2018 Becki Herman RAND Teacher Using Interactive Whiteboard During Lesson: Monkey Business Images: Fotolia_52127023_Subscription_Monthly_M.jpg Business financial accounting calculate: kitzcorner – Fotolia Fotolia_79429953_Subscription_Monthly_M.jpg

In today’s presentation, we will explore: ESSA Evidence Review ESSA funding streams and evidence tiers Intervention types and effects Table talk and outbrief Connecting leadership improvement interventions to outcomes Why and how to use logic models Tool and tips Outcomes and resources Example

School Leadership Interventions under ESSA: Evidence Review

School leadership interventions under ESSA: Evidence review Describes how can ESSA be used to support evidence-based school leadership interventions Summarizes the evidence base for leadership initiatives based on rigorous empirical studies Available at https://www.rand.org/pubs/research_reports/RR1550-3.html

ESSA funding streams to support initiatives focused on school leaders School improvement efforts (7%) Title I School leader pipeline improvement activities (up to 3%) Title II, Part A (Supporting Effective Instruction) Performance-based incentives Competitive grants High need schools Title II, Part B (National Programs)

Tiers of evidence in ESSA At least one well-designed and well-implemented experimental study (RCT) Tier I (Strong) At least one well-designed and well-implemented quasi-experimental design (QED) Tier II (Moderate) At least one well-designed and well-implemented correlational study controlling for selection bias Tier III (Promising) High-quality research suggesting activity is likely to improve student or other relevant outcomes Ongoing evaluation Tier IV

Non-regulatory guidance on “evidence-based” (9/16/2016) Design Tier Other Considerations for Tiers I-III Quality of study I, II III “Well-designed and well-implemented” consistent with WWC standards without/with reservations “Well-designed and well-implemented” sample or control to account for group differences Direction, size of impact I-III Statistically significant, positive Body of evidence “Not be overridden by statistically significant and negative…evidence” in other studies that meet WWC standards with or without reservations Sample size, composition Large (350+ people or 50+ groups) and multi-site sample across studies Overlaps populations and settings for proposed activity site Rationale IV “Well-specified logic model that is informed by research or an evaluation” Evaluation Produce Tiers I-III evidence Can be conducted elsewhere

Types of principal initiatives Leadership evaluation systems Pre-service preparation and internships Strategic staff management Professional learning Working conditions School reform models with leadership focus

Intervention descriptions Purpose Intervention type Intervention target Intervention components Duration Intervention support Intervention implementers Intervention developers Cost Summary of evidence

Findings by intervention type (1) Type of Activity Evidence Base (# of studies) Findings Leadership evaluation systems Tier IV (2) Promising models with research-based theory of action Principal preparation programs Tier II (2) Tier III (2) Tier IV (3) Student-achievement gains, principal instructional actions, improvement in competencies, qualifications of teachers Models with research-based theory of action, promising correlational evidence of effectiveness Strategic staff management Tier III (1) Negative findings for principal change Professional learning Tier I (1) Positive or no effect on student achievement; reduced staff turnover; promising coaching model with research-based theory of action Promising models with research and evidence-based theory of action

Findings by intervention type (2) Type of Activity Evidence Base (# of studies) Findings Working conditions Tier II (2) Tier IV (1) Tier IV (2) Mixed effects of autonomy on achievement Incentive/evaluation system correlates with higher student achievement Models with research-based theory of action, promising correlational evidence of effectiveness for School Administration Manager program; performance-based evaluation, support and compensation School improvement models Tier I (1) Tiers I & II (2) Tier II (5) Positive effects on student achievement

Level of evidence by intervention (1) Intervention Type Intervention Name Highest Level of Evidence Leader-evaluation systems Marzano School Leader Evaluation Model  Tier IV   Vanderbilt Assessment of Leadership in Education Principal preparation programs Exemplary preparation programs Tier IV New Leaders Aspiring Principals Program Tier II Principal Pipeline Initiative Principal preparation practices Tier III Principal Residency Network Texas Principal Excellence Program Strategic staff management Principal autonomy

Level of evidence by intervention (2) Intervention Type Intervention Name Highest Level of Evidence Professional learning Arkansas Leadership Academy’s Master Principal Program Tier IV Coaching rural leaders McREL Balanced Leadership Program Tier I   Metropolitan Independent School District Principal Coaching Initiative National Institute for School Leadership Executive Development Program Tier II Working conditions Pittsburgh Principal Incentive Program Principal autonomy School Administration Manager School improvement models Knowledge Is Power Program (KIPP)   University of Virginia School Turnaround Program

Table talk (10 minutes) Given what you know about the ESSA evidence tiers, how would you describe the evidence base underlying your state or district-developed school leadership program(s)? How are you reviewing the evidence base for district plans? How do you examine the impact of your state’s school leadership initiatives?

Share out (1-2 minutes per table) Evidence base for current programs Strategies for reviewing district plan evidence Strategies for reviewing impact of state leadership initiatives

Connecting Leadership Improvement Interventions to Outcomes

Understanding the connections between the interventions and outcomes Why should I take the time? Required for Tier IV under ESSA Maximize impact of investments How do I use logic models? To design or refine interventions To identify best-fit interventions To evaluate interventions

Logic models for selecting, designing, and implementing evidence-based school leadership interventions Unpacks the relationship between types of intervention and student outcomes Tool for state and district policymakers Available at https://www.rand.org/pubs/tools/TL274.html

Source programs 6 Source Intervention Type Evidence Review Other Literature Expert Nomination Total Leader-evaluation systems 2 4 8 Principal preparation programs 9 7 13 29 Strategic staff management 1 3 Professional learning 5 11 Working conditions School improvement models 6

Road map to logic models Problem Statement “What issue am I addressing”   Resources “What do I need”   Activities “What do I do” Outputs “What happens immediately” Outcomes “What are my goals”

Tips Specify the problem you aim to address Start from outcomes, and move backwards to resources Get the big picture: Consider a comprehensive model that includes all of the components Get the detail: Consider unpacking details for individual components Adjust the model as needed

Outcomes What are desired long–term outcomes? What short- and medium-term outcomes lead to desired long-term outcomes?

Long-term outcomes Increased student success in the following areas: Student attendance Student behavior Student achievement Graduation College and career success

Medium-term outcomes Improved schools in the following areas: Instructional quality School culture/climate/environment Retention of high-quality staff

Short-term outcomes Improved leadership capacity in the following competencies: Sets directions, vision, goals Develops professional learning of staff Manages instructional program Manages school environment Manages time strategically and effectively Challenges status quo where ineffective Uses theory, data, and evidence to drive practice Interacts with external stakeholders Communicates and connects effectively Adapts to school needs Inspires staff and promotes innovation

Resources Key questions Types of resources What is needed to implement the intervention? Are those resources available? Types of resources People Facilities Funding Materials Data

Resources: People Who is receiving the intervention? Who will provide the intervention? Who else might affect implementation and effectiveness? How much time is needed? Is additional staff required? How will stakeholders communicate and interact? What are the constraints on human resources?

Resources: Facilities Where will the intervention activities take place? What are the constraints on facilities?

Resources: Funding What are the estimated costs at various stages? Who will provide direct funding, and how? Who will provide in-kind donations, and how? What are the constraints on funding and in-kind donations?

Resources: Materials and data What other materials are needed to support the intervention? What data are needed to support the intervention?

Example: Principal preparation programs

Principal preparation programs: Problem statement Many states and districts struggle with the shortage of new principals who possess competencies needed for effective leadership. While attending training programs, principal candidates often do not receive academic coursework and clinical experiences that adequately prepare them for leadership positions in real life. After graduation, new principals often do not receive sufficient support and mentoring tailored to their individual needs to guide them in the first few years in leadership positions.

Resources

Activities

Activities: Detail (1)

Activities: Detail (2) Recruit and select highly qualified principal candidates In many university-district partnerships, districts were involved in outreach, recruitment, and selection of highly qualified candidates who possessed leadership competencies Some programs used a centralized process that utilized selection criteria developed to choose the most highly qualified candidates from the pool of applicants.

Outputs

Outputs: Detail (1)

Output: Detail (2) Participant candidates gain on-the-job, real-world experience Length and intensity of residency, apprenticeship, or other clinical practices. Proportion of principal candidates who report receiving high-quality mentorship during clinical practice. Extent to which short-term training or coursework is aligned with clinical experiences.

Outcomes

Table talk Identify a school leadership challenge you would like to address in your state. Pick one type of school leadership intervention that could help address that challenge (see pp. 9-10) What outcomes do you desire? (p. 11) What changes in schools, teachers, and principals would be needed to achieve these outcomes? (p. 12-35) What principal improvement activities are likely to lead to those changes? (p. 12-35) What resources are needed to carry out the intervention? (p. 36-37)

Questions?

Contact Becki Herman bherman@rand.org 703-413-1100 x5468 Thanks to colleagues Susan Gates, Lindsay Daugherty, Emilio Chavez-Herrerias, Andriy Bega, Aziza Arifkhanova, Catherine Augustine, Eugene Han, Jennifer Tamargo, Mark Harris, and Stephani Wrabel