1. ESSA Implementation: Evidence on Supporting School Leaders
REL West, West CC
Regional Convening
April 16, 2018
Becki Herman
RAND
Teacher Using Interactive Whiteboard During Lesson: Monkey Business Images: Fotolia_ _Subscription_Monthly_M.jpg
Business financial accounting calculate: kitzcorner – Fotolia Fotolia_ _Subscription_Monthly_M.jpg

2. In today’s presentation, we will explore:
ESSA Evidence Review
ESSA funding streams and evidence tiers
Intervention types and effects
Table talk and outbrief
Connecting leadership improvement interventions to outcomes
Why and how to use logic models
Tool and tips
Outcomes and resources
Example

3. School Leadership Interventions under ESSA: Evidence Review

4. School leadership interventions under ESSA: Evidence review
Describes how can ESSA be used to support evidence-based school leadership interventions
Summarizes the evidence base for leadership initiatives based on rigorous empirical studies
Available at

5. ESSA funding streams to support initiatives focused on school leaders
School improvement efforts (7%)
Title I
School leader pipeline improvement activities (up to 3%)
Title II, Part A (Supporting Effective Instruction)
Performance-based incentives
Competitive grants
High need schools
Title II, Part B (National Programs)

6. Tiers of evidence in ESSA
At least one well-designed and well-implemented experimental study (RCT)
Tier I
(Strong)
At least one well-designed and well-implemented quasi-experimental design (QED)
Tier II
(Moderate)
At least one well-designed and well-implemented correlational study controlling for selection bias
Tier III
(Promising)
High-quality research suggesting activity is likely to improve student or other relevant outcomes
Ongoing evaluation
Tier IV

7. Non-regulatory guidance on “evidence-based” (9/16/2016)
Design
Tier
Other Considerations for Tiers I-III
Quality of study
I, II
III
“Well-designed and well-implemented” consistent with WWC standards without/with reservations
“Well-designed and well-implemented” sample or control to account for group differences
Direction, size of impact
I-III
Statistically significant, positive
Body of evidence
“Not be overridden by statistically significant and negative…evidence” in other studies that meet WWC standards with or without reservations
Sample size, composition
Large (350+ people or 50+ groups) and multi-site sample across studies
Overlaps populations and settings for proposed activity site
Rationale IV
“Well-specified logic model that is informed by research or an evaluation”
Evaluation
Produce Tiers I-III evidence
Can be conducted elsewhere

8. Types of principal initiatives
Leadership evaluation systems
Pre-service preparation and internships
Strategic staff management
Professional learning
Working conditions
School reform models with leadership focus

9. Intervention descriptions
Purpose
Intervention type
Intervention target
Intervention components
Duration
Intervention support
Intervention implementers
Intervention developers
Cost
Summary of evidence

10. Findings by intervention type (1)
Type of Activity
Evidence Base
(# of studies)
Findings
Leadership evaluation systems
Tier IV (2)
Promising models with research-based theory of action
Principal preparation programs
Tier II (2)
Tier III (2)
Tier IV (3)
Student-achievement gains, principal instructional actions, improvement in competencies, qualifications of teachers
Models with research-based theory of action, promising correlational evidence of effectiveness
Strategic staff management
Tier III (1)
Negative findings for principal change
Professional learning
Tier I (1)
Positive or no effect on student achievement; reduced staff turnover; promising coaching model with research-based theory of action
Promising models with research and evidence-based theory of action

11. Findings by intervention type (2)
Type of Activity
Evidence Base
(# of studies)
Findings
Working conditions
Tier II (2)
Tier IV (1)
Tier IV (2)
Mixed effects of autonomy on achievement
Incentive/evaluation system correlates with higher student achievement
Models with research-based theory of action, promising correlational evidence of effectiveness for School Administration Manager program; performance-based evaluation, support and compensation
School improvement models
Tier I (1)
Tiers I & II (2)
Tier II (5)
Positive effects on student achievement

12. Level of evidence by intervention (1)
Intervention Type
Intervention Name
Highest Level of Evidence
Leader-evaluation systems
Marzano School Leader Evaluation Model
 Tier IV
Vanderbilt Assessment of Leadership in Education
Principal preparation programs
Exemplary preparation programs
Tier IV
New Leaders Aspiring Principals Program
Tier II
Principal Pipeline Initiative
Principal preparation practices
Tier III
Principal Residency Network
Texas Principal Excellence Program
Strategic staff management
Principal autonomy

13. Level of evidence by intervention (2)
Intervention Type
Intervention Name
Highest Level of Evidence
Professional learning
Arkansas Leadership Academy’s Master Principal Program
Tier IV
Coaching rural leaders
McREL Balanced Leadership Program
Tier I
Metropolitan Independent School District Principal Coaching Initiative
National Institute for School Leadership Executive Development Program
Tier II
Working conditions
Pittsburgh Principal Incentive Program
Principal autonomy
School Administration Manager
School improvement models
Knowledge Is Power Program (KIPP)  
University of Virginia School Turnaround Program

14. Table talk (10 minutes)
Given what you know about the ESSA evidence tiers, how would you describe the evidence base underlying your state or district-developed school leadership program(s)?
How are you reviewing the evidence base for district plans?
How do you examine the impact of your state’s school leadership initiatives?

15. Share out (1-2 minutes per table)
Evidence base for current programs
Strategies for reviewing district plan evidence
Strategies for reviewing impact of state leadership initiatives

16. Connecting Leadership Improvement Interventions to Outcomes

17. Understanding the connections between the interventions and outcomes
Why should I take the time?
Required for Tier IV under ESSA
Maximize impact of investments
How do I use logic models?
To design or refine interventions
To identify best-fit interventions
To evaluate interventions

18. Logic models for selecting, designing, and implementing evidence-based school leadership interventions
Unpacks the relationship between types of intervention and student outcomes
Tool for state and district policymakers
Available at

19. Source programs 6 Source Intervention Type Evidence Review
Other Literature
Expert Nomination
Total
Leader-evaluation systems 2 4 8
Principal preparation programs 9 7 13 29
Strategic staff management 1 3
Professional learning 5 11
Working conditions
School improvement models 6

20. Road map to logic models
Problem Statement
“What issue am I addressing”
Resources
“What do I need”
Activities
“What do I do”
Outputs
“What happens immediately”
Outcomes
“What are my goals”

21. Tips Specify the problem you aim to address
Start from outcomes, and move backwards to resources
Get the big picture: Consider a comprehensive model that includes all of the components
Get the detail: Consider unpacking details for individual components
Adjust the model as needed

22. Outcomes What are desired long–term outcomes?
What short- and medium-term outcomes lead to desired long-term outcomes?

23. Long-term outcomes Increased student success in the following areas:
Student attendance
Student behavior
Student achievement
Graduation
College and career success

24. Medium-term outcomes Improved schools in the following areas:
Instructional quality
School culture/climate/environment
Retention of high-quality staff

25. Short-term outcomes
Improved leadership capacity in the following competencies:
Sets directions, vision, goals
Develops professional learning of staff
Manages instructional program
Manages school environment
Manages time strategically and effectively
Challenges status quo where ineffective
Uses theory, data, and evidence to drive practice
Interacts with external stakeholders
Communicates and connects effectively
Adapts to school needs
Inspires staff and promotes innovation

26. Resources Key questions Types of resources
What is needed to implement the intervention?
Are those resources available?
Types of resources
People
Facilities
Funding
Materials
Data

27. Resources: People Who is receiving the intervention?
Who will provide the intervention?
Who else might affect implementation and effectiveness?
How much time is needed? Is additional staff required?
How will stakeholders communicate and interact?
What are the constraints on human resources?

28. Resources: Facilities
Where will the intervention activities take place?
What are the constraints on facilities?

29. Resources: Funding What are the estimated costs at various stages?
Who will provide direct funding, and how?
Who will provide in-kind donations, and how?
What are the constraints on funding and in-kind donations?

30. Resources: Materials and data
What other materials are needed to support the intervention?
What data are needed to support the intervention?

31. Example: Principal preparation programs

32. Principal preparation programs: Problem statement
Many states and districts struggle with the shortage of new principals who possess competencies needed for effective leadership. While attending training programs, principal candidates often do not receive academic coursework and clinical experiences that adequately prepare them for leadership positions in real life. After graduation, new principals often do not receive sufficient support and mentoring tailored to their individual needs to guide them in the first few years in leadership positions.

33. Resources

34. Activities

35. Activities: Detail (1)

36. Activities: Detail (2)
Recruit and select highly qualified principal candidates
In many university-district partnerships, districts were involved in outreach, recruitment, and selection of highly qualified candidates who possessed leadership competencies
Some programs used a centralized process that utilized selection criteria developed to choose the most highly qualified candidates from the pool of applicants.

37. Outputs

38. Outputs: Detail (1)

39. Output: Detail (2)
Participant candidates gain on-the-job, real-world experience
Length and intensity of residency, apprenticeship, or other clinical practices.
Proportion of principal candidates who report receiving high-quality mentorship during clinical practice.
Extent to which short-term training or coursework is aligned with clinical experiences.

40. Outcomes

41. Table talk
Identify a school leadership challenge you would like to address in your state.
Pick one type of school leadership intervention that could help address that challenge (see pp. 9-10)
What outcomes do you desire? (p. 11)
What changes in schools, teachers, and principals would be needed to achieve these outcomes? (p )
What principal improvement activities are likely to lead to those changes? (p )
What resources are needed to carry out the intervention? (p )

42. Questions?

43. Contact Becki Herman bherman@rand.org 703-413-1100 x5468
Thanks to colleagues Susan Gates, Lindsay Daugherty, Emilio Chavez-Herrerias, Andriy Bega, Aziza Arifkhanova, Catherine Augustine, Eugene Han, Jennifer Tamargo, Mark Harris, and Stephani Wrabel