2. 2 ©2014 U.S. Education Delivery Institute Session Objectives At the end of this session, you will: ▪ Understand the core components and tools of driving delivery ▪ Have heard of how routines have been useful in for the Oregon School Turnaround and the Hawai’i Department of Education Routines should be considered at the start of an engagement. They are the main tool to drive progress once planning is complete.

3. 3 ©2014 U.S. Education Delivery Institute We will focus on three essential elements that make up how a system drives delivery Develop a foundation for delivery Understand the delivery challenge Plan for delivery A.Evaluate past and present performance B.Understand drivers of performance and relevant activities A.Determine your reform strategy B.Set targets and establish trajectories C.Produce delivery plans A.Establish routines to drive and monitor performance B.Solve problems early and rigorously C.Sustain and continually build momentum Drive delivery A.Define your aspiration B.Review the current state of delivery C.Build the delivery unit D.Establish a “guiding coalition” 2341 Create an irreversible delivery culture 5 A. Build system capacity all the time B. Communicate the delivery message C. Unleash the “alchemy of relationships”

4. 4 ©2014 U.S. Education Delivery Institute In fact, routines are so critical to the success of a delivery effort that we like to call them the “engine” of delivery ▪ Regularly scheduled checkpoints to assess if delivery is on track ▪ Engine that drives delivery forward: Without routines, delivery will stall or eventually fall off the agenda ▪ A source of structure and discipline to create order in complex public sector systems What are routines? ▪ Monitor performance: Understand if system is on track to deliver aspirations, using predetermined assessment frameworks ▪ Diagnose problems: Surface issues that are inhibiting progress and analyze data to pinpoint causes ▪ Address problems: Provide a venue to discuss and decide how to overcome challenges What purpose do routines serve?

5. 5 ©2014 U.S. Education Delivery Institute Effective routines share similar characteristics Agree on a common purpose Arrive at a shared view of performance and progress Identify and solve problems Encourage learning and collaboration Identify and commit to clear next steps

6. 6 ©2014 U.S. Education Delivery Institute This self-assessment rubric can be used to assess and improve existing routines CategoryWhat weak performance (1) looks likeWhat strong performance (4) looks like Agreeing on a common purpose: Do all people participating in the routine clearly understand and agree on its purpose – to discuss performance against priorities, to identify and solve problems, and to identify and commit to clear next steps? ■ Routines are check-ins during which participants give updates on their work, with little or no connection to priorities ■ Preparation for routines is minimal; mostly conducted on-the-fly as a way of touching base ■ Priorities are well understood and agreed on by all participants as the anchor for discussion ■ Agenda, key messages and discussion questions, and relevant data are clear, concise, and well- prepared in advance of the meeting Arriving at a shared view of performance and progress: ■ Is the discussion structured to help participants discuss and agree on current progress against priority goals? ■ What range of evidence is used to support this discussion? ■ Quantitative evidence is sporadic and inconsistent ■ Data are presented in raw format with little or no attempt to discern patterns or implications ■ Discussion is grounded more in opinion and anecdote than it is in evidence ■ A wide range of evidence is presented in a way that is clear, sharp, and consistent, including outcome data, leading indicator data, and evidence on quality of implementation ■ Debate is vigorous but usually results in consensus about overall performance/progress, areas of strength, and areas of weakness Identifying and solving problems: ■ Does the routine help participants to identify and agree on the most critical barriers to achieving priority goals? ■ Does the routine create a space for creative problem-solving that empowers participants to address these challenges? ■ Problems may be identified but are too vague to be actionable; their root causes are poorly understood, if at all ■ Discussion tends to dwell on the problems, with little or no attempt made to actively seek creative solutions ■ Discussion allows participants to identify specific barriers to success, with a focus on root causes that are actionable ■ Discussion encourages all participants to reflect on what it will take to overcome these barriers and to identify realistic solutions Encouraging learning and collaboration: Does the routine encourage participants to identify challenges that are common amongst their peers and to learn from each other’s best and most promising practices? ■ When routines bring peers together, they are reluctant to engage in open dialogue about their own performance and how it compares to others ■ Reluctance may make it difficult to even present data in a way that compares performance projects ■ When routines bring peers together, part of the discussion is structured to encourage learning and sharing across peers, including common challenge and best/promising practices ■ Cross-project comparisons create a spirit of friendly competition and professional learning about how to improve performance across all projects Identifying and committing to clear next steps: Does every routine produce a clear and actionable list of next steps for all relevant participants that can be tracked at future routines? ■ Next steps are not identified in the routine, or they are listed with little refinement to ensure that they can be carried out ■ Next steps may be too vague to be actionable, or they may be one-sided lists of requests ■ There is little or no tracking of next steps between routines ■ Routine ends with the identification of clear next steps for all participants – both junior and senior – that are specific, actionable, and time-bound ■ Follow-through on next steps is tracked and becomes evidence for reaching consensus on progress and performance in future routines, so that next steps tend to be accomplished as agreed

7. 7 ©2014 U.S. Education Delivery Institute To prep for stocktakes or otherwise understand progress (or lack thereof) goal teams may assess the likelihood of delivery Assessment Framework Red Amber/Red Amber/Green Green Highly problematic – requires urgent and decisive action Problematic – requires substantial attention, some aspects need urgent attention Mixed – aspect(s) require substantial attention, some good Good – requires refinement and systematic implementation Recent performance against trajectory and milestones Likelihood of delivery Key Degree of challenge Quality of planning, implementation and performance management Capacity to drive progress Stage of delivery L/M/H/VH 1/2/3/4 JudgementRatingRationale Summary

8. 8 ©2014 U.S. Education Delivery Institute To arrive at a likelihood of delivery, we use this Assessment Framework

9. 9 ©2014 U.S. Education Delivery Institute Judgments can be rolled up for a succinct message to the system leader.

10. ODE School Improvement How are schools doing? (HASD) Quarter 1 Review December 13, 2013

11. 11 ©2014 U.S. Education Delivery Institute ▪ Understand overall goal trajectory and progress ▪ Review the overall picture of schools ▪ RNCs reflect on school status (region bright spots & challenge areas) ▪ Review schools by region – Trends – Bright spots – Problem solving areas – So what do we do about it? – Review next steps ▪ Reflect on the routine and identify next steps The Turnaround routine had these objectives How are schools doing?

12. 12 ©2014 U.S. Education Delivery Institute Roles and responsibilities in the Oregon “How are schools doing?” turnaround routine ODE Director of School Improvement ▪ Asks tough questions that challenge and support ▪ Actively engages in problem-solving ▪ Holds others accountable for results Education Northwest Director ▪ Asks tough questions ▪ Ensure that coaches are providing support and building capacity ▪ Provide support to RNCs and coaches ODE Staff Support ▪ Designs agenda, keeps meeting on track ▪ Prepares data and evaluations ▪ Works with ODE Director, RNCs, and coaches to prepare ODE Education Specialists ▪ Holds day-to-day accountability for the plan’s success ▪ Manages a strategy for ODE Turnaround ▪ Shares project insights into school implementation RNC ▪ Holds day-to-day accountability for the plan’s success ▪ Manages the coaches to implement supports ▪ Works with ODE and provides evidence for current assessment of progress Coaches ▪ Direct support to schools ▪ Holds day-to-day accountability for the plan’s success ▪ Provides current assessment of progress in schools

13. 13 ©2014 U.S. Education Delivery Institute 8:45 - 9:25  Understand overall goal progress  Review the overall picture of schools 9:25 - 9:40  RNCs reflect on school status (region bright spots & challenge areas) 9:40 – 11:10 (15 min per region)  Review schools by region − Trends − Bright spots − Problem solving areas − So what do we do about it? − Review next steps 11:10-11:30  Reflect on the routine  Identify next steps The routine started with a review of overall progress How are schools doing?

14. 14 ©2014 U.S. Education Delivery Institute The routine addresses two major goals for school turnaround and four major strategies to support schools ODE Goals…Support to schools… 1.Overall rating Focus and Priority Schools rated a level 3 or better on their state school report card 50% by June 2014 75% by June 2015 2. Student growth Priority and Focus schools will achieve growth for all students and for all subgroups greater than the state average 50% by June 2014 75% by June 2015 ▪ Network ▪ CAP plans ▪ SST pilot ▪ Supports and Interventions Routine Frequency ▪ Four times a year – Beginning Dec, end Feb, end May, and end Aug – Sessions lasting 2-4 hours

15. 15 ©2014 U.S. Education Delivery Institute The team looked at overall progress on the top-level metrics… Overall school ratings…Overall growth and subgroup growth…

16. 16 ©2014 U.S. Education Delivery Institute …And then focused on interim data to understand progress towards improving results Data Included in Routine ▪ School and strategy assessment framework ratings ▪ EGMS ▪ Indistar ▪ CAP Reviews ▪ Formative Assessment Data – 70 schools reported reading data – 37 schools reported math data Data Missing in Routine ▪ 5 coaches did not submit survey data – East Gresham – Glenfair – Ontario – Community – Cascade ▪ 2 schools are new and do not have a coach (no information reported) – Nixyaawii – Roosevelt ▪ SST data is only included for 2 schools

17. 17 ©2014 U.S. Education Delivery Institute Coaches rated the likelihood of strategies to be able to move their school to the overall rating goals At the strategy level, evidence of progress is the area of greatest need.

18. 18 ©2014 U.S. Education Delivery Institute Coaches used the assessment framework to assess the likelihood of delivery overall and of each school’s strategies Likelihood of delivery Quality of planning Capacity to drive progress Evidence of effectiveness JudgementRatingDefinition Planning includes strategies and measures of progress. How/how much does each strategy impact goal? How well are we engaging with all of those involved from leader to student? How are we building capacity to drive implementation? What do the latest data say about our progress on the goal and its underlying strategies? Red Orange Yellow Green Highly problematic – requires urgent and decisive action Problematic – requires substantial attention, some aspects need urgent attention Mixed – aspect(s) require substantial attention, some good Good – requires refinement and systematic implementation Key

19. 19 ©2014 U.S. Education Delivery Institute Most schools reported their formative assessment results, but some were not even collecting it Regions show quite variation in the % at core.

20. 20 ©2014 U.S. Education Delivery Institute Fourteen schools reported Q1 and Q2 Reading Formative Assessment data Some schools are testing all students, some are progress monitoring a subset.

21. 21 ©2014 U.S. Education Delivery Institute These indicators were used to assess the level of rigor in the schools’ planning…

22. 22 ©2014 U.S. Education Delivery Institute …and this one was used to determine whether schools were spending the funds available to them Schools that claimed <100% in school year 2012-13

23. 23 ©2014 U.S. Education Delivery Institute 8:45 - 9:25  Understand overall goal progress  Review the overall picture of schools 9:25 - 9:40  RNCs reflect on school status (region bright spots & challenge areas) 9:40 – 11:10 (15 min per region)  Review schools by region − Trends − Bright spots − Problem solving areas − So what do we do about it? − Review next steps 11:10-11:30  Reflect on the routine  Identify next steps The RNCs made some overall reflections… How are schools doing?

24. 24 ©2014 U.S. Education Delivery Institute 8:45 - 9:25  Understand overall goal progress  Review the overall picture of schools 9:25 - 9:40  RNCs reflect on school status (region bright spots & challenge areas) 9:40 – 11:10 (15 min per region)  Review schools by region − Trends − Bright spots − Problem solving areas − So what do we do about it? − Review next steps 11:10-11:30  Reflect on the routine  Identify next steps …And then reviewed each region to determine next steps How are schools doing?

25. 25 ©2014 U.S. Education Delivery Institute ▪ What is one school that is a bright spot? What is going well? ▪ What are the top 2 most challenged schools? ▪ What is most critical to improvement? ▪ What next steps are needed to support these schools? These questions guided the discussion in each region

26. 26 ©2014 U.S. Education Delivery Institute Region Strategy Implementation Data

27. 27 ©2014 U.S. Education Delivery Institute Region: Bright Spot

28. 28 ©2014 U.S. Education Delivery Institute Region: Challenging School of Focus

29. 29 ©2014 U.S. Education Delivery Institute Region: Challenging School of Focus

30. 30 ©2014 U.S. Education Delivery Institute The team discussed how the routine went after discussing each region ▪ What worked about this routine? ▪ What we can we improve or change? Agree on a common purpose Arrive at a shared view of performance and progress Identify and solve problems Encourage learning and collaboration Identify and commit to clear next steps

31. 31 ©2014 U.S. Education Delivery Institute Exercise: Using the assessment framework WhatHowMaterials ▪ We are rating our likelihood of implementing the following outcome: Happy hour will start promptly at 5 pm today. ▪ Use the assessment framework to come to a rating on overall likelihood of success by thinking about the quality of planning, capacity, and evidence related to this goal. ▪ Share and calibrate the ratings ▪ Individually ▪5▪5 Time ▪ Assessment framework ▪ Flipchart ▪ Dots ▪ Whole group ▪ 10

32. 32 ©2014 U.S. Education Delivery Institute Hawai’i has seen encouraging results in its Race to the Top work and on NAEP ▪ Went from High Risk status in Race to the Top to being removed from any risk status in two years ▪ The NAEP 4 th and 8 th grade math average scale score increased 4 points from 2011 to 2013 ▪ NAEP 8 th grade reading average scale score increased 3 points from 2011 to 2013 Recent Hawai’i progress Note: Increases in NAEP scores are statistically significant according to NCES.

33. 33 ©2014 U.S. Education Delivery Institute Hawai’i‘s Educators are focused on the same six strategies at all levels Hawai’i‘s six priority strategies Academic Review Teams Common Core State Standards Comprehensive Student Supports Educator Effectiveness Systems Formative Instruction/ Data Teams Induction and Mentoring State, complex areas, and school levels

34. 34 ©2014 U.S. Education Delivery Institute

35. 35 ©2014 U.S. Education Delivery Institute The state team established routines where they use the implementation data to inform decisions

36. 36 ©2014 U.S. Education Delivery Institute During routines, school and complex area leaders come to a shared view of progress on each priority strategy Excerpt from strategy assessment rubric

37. 37 ©2014 U.S. Education Delivery Institute Data are then organized into complex area summaries and statewide reports that show field implementation progress Complex area and statewide field indicator summaries, September 2013

38. 38 ©2014 U.S. Education Delivery Institute The state team collects judgments from school routines Reporting structure for Hawai’i implementation data ▪ Come to shared view of progress ▪ Reports to CA ART lead School-level routines Complex Area ART Lead State ART Lead State level routines Targeted state support ▪ Collects school level data ▪ Discusses results with complex area leaders ▪ Reports to state ART lead ▪ Compiles data ▪ Generates report summaries for state level routines ▪ Assess school and CA level progress ▪ Determine next steps of support ▪ Provide necessary support to CAs and schools

39. Thank You