UTILIZING IMPLEMENTATION SCIENCE FRAMEWORKS TO ENSURE EFFECTIVE BEHAVIORAL & INSTRUCTIONAL SUPPORT SYSTEMS (EBISS) SPRING 2016 PENDLETON, VALE AND REDMOND OREGON WELCOME OREGON SCALING-UP EBISS Oregon

MARICK TEDESCO, PH.D. STATE TRANSFORMATION SPECIALIST SARA FALCON, PH.D. OREGON EBISS INITIATIVE INTRODUCTIONS

1.PROVIDE A REVIEW OF THE CRITICAL FEATURES OF EFFECTIVE BEHAVIORAL AND INSTRUCTIONAL SUPPORT SYSTEMS (EBISS) 2.INTRODUCE KEY CONCEPTS OF IMPLEMENTATION SCIENCE (IS) 3.DEVELOP UNDERSTANDING OF THE FRAMEWORKS ASSOCIATED WITH IS 4.IDENTIFY NEXT STEPS FOR YOUR TARGET INITIATIVE GOALS FOR TODAY

INTRODUCTIONS DISTRICTS District Number of team members Team member roles

AGENDA 8:00 – 8:30Registration and Pre Test 8:30 – 9:00Introductions, Data requirements review 9:00 – 10:15Competency Drivers – Staff Selection 10:15 – 10:30Break 10:30 – 11:30Organization Drivers 11:30 – 12:00 Improvement Cycles 12:00 – 1:00Lunch 1:00 – 2:00 Terms of Reference 2:00 – 2:15Post Test and Evaluation 2:15 – 3:45Team Time

ACTIVE IMPLEMENTATION FRAMEWORKS WHEN Stages WHO Teams WHAT Intervention HOW Cycles HOW Drivers

Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Competency Drivers Organization Drivers Leadership IMPROVED EDUCATIONAL OUTCOMES Interventions meet Implementation Consistent Use of Educational Innovations © Fixsen & Blase, 2008 Adaptive Technical

THERE ARE 3 CATEGORIES OF IMPLEMENTATION DRIVERS: Competency Drivers – are mechanisms to develop, improve and sustain one’s ability to implement an intervention as intended in order to benefit students. Organization Drivers – are mechanisms to create and sustain hospitable organizational and system environments for effective educational services Leadership Driver – focus on providing the right leadership strategies for different types of leadership challenges. These leadership challenges often emerge as part of the change management process needed to make decisions, provide guidance, and support organization functioning.

IMPLEMENTATION DRIVERS COMMON FEATURES OF SUCCESSFUL SUPPORTS TO HELP MAKE FULL AND EFFECTIVE USE OF A WIDE VARIETY OF INNOVATIONS

Core Implementation Components © Fixsen & Blase, 2008 Positive Outcomes for Students Competency Drivers Organization Drivers Leadership Effective Educational Practices How : What : Why : Professional Development/ Professional Learning Professional Development/ Professional Learning Capacity to provide direction and vision

Performance Assessment (Fidelity) Coaching Training Selection Competency Drivers © Fixsen & Blase, 2008 Implementation Drivers

STAFF SELECTION Selection of staff, from an Active Implementation perspective is different from “selection as usual” in two important ways: Mutual Selection Behavioral Rehersals and Role Play components of the interview process Competency Driver

STAFF SELECTION Selection is viewed as a “mutual” process. For example, a school or district is deciding whether or not to select an individual to join them and the process allows the applicant to understand the expectations related to the position in order to determine if their skills abilities and attitudes are a good “fit”. Competency Driver

STAFF SELECTION Selection includes “role play” or “behavior rehearsal” processes. These processes allow interviewers to observe how applicants respond to feedback and how able and willing they are to learn new practices. This process also provides insight into how an applicant might respond to feedback and data. A desire and ability to learn and grow are critical for ongoing improvement. Competency Driver

STAFF SELECTION Think about your recruitment, interviewing, and selection process. After reviewing the two key distinctions discussed in the slides above, what are two things you could do to improve the Selection process (including recruitment, interviewing, and selection) in your setting so that it better supports effective implementation? Competency Driver

SELECTION Purposes Select for the “tough to teach traits” Screen for pre-requisites Set expectations for new hires – use of data, coaching Allow for mutual selection Improve likelihood of retention after “investment” Improve likelihood that training, coaching, and supervision will result in implementation Implementation Drivers Competency Drivers

SELECTION DRIVER CHALLENGES “We have who we have …this doesn’t apply to us!” What’s the way forward? Implementation Drivers Competency Drivers

INTEGRATED AND COMPENSATORY IMPLEMENTATION DRIVERS Integrated Consistency in philosophy, goals, knowledge and skills across these processes (S/T/C/SE/DSDS/FA/SI) Compensatory At the practitioner level At the program level Competency Drivers Organization Drivers Leadership Drivers Integrated & Compensator y

BENEFITS OF DRIVER-BASED ACTION PLANNING Infrastructure needed becomes visible to all Strengths and progress get celebrated Next right steps are planned and results measured Resources can be aligned and re-purposed to improve implementation Implementation Drivers Action Plans

NEXT STEPS Locate the Staff Selection activity to review job description and interview samples. For the purpose of this module, consider that these represent selection practices from a single district. Utilize the Staff Selection best practices self-assessment to rate the samples discussed above. Develop two action items to address a category that needs improvement.

ORGANIZATION DRIVERS Change Organizations and Systems Create and sustain hospitable organizational and system environments for effective services Develop functional data systems that can be used to inform decision-making Implementation Drivers Organization Drivers

All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get.“ — R. Spencer Darling Business Expert ORGANIZATIONAL CHANGE

SYSTEM CHANGE EXISTING SYSTEM EFFECTIVE INNOVATIONS ARE CHANGED TO FIT THE SYSTEM EXISTING SYSTEM IS CHANGED TO SUPPORT THE EFFECTIVENESS OF THE INNOVATION EFFECTIVE INNOVATION

SYSTEM CHANGE Creating Capacity for Competent Change New innovations do not fare well in current organizational structures and systems Develop new position descriptions and job functions in State Departments of Education and in Regional and District systems. “Systems trump programs.” —Patrick McCarthy, Annie E. Casey Foundation

Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Competency Drivers Organization Drivers IMPLEMENTATION DRIVERS © Fixsen & Blase, 2008

DECISION SUPPORT DATA SYSTEMS Purposes Monitor and improve student outcomes through data-based decisions Provide information to assess effectiveness of intervention and prevention practices Analyze the relationship of fidelity to outcomes Guide further program development Detect discrete issues as well as systemic issues Engage in continuous quality improvement Of the Intervention and the Drivers Celebrate success Be accountable to parents, Board of Education taxpayers, and other funders Implementation Drivers Organization Drivers

Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Competency Drivers Organization Drivers IMPLEMENTATION DRIVERS © Fixsen & Blase, 2008

FACILITATIVE ADMINISTRATION Purposes Creates an internally hospitable environment for the new way of work – at the level of the “agency” (e.g. school, District) Facilitates the installation, implementation, and improvement of the Drivers for each innovation Takes the lead on Systems Interventions Looks for ways to make the direct work of practitioners (e.g. teachers, school staff) and administrators more effective and less “burdensome”!! Implementation Drivers Organization Drivers

Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Competency Drivers Organization Drivers IMPLEMENTATION DRIVERS © Fixsen & Blase, 2008

SYSTEMS INTERVENTION Purposes Identify and “lift up” systemic barriers and facilitators to the next level to improve support for the new way of work Create an externally “hospitable” environment for the new way of work Embed facilitators and strengths Contribute to cumulative learning in multi-site projects Implementation Drivers Organization Drivers

CHALLENGES Data Systems that.. Warehouse the data and are not timely Produce reports that are not provided at the actionable unit And… Site-based administration that is not responsive to feedback Systems that block feedback or do not respond to it Organizational Drivers Organization Drivers

BIG IDEAS THAT HELP Linked Teaming Structures Linking Communication Protocols (e.g. agenda links, meetings dedicated to feedback) Terms of Reference Practice to Policy Feedback surveys and action plans Organizational Drivers Organization Drivers

Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Competency Drivers Organization Drivers IMPLEMENTATION DRIVERS © Fixsen & Blase, 2008 Leadership AdaptiveTechnical

MANAGING THE FLOW Technical Adaptive

DRIVERS INTEGRATION WITH OTHER FRAMEWORKS Implementation Drivers Teams Locus of Responsibility Monitor Integration Cycles Within a Driver Among Drivers Feed Forward Feedback Stages Exploration Installation Initial Implementation Full Implementation

INTEGRATED AND COMPENSATORY IMPLEMENTATION DRIVERS Integrated Consistency in philosophy, goals, knowledge and skills across these processes (S/T/C/SE/DSDS/FA/SI) Compensatory At the practitioner level At the program level Competency Drivers Organization Drivers Leadership Drivers Integrated & Compensator y

BENEFITS OF DRIVER-BASED ACTION PLANNING Infrastructure needed becomes visible to all Strengths and progress get celebrated Next right steps are planned and results measured Resources can be aligned and re-purposed to improve implementation Implementation Drivers Action Plans

NEXT STEP Utilize the Organization Drivers Mapping Tool with your team to determine next steps for assessment and action planning.

Active Implementation What about Improvement Cycles? HOW Cycles

WHAT HAPPENS PRIOR TO IMPLEMENTING A PDSA CYCLE? 1.Team convenes to decide on a change objective 2.Team works to understand the problem 3.Team brainstorms a number of possible changes 4.Possibilities are prioritized and one is selected for testing 5.The test occurs in one portion of the agency 40

Shewhart (1924); Deming & Juran (1948); Six-Sigma (1990) Plan – Decide what to do Do – Do it (be sure you did it as intended) Study – Look at the results Act – Make adjustments Cycle – Do over and over again until the intended benefits are realized PDSA Cycles: Trial & Learning

IMPROVEMENT CYCLE USES Rapid Cycle For a Particular Problem Practice Improvement Usability Testing Practice-Policy Feedback Loops Transformation Zones

IMPROVEMENT CYCLE USES Rapid Cycle For a Particular Problem Practice Improvement Usability Testing Practice-Policy Feedback Loops Transformation Zones

RAPID CYCLE Problem-solving Team Lead identified Right people on the team Time-limited to address the problem “Solves” the problem Team disbands

RAPID CYCLE Practice Improvement Relatively short-cycle On-going efforts to improve practices and competencies Define the Problem Define Metrics Measure Intervene Measure Again Use data to achieve better outcomes and “embed” solutions

IMPROVEMENT CYCLE USES Rapid Cycle For a Particular Problem Practice Improvement Usability Testing Practice-Policy Feedback Loops Transformation Zones

P URPOSES WHY IS IT HELPFUL? WHAT’S THE PURPOSE? The overarching purposes of usability testing are to improve and better operationalize “Challenging” intervention components. The Implementation Drivers (especially competency drivers) Fidelity and outcome data collection processes So that major ‘bugs’ are worked out of the processes and the processes can be effective 47

USABILITY TESTING PROCESS 4 or 5 “typical users” to try out the current version of the implementation methods and/or the innovation practices Fix the problems from the first “trial” Repeat with a new group of 4 or 5 typical users Fix the problems form the second “trial” After 4 or 5 “trials” you should have most of the problems solved

RAPID-CYCLE TESTING Rapid-Cycle changes  Quick experiments; do-able within 2 weeks PDSA cycles  P lan the change  D o the plan  S tudy the results  A ct on the new knowledge 49

PLAN Define the change behaviorally… precisely what will be done? Who will implement the change? What preparation needs to be done before starting the change? Clarify who will measure the change and who will review the data regularly to share with the team. 50

DO Note the start date for the cycle How long will the initial test last? How often will the team meet to assess progress and review data? During the test, the team considers what improvements might need to be made to improve the results 51

RAPID-CYCLE TESTING - EXAMPLE Rapid-Cycle change  “Barrier question” PDSA cycle with 2 Principals PDSA cycles  P lan: Design question  D o: 2 schools use on 1 grade level each  S tudy: 70% of teachers respond positively to 2 post-observation sessions compared to 50%  A ct: Try with another 8-10 teachers 52 8

STUDY After the agreed upon test period the team reviews the results Team leader assembles and graphs the data for the team The team deliberates what it has learned Based on the learning, the team considers whether a change in strategy is in order 53

ACT In light of what has been learned, the team decides what to do next Make an adjustment in the strategy to get closer to the objective Increase the objective if the initial target has been met and the team believes even more progress can be made Combine another change with the existing strategy to start a new cycle Abort the existing strategy and start a new cycle 54

IT IS IMPORTANT TO REMEMBER… It often takes more than one cycle to achieve your objective. By changing only 1 thing at a time you know the impact of your change. Sometimes several changes are necessary to maximize the improvement you seek. 55

CHANGE CYCLES 56 Hunches Theories Ideas Changes That Result in Improvement A P S D AP SD D S P A DATA Cycle 1 Cycle 2 Cycle 3 Cycle 4 AP SD

KEYS TO PDSA SUCCESS 1.Be clear about your target objective 2.Implement only 1 change at a time 3.Make sure all participants are implementing the change as planned 4.Implement the change in a small portion of the agency 5.Study the results data before making a modification to the plan 6.Do not hesitate to start a new cycle when the data indicates or the team is convinced that a modification will improve results 57

AND LAST… Keep measuring and studying the results until you reach your objective and determine you have improved as much as you can 58

ACTIVITY Utilize the PDSA Planning Form with your team to identify and document a plan for initiating a component of your targeted initiative. 59

IMPLEMENTATION STAGES Outline the integrated, non ‐ linear process of deciding to use an effective intervention and finally having it fully in place to realize the promised outcomes. Active implementation stages are Exploration, Installation, Initial Implementation and Full Implementation.

TERMS OF REFERENCE Terms of Reference: A Tool and Process for Clarity and Communication 61

SETTING THE STAGE FOR MULTI-LEVEL GROUP WORK Establishing Terms of Reference (ToR) Terms of Reference (ToR) are detailed in a written document Outlining the purpose of the project or group, How it will be structured and implemented. It is like an internal Memorandum of Understanding (MOU) Helps you clarify communication, mandates, and core features and functions of the group 62

TERMS OF REFERENCE ToR is quite frequently used in business and planning endeavors Creating a detailed ToR is a critical part of any project The ToR is an important Plan step in a PDSA Cycle for Group Work Do Plan Study Act 63

BENEFITS ToR can facilitate work because: It establishes common ground and a common understanding of why a group has been formed and what is expected. It is a proactive way to ensure that there is agreement about important dimensions of the project. It clarifies both the role of the team and may also clarify individual roles on the team Increases comfort and legitimacy of “all voices” on the team Such agreement, in the absence of work or issues, helps the group talk more openly about diverse opinions and reach consensus. 64

TOR FOR AGREEMENT CREATION: COMMON VALUES Details “values and ways of work” that the group will abide by/use/review throughout the project An opportunity – in the absence of the actual work to: Surface Challenges Get confusion on the table and move to clarity Reduce the likelihood of ‘distractions’ during the work of the group 65

KEY TOR COMPONENTS Vision for the Work Together Scope and Objectives Expected Outcomes and Deliverables Boundaries (what it is and is not, when are we done) and Limitations Authority, Accountability and Reporting Requirements Linking Communication Protocols for Alignment (with whom do we communicate, how, and how often and for what purpose) Roles and Functions of individuals (who participates in what ways) Leadership Term Membership Orientation for New Members Resources available to the project Decision-making process Values and Ways of Work Implementation Plans 66

USE IT OR LOSE IT (DO) If the document is actively used and referenced throughout the life of the project it can: Help maintain focus by: Using the ToR so that the project or group remains on track or on “mission” Serving as an aid to troubleshooting to maintain focus (e.g. “are we still aligned on values and way of work?”, “do we have the authority, people, resources we need right now?”) Serve as a ‘prompt’ before a meeting, a discussion, or a decision. Do Plan Study Act 67

USE IT OR LOSE IT (DO) If the document is actively used and referenced throughout the life of the project it can: Help refocus when group dynamics are bumpy (e.g. “I wonder if we should revisit our agreed upon way of work, we said…..but now I am hearing…, So have we changed our way of work or scope of work or have we strayed from our roles/responsibilities, values?”). Support “ getting on the balcony” with respect to group process, dynamics, and agreement creation – “let’s think about our end goals” Do Plan Study Act 68

REVISIT, REFLECT, REVISE Nothing is carved in stone Revisit and Reflect ( Study) In the absence of actual issues - Get on the Balcony Identify emerging challenges The very “work of the group” may call for a change in the ToR Opportunity to be sure all are heard Revise ( Act ) Change any elements to improve group functioning or the success of the overall project Go again (PDSA) Do Plan Study Act 69

NEXT RIGHT STEPS Getting Started How might you go about introducing the idea of Terms of Reference to a group or project team that is just forming (e.g. the Team’s role: New functions for District and School Leadership Teams)? What “case” could you make? Who would you talk with before ‘making the case’? Getting Re-started Is a current group, committee, or project “stuck” or wrestling with adaptive challenges? Could they benefit from “re-starting” by developing a Terms of Reference document? How and with whom might you begin? 70

ACTIVITY Team members to brainstorm a list of the characteristics of the ineffective meetings they’ve attended, and write it on chart paper. Then ask them to list the characteristics of effective meetings. Using the lists as a basis, ask team members to volunteer ideas for specific agreements needed to ensure effective team functioning. You may wish to suggest topics to address (e.g., attendance, agenda, minutes, meeting tasks, tasks between meetings, decision-making, handling conflict). Coach the members who introduce ideas to say precisely what they need from the group, and how agreement about it will promote individuals’ comfort and the team’s effectiveness. If no one says any- thing contradictory, you can accept all these needs as group agreements. 71

RESOURCE Complete the Developing Your Terms of Reference (TOR) worksheet with your team. 72

IMPLEMENTATION MATH Effective Interventions (The “WHAT”) Effective Implementation (The “HOW”) Positive Outcomes for Students (The “WHY”)

PLAN FOR CHANGE It is not a “school problem” District is the point of entry for sustainable school improvement Use short-term infusion of resources Establish long-term, district-based capacity for quality

75

IMPLEMENTATION DRIVERS ARE DYNAMIC! A key feature of Implementation Drivers is their integrated and compensatory nature: Integration – means that the philosophy, goals, knowledge and skills related to the program or practice are consistently and thoughtfully expressed in each of the Implementation Drivers. Compensatory – means that the skills and abilities not acquired or supported through one driver can be compensated for by the use of another driver.

Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Adaptive Technical Integrated & Compensatory Competency Drivers Organization Drivers Leadership IMPROVED EDUCATIONAL OUTCOMES Interventions meet Implementation Consistent Use of Educational Innovations © Fixsen & Blase, 2008

NEXT STEPS AND TIMELINES 78

TEAM TIME ACTIVITY: ACTION PLANNING CONTINUED 79