Establishing a District Evaluation Plan Susan Barrett sbarrett@midatlanticpbis.org Director, Mid-Atlantic PBIS Network Sheppard Pratt Health System

Outcomes District Team Clear function/purpose (aligned with SPP) Members with clear role and function Communication/feedback loops Implementation/Evaluation Plan

Professional Development Training Curriculum Training Capacity Site Selection Readiness Coaching Implementation and Evaluation Plan What data point should change? How will we track progress and show impact? Selection of the Tools Data Base PBS Surveys, PBS Eval Local data base Start here Marketing and Dissemination Annual Reports Presentations Newsletters

If something is worth doing once, it's worth building a tool to do it. Evaluation Template- What will the impact be? Identify Progress Monitoring Tools Tier 1 Tiered Fidelity Inventory School-wide Evaluation Tool (annual for 20%) Tier 2/3 I-SET

Implementation Tools How can we organize the work? Can we work smarter to provide coaching and technical assistance by tracking progress across phases? Can district and state use Phases of Implementation (POI) to track progress and assign resources?

Exploration Stage Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward Installation Resources needed to implement innovation with fidelity and desired outcomes are in place Initial Implementation Innovation is in place in schools, implementation largely guided by external TA providers Full Implementation- Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation Innovation and Sustainability Innovation is adapted to fit local context, innovation becomes more efficient and is integrated with other initiatives POI

POI Tools-Coach POI- Illinois- All Tiers PIC- Florida- All Tiers Implementation Phase Inventory- MD Tier 1- strong reliability

Lessons Learned Training sequence and materials are developed Refinement of implementation and deepening content expertise in the State, District and schools continuous and areas of improvement are identified by data A unified, expanded cadre of trainers and implementation coaches prepared to deliver professional development Trainers understand content and implementation science

Lessons Learned An aligned training and support system for both trainers and implementation coaches. Standardized training and coaching materials. Standardized data collection system, tools and processes with practice-level feedback loops at each level of the system for planning and continuous improvement.

Readiness Structures 4-6 months of “Pre-Work” Tier 1 Leadership Forum- Planning Phase Checklist SWIS Readiness Leaders, Coaches, New Teams walk in prepared with basic knowledge, data, SIP Advanced Tiers Readiness- completed by Local Coordinator and Team CICO Readiness, CICO SWIS Readiness Readiness T2/T3

Training Content- Example “Anchored on the tool” Overview Critical Feature Modules Trainer Presentation Workbook (self assess, activity , action step) Implementation Snapshot Team Workbook

Professional Development Blueprint… Lists and describe minimum core readiness, skill, and performance competencies for SWPBS trainers. Lists and describe minimum outcome competencies for SWPBS implementation by (a) school staff, (b) school leadership teams, and (c) coaching personnel. Provides self-assessment guides for monitoring training and coaching progress. Provides evaluation procedures for assessing training and coaching fidelity and integrity. Recommends schedules and sequence for training and coaching activities. Assumes that trainers have implementation experience and fluency with SWPBS practices and systems. Focuses on the school, district/region, and state as the context for sustained training implementation and support.

Rethinking Technical Assistance Moving from a case by case expert model to building expertise in the school Focus of all TA is on teaching the school team to solve problems or address challenges for themselves Shift from providing answers to asking questions Shift from developing plans to prompting plan development Shift from being viewed as the expert to being viewed as a facilitator Will not replace need for specialist, re-focus all to building capacity

Build parallel systemic processes Provide school/district teams with a process to address the presenting challenge (SWPBS) Develop a parallel process for districts/states to support school implementation and continue to expand with integrity (District /State Leadership Team)

CSBS Training Model Creating the Conditions for Skill Development Intro to Administrators Readiness/Planning Phase Team and Coach Training Event Facilitated Action Planning- Team Time Workbook Track Fidelity/Progress Access to Community of Practice-knowledge sharing Track Outcomes Building Training Capacity- Scope and Sequence Training Event Implementation Team Workbook

PBIS Building Capacity and Sustainability Project Leadership Team State Implementation Team Regional Implementation Team District Coordinators Coaches We will be working through exploration to build this system of support Team Leaders Problem-Solving Teams School Staff, Families, Transportation, Communities

State Example March 30 July 18-22 Feb- June Spring Forum Administrator attends July 18-22 New Team Training Feb- June Planning Phase for New Teams Checklist Completed Sept. 1st – Feb. 28th New SWIS and CICO-SWIS subscriptions setup and invoiced Sept Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Oct 21 State Coaches Meeting Nov 10 IPI Due Dec 6 State Coaches Meeting April 7 State Coaches Meeting April 10 IPI Due August Team Planning SWIS Readiness Completed Establishing Annual Training/Meeting Cycles ** Regional Returning Team Events held during Summer- Dates TBA **CICO Training completed Regional by request

Training Capacity Fidelity Delivering the training content as it is intended to be delivered is essential Participants who attend a PBIS training regardless of what part of the state they reside, should have the same training experience This is a challenge as scaling occurs TOT developmental process Learner, Content Fluent, Co-trainer, Trainer

Implementation Snapshot Used during training to clearly define each critical feature and the research that supports use and potential benefit to overall school climate. The trainer facilitator will provide training slides for each feature. Teams will have an opportunity to use their guidebook and items brought to the training to use during action planning sessions. Roles of each participant are clearly described each snapshot to ensure clear expectations. Administrators are expected to play an active role. Coaches are expected to guide process, communicate with the district coordinator and share implementation products and examples as needed. Teams will walk away from the training with an annual action plan that will help guide the effort.

Systems Practices/Skills Data Building Coaching Capacity State Occurs at ALL Levels Systems Conditions that support skill development for staff Policy and Procedures alignment Budget Re-allocation Recruitment and Selection of Coaches Supervision of Coaching within Organization Training Curriculum and Scope and Sequence Access to certification Facilitative Administrator Supports Practices/Skills The technical skill set required to achieve fidelity Problem Solving (Team, Classroom, Staff, Student) Team Building/Collaboration Delivering Feedback Behavioral Consultation State District Building Classroom Staff Student Family Data Information required to guide skill development process Action Plan with short/long term measurable goals Self Assessment Process Measures/Fidelity Checks Performance Feedback Measure Progress Monitoring Tools Evaluation Tools Student Outcomes Data used for continuous regeneration (PEP/PIP) 20

Breaking it down … Stages of Implementation State and District Level Implementation Guide

Stages of Implementation Exploration/Adoption Installation Initial Implementation Development Commitment Full Implementation Establish Leadership Teams, Set Up Data Systems Provide Significant Support to Implementers Innovation and Sustainability Embedding within Standard Practice Should we do it? Improvements: Increase Efficiency and Effectiveness Doing it right Doing it better

Exploration Stage Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward Installation Resources needed to implement innovation with fidelity and desired outcomes are in place Initial Implementation Innovation is in place in schools, implementation largely guided by external TA providers Full Implementation- Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation Innovation and Sustainability Innovation is adapted to fit local context, innovation becomes more efficient and is integrated with other initiatives

State District Implementation Guide Workbook organized by POI Specific outcomes for each Phase Guiding Questions to build case/consensus Facilitated Activities that guide similar TA process Never meant to be a “paper product”

Professional Development/ Technical Assistance Guiding Questions 2/20/2018 Exploration Professional Development/ Technical Assistance Guiding Questions Hyperlink to guided questions, then facilitated exploration activities, and then professional development/ta. After the professional development/ta slide, just advance forward to complete ppt presentation. Facilitated Exploration Activities Outcomes

Exploration Phase Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward

Before you add one more thing…. We plan for systems to support top two tiers .Creating foundations to support the maintenance of those behaviors. www.safetycenter.navy.mil

Exploration Phase Major emphasis is on long term commitment, innovation and implementation are discussed so that stakeholders understand system-level features necessary for initial and sustained implementation of the innovation

GOAL: 100% of students achieve TIER I: Core, Universal GOAL: 100% of students achieve at high levels Tier I: Implementing well researched programs and practices demonstrated to produce good outcomes for the majority of students. Tier I: Effective if at least 80% are meeting benchmarks with access to Core/Universal Instruction. Tier I: Begins with clear goals: What exactly do we expect all students to learn ? How will we know if and when they’ve learned it? How you we respond when some students don’t learn? How will we respond when some students have already learned? Questions 1 and 2 help us ensure a guaranteed and viable core curriculum Use Guiding Questions to get folks thinking in an integrated way District facilitates discussions among schools on curriculum standards preschool through12th grade. Systematic process for monitoring, evaluating, reviewing curriculum. Florida PBS 29 29 29

For approx. 20% of students TIER II: Supplemental, Targeted Tier II For approx. 20% of students Core + Supplemental …to achieve benchmarks Tier II Effective if at least 70-80% of students improve performance (i.e., gap is closing towards benchmark and/or progress monitoring standards). Where are the students performing now? Where do we want them to be? How long do we have to get them there? How much do they have to grow per year/monthly to get there? What resources will move them at that rate? District facilitates discussions among schools on curriculum standards preschool through12th grade. Systematic process for monitoring, evaluating, reviewing curriculum. 30 30 30

Intensive, Individualized Tier III For Approx 5% of Students Core + Supplemental Intensive Individual Instruction …to achieve benchmarks Where is the students performing now? Where do we want him to be? How long do we have to get him there? What supports has he received? What resources will move him at that rate? Tier III Effective if there is progress (i.e., gap closing) towards benchmark and/or progress monitoring goals. TIER III: Intensive, Individualized District facilitates discussions among schools on curriculum standards preschool through12th grade. Systematic process for monitoring, evaluating, reviewing curriculum. 31 31 31

Installation Resources needed to implement innovation with fidelity and desired outcomes are in place Identify: Coordinator State Team Implementation Team Trainers, Coaches Roles and Functions must be clear Use the Blueprint and Team Workbook

Initial Implementation Commitment to Capacity Building Demonstrated High Fidelity/High Impact Demand Increases State Team won’t be able to keep up with demand

Features Point of Contact and Coaches become Local Coordinators Transfer role to local person Use phase of implementation to guide decision points Meet with local team to build action plan- model after state team

District Level Phases Phase One: Commitment to Collaboration Role of District Role of State Leadership Team ID Point of Contact Small number of schools trained in PBIS Small number of coaches identified and trained Conduct Site visit to school two times/school year (completion of “implementation phase checklist”) Provide technical assistance to local coaches Provide SWIS training to schools as needed Maintain communication with Point of Contact Nominate Exemplar Nag for Forms Attend State Leadership Team Meetings monthly Complete SET and SET Report as needed Coordinate Regional Meetings for team leaders and coaches Phase Two: Commitment to Capacity Building More than 5% of schools trained in PBIS within district/region Coach facilitator or lead coach identified (% of time allocated for PBIS activities) Informal District/Regional team identified SWIS facilitators identified and trained SET Assessors identified and trained Provide support and technical assistance to coach facilitator (coach facilitator coordinates site visits, “implementation phase checklist”, SWIS facilitation, SET assessment and support to local coaches) Conduct site visits to schools by special request only Assist with district self assessment and action plan Assist with dissemination activities such as presentation to LMB, Community, Businesses, Superintendent Conduct capacity building awareness activities using PBIS Blueprint with key stakeholders Submit formal reports to PBIS management team to include progress notes, updates, training needs Conduct needs assessment Conduct SWIS facilitator training, SET assessor training Phase Three: Commitment to Large Scale Implementation Identification of PBIS Coordinator Working Budget District/Regional Action Plan in place to address capacity building around training, coaching, evaluation, coordination District/Regional PBIS functioning team Development of database Progress Report in development Development of newsletters, brochure and other marketing material Alignment with LSS Master Plan, Charter, Code of Conduct Formal report cycle to key stakeholders (superintendent, board, LMB, community/business, Parent organizations) Active communication and coordination with PBIS Coordinator and PBIS District/Regional Team Attendance to district/regional monthly meetings Identify resources such as grant opportunities Promote visibility to State Assist with marketing/dissemination/policy

Full Implementation Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation “New” is now standard practice

Innovation and Sustainability Innovation and Integration Demonstrated impact throughout Change/Adapt to fit culture every year Renew Commitment Easier, More Efficient, Cost Reduces Organizational Framework allows for integration Educators as better consumers

Hybrid Training: Face to Face combined with Use of Technology Training – always an evolving process…. Continuous opportunities to deepen both content and implementation knowledge and skills. Establish training cycle that includes: Face to face training and planning Storage space Adobe connect

Using a Google Drive, Dropbox “Fast” access to content/tools/resources Targets group needs Build with group Interactive or Storage Space