Implementing intensive intervention: Lessons learned from five years of technical assistance Council for Exceptional Children April 2017: Boston, MA

Introductions Allison Gruner Gandhi, Ed.D., Interim co-Director of the National Center on Intensive Intervention (NCII) Laura Kuchle, Ph.D., NCII, Evaluation Coordinator Chris Lemons, Ph.D., Senior Advisor to NCII and Assistant Professor in the Department of Special Education at Peabody College of Vanderbilt University Leslie Anderson, M.Ed., Director of Pupil Personnel Services, Bristol Warren Regional Rhode Island Douglas Fuchs, Ph.D., Senior Advisor to NCII and Professor and Nicholas Hobbs Chair in Special Education and Human Development at Peabody College of Vanderbilt University

National Center on Intensive Intervention Allison Gandhi, NCII

National Center on Intensive Intervention (NCII)

What is intensive intervention? Intensive intervention is designed to address severe and persistent learning or behavior difficulties. Intensive interventions should be— Driven by data Characterized by increased intensity (e.g., smaller group, expanded time) and individualization of academic instruction and/or behavioral supports Read slide to highlight distinguishing characteristics of intensive intervention.

Multi-tiered Systems of Support (MTSS) Policies and practices within schools should ensure that students with disabilities have access to intensive intervention. When students with disabilities do not respond to Tier 1 and Tier 2 supports, they must have access to intensive intervention if they require it. Students may receive services at different levels of support in different areas. For example, a student with a learning disability in reading may need intensive intervention in reading while needing only core instruction in mathematics. Students with disabilities who are not making adequate progress in their current instructional program Students who present with very low academic achievement and/or high-intensity or high-frequency behavior problems (typically those with disabilities) Students in a tiered intervention system who have not responded to secondary intervention programs delivered with fidelity

What is NCII’s approach to intensive intervention? DBI graphic What is NCII’s approach to intensive intervention? Data-Based Individualization (DBI): A systematic method for using data to determine when and how to provide more intensive intervention

NCII Technical Assistance Intensive Training modules, guides, tool kits, rubrics, webinars, briefs, and videos Targeted Presentations, keynotes, conferences, booster trainings, coaches corner Universal NCII website, guides, ATE, Tools Chart NCII Activities included— Intensive Technical Assistance to 26 schools; 12 districts; 4 states over five years 400+ professional development events Dissemination of universal TA resources

New Center Kicking Off

NCII 2.0 (2016-2021) Extend work from school/LEA to SEA level Extend work to institutions of higher education Emphasis on integrated academic and behavior planning Address issues of identification and intervention for culturally and linguistically diverse students

Assessing DBI Implementation in NCII Intensive TA Schools Laura Berry Kuchle, NCII

What Is Fidelity? Extent to which an intervention’s core components have been implemented as planned (Nelson et al., 2012) For DBI, this includes the instructional platform, adapted iterations of intervention, ongoing progress monitoring, and decision-making procedures. See IRIS module for additional information: http://iris.peabody.vanderbilt.edu/module/fid/

Levels of Fidelity Level Key Questions Example Assessment Methods Student Are assessment and intervention being carried out as planned? Teacher logs Observations Record review Systems (school, team) Are essential components of DBI being implemented consistently? Are there systems-level problems that hinder DBI implementation? Rubric or interview with record review Record review might include student plans and graphs, team meeting notes or checklists, logs, written procedures

Why Does Fidelity Matter? Fidelity assessment provides Evidence that DBI is being implemented as intended Guidance on how to improve DBI implementation Does the interventionist or team need additional training or support? Is there a systems-level problem? For example Scheduling prevents sufficient intervention time Staff do not have access to evidence-based instructional platforms or rigorous assessment tools We need to know about implementation in order to interpret student outcomes. Training support could be around DBI process (e.g., using data to make decisions) Intervention Assessment

Measuring School-Level Implementation (Pulse Checks) NCII’s DBI Implementation Rubric and Interview Aligned with the essential components of DBI Reflects the infrastructure needed for successful implementation in Grades K–6 (academics and behavior) Identifies structures, resources, and practices needed for quality school-level implementation of DBI Reflect lessons learned from NCII knowledge development sites Intensive intervention is most likely to be facilitated when implemented as a component of a multi-tiered system of support (MTSS) Family engagement supports implementation Intensive behavioral intervention often is more challenging because of limited progress monitoring Inconsistent decision rules for intensifying supports can hinder intervention planning and resource allocation Hidden inefficiencies may overtax staff unnecessarily

Components/Infrastructure Required for DBI Implementation System Features Data and Decision Making Intervention DBI Process DBI Evaluation 26 items

Rating Scale 1–5 points with anchors for 1 point = little or no implementation 3 points = partial or inconsistent implementation 5 points = complete and consistent implementation Can score 2 or 4 when the school falls between anchors. Based on the structure of the Center on Response to Intervention’s RTI Essential Components Integrity Rubric http://www.rti4success.org/resource/essential-components-rti-integrity-rubric-and-worksheet

DBI Implementation Interview Script and note-taking template for gathering information to evaluate rubric Sample questions for each rubric item

DBI Implementation “Pulse Checks” Interviewed school DBI implementation teams in 2014, 2015, 2016 Rubric and interview revised twice Pulse Check Meetings Challenge: changing sample Two facilitators led interview: NCII staff and consultants, including NCII school coach. Co-rated to examine interrater agreement 2014: 17 schools. Ongoing NCII training and coaching for approximately 1–2 school years 2015: 16 schools. Second interview for 14 schools. First interview for 2 schools 2016: 19 schools. 2 new schools Revisions Challenge: Changing protocols Worked with TA team to clarify language and better align interview questions to rubric anchors. Version 1 – 17 schools in 2014. Version 2 – 13 schools in 2015. Refined 2 items with lower interrater agreement. Added item on participation of SWDs. Version 3 – 3 schools in Fall 2015, 19 schools in 2016. Separated leadership item into separate items for school and district leadership. Interview supports separate ratings for reading, math, and behavior Interrater agreement varied by item but improved over time. The mean difference between two raters scoring the same item for the same school decreased from .51 to .48 to .40. For version 3, raters had perfect agreement 63% and agreement within 1 point 98% (ranging from 92-100% at the item level).

Overall Scores 2014 2015 2016 Average Rating 2.99 3.24 3.56 Minimum School Score 2.17 1.83 1.66 Maximum School Score 3.58 4.26 4.31 Notes: Samples varied across years. For schools rated in multiple implementation areas using V3, means shown here reflect averages across areas. Challenge: Samples varied across years. We had 17 schools in 2014, 16 in 2015, and 19 in 2017. We lost some of our original schools while adding expansion schools. Only 14 schools were measured all three years. Even within a school, teams changed. Note: When looking only at our 14 consistent schools, the means are very similar to what you see here. Overall means: 2014: 3.05 2015: 3.27 2016: 3.49

Changes in Scores Year to year changes vary across schools and items Why might some scores drop? Staff turnover – changes in team or leadership Change in area(s) of implementation being rated Better understanding of what strong implementation looks like In 2015-16, one district faced significant changes in school and district leadership. These two schools had the two lowest mean scores so far in 2016, lowering the overall mean from 3.76 to 3.54.

High-Scoring Items Historically high and mean rating above 4 in 2016 DBI team composition School Leadership support for DBI DBI team regular meeting Academic progress-monitoring tools New for 2016: Participation of SWDs (M=4.0) In general, schools reported relative strengths in the areas of… A regular meeting schedule for teams supporting DBI decision making DBI decisions made by appropriate team or with support of needed expertise Technically rigorous academic progress-monitoring tools that are sensitive to student improvement District and school leadership support for DBI For 2016, stronger school (4.23) than district leadership (3.52) support We are now seeing higher scores for SWDs having access to DBI and infusion of DBI strategies into IEPs for those needing more intensive supports. 2015= 3.5

Low-Scoring Items Historically low and mean rating below 3 in 2016 Assessment fidelity Intervention fidelity Evaluation Decision rules Previously low items showing improvement in 2016: overall implementation of the DBI process (M=3.19) and behavioral PM tools (M=3.40) Last year overall implementation was 2.72 and behavior PM was 2.7

Using Pulse Checks to Improve Intensive TA Interview findings informed TA plans and coaching Informed our approach to intensive TA Debrief with interview facilitators and coaches informed identification of essential elements of DBI implementation Implications for expanding to new schools Beginning in 2014-15, incorporated interview findings into TA plans (now more individualized at school level). Facilitators recommended next steps.

Essential Elements of DBI Implementation Staff commitment Student plans Student meetings Valid, reliable data Inclusion of students with disabilities Some intensive TA districts are expanding to additional schools. We now share these essential elements to help them make an informed commitment.

Lessons Learned: Five Years of Implementation Chris Lemons, Vanderbilt University

A picture of the NCII resources page

Lessons Learned Data to inform lessons learned came from a variety of sources including Annual “pulse check” interviews with site-based intervention teams Interviews with practitioners and leadership Lessons learned manuscript that compiled themes from individual sites and from across sites

#1 Support from Leadership is Essential Strong administrator support from general and special education at the district and campus level is essential Leadership support helps ensure DBI is ‘connected’ or ‘embedded’ in the district’s or school’s broader initiatives

#2 Solid Tier 1 and 2 Foundations Allow School Staff to Focus Efforts on DBI We can not wait to implement DBI until Tier I and Tier II are perfect However……. instructional foundations and systemic processes for Tier I and Tier II should ideally be in place for intensive intervention to work best

#3 Planning for Implementation Start small and move forward one step at a time. It is better to have initial successes than to overcommit and end up frustrated Planning needs to take into account realistic challenges

#4 Formalizing Procedures Helps Ensure Ongoing DBI Implementation. Strong standard programs and clear procedures helped sites focus more time on students’ response to intervention and problem solve needed adaptations Established protocols (e.g., scripted meeting agendas) and systematized procedures helped when there was staff turnover

#5 Trust the process Implementation is hard work. It requires commitment and creative thinking Commit to implementing the process for a given period of time and know that it will likely be bumpy along the way— so, don’t give up.

Director of Student Support Get ready because it is hard work. Don’t think there is anything in this process that does not challenge your intellect…and professional knowledge. You need a level of commitment that even when it gets difficult, you will not sacrifice the time you’ve set aside or the direction that you’ve given in terms of implementation of intervention. This has to be a priority… The administrator has to be able and willing to commit resources… in order to be able to maintain fidelity to the plan.

MTSS Coordinator Data-based decision-making is not for wimps…You really got to go in knowing that it’s hard work….to really look at the data …Once you’ve made that commitment you can’t just give up…The only answer we’ve found so far is sometimes just going ahead and doing what we need to do and having the people look at the success.

District Level Support Leslie Anderson, M.Ed. Director of Pupil Personnel Services, Bristol Warren Regional, Rhode Island

NCII Implementation Phases 2012-2013 Year 1 Learn the Process 2013-2014 Year 2 Practice the Process (1 student) 2014-2015 Year 3 Implement the Process (5 students) 2015-2016 Year 4 Full Implementation School Wide with Coaching 2016-2017 Year 5 Independent School Wide Practice

Functions of Leadership in Systematic Change Consensus Infrastructure Implementation Support for individuals experiencing change Professional development to improve knowledge and skills Engaged building level leadership Efficient structures Effective systems for instruction

Critical Features of DBI Implementation Focus on Continuous Improvement Active support and involvement of district and building level leadership Administrative and staff commitment Strong Leadership Commitment based on clear understanding and acceptance of “non-negotiables” Alignment of intensive intervention efforts with existing initiatives: RTI, MTSS, etc. Ongoing Professional Development From the implementation work 39

Implementation Experience Success! School A Minimal Success! School B Building leadership “buy in”/participation Vocal teachers Full participation by teachers Understanding of the “work” and rolling up their sleeves 2 years with no leadership “buy in” Teachers generally passive “just tell me what to do and I’ll do it Looking to others for answers

Critical Features of DBI /Procedure and Protocol w Efficient and Effective Meetings Consistent, clear expectations for staff from administration Have a specific time to meet and meet frequently. Adhere to meeting protocols and roles Use technology to: Provide easy access student information and data entry Make the plan-creation process transparent and clear Talk through changes to process, etc. Build capacity Integrate DBI meeting process into formal IEP meetings

Pictures of Success!

Pictures of Success!

Lily’s Success!

District Level Considerations to Support Implementation Professional Development: Ongoing Multi-Level Professional Development/Collaboration Administrators (Train-the-Trainer) Teachers and Service providers Even with procedures and protocols , this is labor intensive and requires expertise! Identifying Funding Sources for: Professional Development Meeting schedule Build District-University Partnerships Administrators are in need of PD to understand the process. Cannot expect the

Training Teachers to Support Capacity Pre-service training generally does not focus on implementing meaningful individualization Teachers need more training in specific, DI in reading and mathematics Teachers need more professional development about the “Truths/Myths” of progress monitoring Teachers need permission to progress monitor at a child’s instructional level

Intensive Intervention Pilot Partnership

Intensive Intervention Pilot 27 teachers participating over a 2 year course of study Equivalent to 4 graduate courses, 1 course coaching Intensive Intervention, Reading, Mathematics, Behavior and Coaching State partnership for Intensive Intervention Credential University partnership UCONN, BU and University of TX at Austin

Benefits of Intensive Intervention Student Progress (Tier 3, IEP and intensive) Define specialized instruction Clear plan for instruction Improved IEP goals All students with intensive needs have access to core curriculum as well as intensive interventions Inform and improve RTI, MTSS district wide Inform and improve core curriculum decisions district wide Common language and understanding across the district regarding intervention at all tiers

RTI Implementation Is Rocket Science Douglas Fuchs, Vanderbilt University

In Summary ….. Leadership is essential, though who the leaders are can vary Understand that implementation is challenging work and barriers will arise The use of data requires practice. Sites often spend a lot of time addressing how to collect and analyze data Sites with protocols and processes in place helped staff focus on intervention

Reflections on Lessons Learned Can all schools implement 3 tiers of RTI (and special education)? Can a majority of schools do so? If some number of schools can’t do so, doesn’t it make sense to encourage them to implement 2 tiers? Shouldn’t we have a Plan B for such schools?

National Evaluation of RTI in 146 “Impact Schools” Funded by IES as an “effectiveness” study, MDRC conducted the first (and only) RTI national evaluation in 2011-2012. This study did not address the effectiveness of RTI. Instead, it asked the question: Does the use of class-wide screening with a benchmark for designating students for Tiers 2 and 3 intervention increase their reading performance at year’s end? Implementation data collected from the Impact Schools revealed considerable problems.

3 Major Implementation Problems in the Impact Schools Cut-point at the 41st percentile, which exceeds the validated benchmarks of most commercial screens. Nearly 50% of Impact Schools provided intervention to students above this cut point. This is problematic because it “thins” out the resources. Only 62% of Impact Schools claimed at least one student at Tiers 1, 2, and 3. That is, many schools did not assign a single student to one of their tiers. Takeaway: Many Impact Schools didn’t provide intensive intervention to all students who needed it.

Milwaukee RTI Evaluation Funded by IES as a fidelity study and conducted in 2014- 2015. Methods: REL Midwest constructed a measure to evaluate implementation of 6 RTI components, including “leadership” and “collaboration,” on a 5-point scale that yielded “little,” “inadequate,” “adequate,” and “full” fidelity. Fidelity data were collected in 68, K-5 schools. Findings: Following 2 years of implementation, 47 of the 68 schools (69%) were still inadequately implementing the multi-tiered instruction.

Heartland Iowa For nearly 15 years, evaluations conducted by Heartland (IA) staff of RTI effectiveness have indicated no effects (e.g., Ikeda et al., 2005) or equivocal effects (Till, 2003) of its “problem-solving” RTI model. Why? For the same reason many practitioners in the National and Milwaukee evaluations had difficulty implementing RTI: For many teachers and schools, it is too complex given all that they already have to do.

Simpler RTI Frameworks It is important to say that many educators are implementing 3-tier RTI models with adequate or better fidelity. But it is also true that many are not, despite best intentions and hard work. We must accept that some number of schools will not implement 3-tier models with fidelity. There are two reasonable responses to this fact: conduct more efficient 3-tier models or adopt 2-tier models.

3-Tier Efficiency Successful RTI schools do 2 things well. First, they have a strong Tier 1 (i.e., class-wide, supplemental programs and behavior management systems), which reduces non-responsiveness from 20-25% to 10-15% and reduces the number of students requiring Tier 2. Second, they use a gated screening process to reduce false positives for Tier 2 instruction. Fewer students in Tier 2 permits smaller and more effective instructional groups.

2-Tier Model Eliminating Tier 2 can help schools focus energies on Tier 1 and special education. By definition, an instructionally strong Tier 1 helps most children achieve and is also the right environment to find “true positives”—“dually-discrepant” students, performing 2 SDs below the mean level and growth rate of peers. Special education instruction is DBI. 3 reasons why special education must be part of RTI frameworks. (a) Special and general educators must work together. No separate silos that undermine communication and cooperation. (b) DBI is a potentially important and effective complement to Tier 1 in the absence of Tier 2. (c) Special and general educators should be held equally accountable.

Audience Questions

Connect to NCII Sign up on our website intensiveintervention.org to receive our newsletter and announcements Follow us on YouTube and Twitter YouTube Channel: National Center on Intensive Intervention Twitter handle: @TheNCII

Disclaimer This presentation was produced under the U.S. Department of Education, Office of Special Education Programs, Award No. H326Q110005. Celia Rosenquist serves as the project officer. The views expressed herein do not necessarily represent the positions or policies of the U.S. Department of Education. No official endorsement by the U.S. Department of Education of any product, commodity, service, or enterprise mentioned in this webinar is intended or should be inferred. 62

Contact US National Center on Intensive Intervention 1000 Thomas Jefferson Street NW Washington, DC 20007-3835 www.intensiveintervention.org ncii@air.org @TheNCII