Strand A: How Can We Make Intensive Intervention Happen? Considerations for Knowledge Development, Implementation, and Policy Session 3: From Know-How to Action: Assessing and Improving School-Level Implementation of Data-Based Individualization Laura Berry Kuchle, Christopher Lemons, T. Chris Riley-Tillman, Louis Danielson This document was produced under U.S. Department of Education, Office of Special Education Programs, Award No. H326Q Celia Rosenquist serves as the project officer. The views expressed herein do not necessarily represent the positions or policies of the U.S. Department of Education. No official endorsement by the U.S. Department of Education of any product, commodity, service or enterprise mentioned in this document is intended or should be inferred.

 Introduction to DBI  Why Do We Need to Assess DBI Implementation?  Developing a Rubric  DBI Implementation Rubric  Use in NCII Intensive Technical Assistance Sites  Lessons Learned: Strategies for Improving Technical Assistance and DBI Implementation  NCII Resources to Support Implementation Session Overview

3 What is Intensive Intervention? Intensive intervention addresses severe and persistent learning or behavior difficulties. Intensive intervention should be  Driven by data  Characterized by increased intensity (e.g., smaller group, expanded time) and individualization of academic instruction and/or behavioral supports

4 A Bird’s Eye View of DBI

Why Do We Need to Assess DBI Implementation? 5

 Extent to which an intervention’s core components have been implemented as planned (Nelson et al., 2012)  For DBI, this includes the instructional platform, adapted iterations of intervention, ongoing progress monitoring, and decision-making procedures.  See IRIS module for additional information: What Is Fidelity? 6

 Student level For a given student, are assessment and intervention being carried out as planned? Student-level fidelity may be assessed through teacher logs, observations, or record review  Systems level (school or team) Are essential components of DBI being implemented consistently? Are there systems-level problems that hinder DBI implementation? A systems-level rubric or interview might examine team meeting checklists, logs, or other records Levels of Fidelity 7

 Fidelity assessment provides evidence that DBI is being implemented as intended Assessment, intervention, and decision-making processes Student and systems levels  Provides guidance on how to improve DBI implementation Does the interventionist need additional training or support? Is an intervention adaptation needed? Is there a systems-level problem? (e.g., scheduling prevents sufficient intervention time, staff do not have access to evidence-based instructional platforms) Why Does Fidelity Matter? 8

Developing a Rubric 9

 Based on the structure of the Center on Response to Intervention’s RTI Essential Components Integrity Rubric rti-integrity-rubric-and-worksheet rti-integrity-rubric-and-worksheet  Identify structures, resources, and practices needed for quality school-level implementation of DBI  Reflect lessons learned from NCII knowledge development sites Developing a Systems-Level Rubric 10

 Purpose: to learn about strategies for implementing intensive intervention from sites that have demonstrated positive outcomes for students with disabilities (SWDs)  Implementing Intensive Intervention: Lessons Learned From the Field (NCII, 2013) -intensive-intervention-lessons-learned-field -intensive-intervention-lessons-learned-field Key Findings From Knowledge Development Sites 11

1.Intensive intervention is most likely to be facilitated when implemented as a component of a multi-tiered system of support (MTSS) 2.Family engagement supports implementation 3.Intensive behavioral intervention is often more challenging because of limited progress monitoring 4.Inconsistent decision rules for intensifying supports can hinder intervention planning and resource allocation 5.Hidden inefficiencies may overtax staff unnecessarily Lessons Learned 12

DBI Implementation Rubric 13

 NCII’s DBI Implementation Rubric and Interview: implementation-rubric-and-interview implementation-rubric-and-interview  Aligned with the essential components of DBI and the infrastructure that is necessary for successful implementation in Grades K–6 Measuring School-Level Implementation 14

1.System Features 2.Data and Decision Making 3.Intervention 4.DBI Process 5.DBI Evaluation Components/Infrastructure Required for DBI Implementation 15

a)Leadership b)School Schedules c)Resources d)Cultural and Linguistic Responsiveness e)Communication With and Involvement of Parents f)Communication With and Involvement of All Staff 1. System Features 16

a)Process to Identify Students in Need of Intensive Intervention b)Academic Progress-Monitoring Tools c)Behavior Progress-Monitoring Tools d)Progress-Monitoring Procedures e)Diagnostic Assessment f)Fidelity of Assessment Implementation 2. Data and Decision Making 17

a)Intensive Intervention b)Adaptation of Intervention c)Fidelity of Implementation of Intervention d)Interventionist Characteristics e)Relationship to Core Instruction 3. Intervention 18

a)Team or Appropriate Support Mechanism b)Regular Meetings c)Data Review d)Documentation e)Decision Rules f)Overall Implementation of DBI Process 4. DBI Process 19

a)Evaluation 5. DBI Evaluation 20

21 Rating Scale 1–5 points with anchors for  1 point = little or no implementation  3 points = partial or inconsistent implementation  5 points = complete and consistent implementation

22 DBI Implementation Interview  Script and note-taking template for gathering information to evaluate rubric  Sample questions for each rubric item

Use in NCII Intensive Technical Assistance Sites 23

 Spring and fall 2014: 17 schools in 4 states Ongoing NCII training and coaching for approximately 1–2 school years  Two facilitators led interview NCII staff and consultants, including NCII school coach  Co-rated for reliability checks DBI Implementation “Pulse Checks” 24

When two facilitators scored the same item for the same school:  Mean/average difference was 0.51 points on the 5-point scale  Mean difference across items ranged from 0.12 to 0.94 points How Much Did Raters’ Scores Vary? 25

26 How Often Did Both Raters Give the Same Score? Within One Point? Inter-rater Agreement on Item Ratings Across All ItemsRange Perfect Agreement 58%29%–88% Agreement Within 1 Point 94%76%–100%

 Guideline: Refine rubric anchors and/or interview questions for items with <80% agreement within one point  Item 5a, Evaluation 76% agreement within one point Mean point difference = 0.82  Item 4a, Team or Appropriate Support Mechanism 82% agreement within one point Mean point difference = 0.94 Items With Low Agreement 27

 Mean score across all items and schools was 2.99 Suggests partial or inconsistent implementation Expected at this point in time  Schools’ mean scores ranged from 2.17 to 3.58 Overall Scores 28

29 Items With Highest Scores ItemMean 3d. Interventionist Characteristics3.88 4a. Team or Appropriate Support Mechanism3.82 4b. Regular Meetings3.82 3e. Relationship to Core Instruction3.65 1a. Leadership3.53 2b. Academic Progress-Monitoring Tools3.47

 Trained, experienced interventionists  Teams to support decision making for DBI, with a regular meeting schedule  Aligning intervention to student need and core expectations, addressing prerequisite skills as appropriate  District and school leadership support for DBI  Technically rigorous academic progress-monitoring tools that are sensitive to student improvement These Scores Suggest Relative Strengths in the Areas of… 30

31 Items With Lowest Scores ItemMean 5a. Evaluation2.12 2c. Behavior Progress-Monitoring Tools2.18 2f. Fidelity of Assessment Implementation2.24 3c. Fidelity of Implementation of Intervention2.29 4e. Decision Rules2.35 4f. Overall Implementation of DBI Process2.35

Evaluation  Schools may informally review implementation without setting specific goals  Evaluation may be difficult or a low priority if other pieces are not clearly articulated and in place Behavior progress monitoring  Fewer validated tools available compared to academics  New to many schools Why Might These Items Have Relatively Low Scores? 32

Fidelity  Few schools systematically monitor Decision rules  May not be clearly articulated in one or more areas  Application may not be consistent Overall implementation  May be inconsistent or poorly documented Considering Low Scores, Continued… 33

Strategies for Improving Technical Assistance and DBI Implementation 34 Lessons Learned

 Cross-state coach meeting in June 2014  Discussed DBI Interviews and coaching activities throughout the year to identify essential elements of DBI implementation—implications for expanding to new schools  Incorporated interview findings into 2014–15 TA plans; now more individualized at school level  Created Coaches’ Corner website and Professional Learning Community to share resources throughout the year Learning From Initial DBI Interviews 35

 Staff commitment  Student plans  Student meetings  Valid, reliable data  Inclusion of students with disabilities Essential Elements of DBI Implementation 36

Staff Commitment Key Element Flexibility Within Implementation Commitment of:  Principal  Intervention staff  Special educators  Specific intervention staff involved including staff who work with students with intensive needs in the area(s) of concern. (e.g., reading specialists, social workers)

38 Student Plans Key Element Flexibility Within Implementation Student plans are developed and reflect:  Accurate and timely student data  Goal(s) for the intervention based on valid, reliable assessment tools  Timeline for executing and revisiting the intervention plan  Content area(s)  Number of student plans  Grade level(s)

39 Student Meetings Key Element Flexibility Within Implementation  Student meetings are data driven  There is a regularly scheduled time to meet  Meetings are structured to maximize efficiency and focused problem solving  Frequency  Schedule  Team members

40 Progress Monitoring Key Element Flexibility Within Implementation  Valid, reliable progress monitoring tools are used.  Data are graphed.  Data are collected at regular intervals.  Choice of tool  Use of progress- monitoring data at other tiers

41 Students With Disabilities Key Element Flexibility Within Implementation  Students with disabilities must have access to intensive intervention.  Who delivers intervention for students with disabilities  Inclusion of students with and without IEPs

NCII Resources to Support Implementation 42

 Systems level considerations—infrastructure and staff commitment Module to assess and support readiness to implement DBI  Identifying intervention and assessment tools for mathematics and behavior Tools charts Sample lessons and activities  Consistent procedures and documentation Student meeting tools Addressing Common Barriers to Implementation 43

 Module introducing schools to DBI and infrastructure needed for implementation: ready-implement-intensive-intervention-infrastructure-data- based-individualization ready-implement-intensive-intervention-infrastructure-data- based-individualization  Handouts and activities to assess readiness and begin action planning Getting Ready to Implement Intensive Intervention: Infrastructure for DBI 44

Tools Charts Academic Progress Monitoring t/progress-monitoring t/progress-monitoring Academic Intervention t/instructional-intervention-tools t/instructional-intervention-tools Behavioral Progress Monitoring t/behavioral-progress-monitoring-tools t/behavioral-progress-monitoring-tools Behavioral Intervention t/behavioral-intervention-chart t/behavioral-intervention-chart

46 Mathematics: Sample Lessons & Activities activities/mathematics

47 Implementing Behavioral Strategies: Considerations and Sample Resources considerations-and-sample-resources

48 Individual Student Meeting Tools data-meetings

 NCII website  Examples of Standards-Aligned Instruction Across Tiers  DBI Training Series  Webinars  Connect to NCII Learn More About DBI 49

50 Find Out What NCII Has to Offer

51 Examples of Standards-Aligned Instruction Across Tiers across-levels-tiered-system

52 DBI Training Series  Eight training modules focusing on components of DBI for academics and behavior  One module focused on readiness and action planning  Include Slides and speaker notes Activities Coaching guides

53 Webinars View archived webinars and look for announcements about the next live webinar:

54 Connect to NCII  Sign up on our website to receive our newsletter and announcements  Follow us on YouTube and Twitter YouTube Channel: National Center on Intensive Intervention National Center on Intensive Intervention Twitter handle:

National Center on Intensive Intervention. (2013). Implementing intensive intervention: Lessons learned from the field. Washington, DC: U.S. Department of Education, Office of Special Education Programs. m_Field_0.pdf m_Field_0.pdf Nelson, M. C., Cordray, D. S., Hulleman, C. S., & Sommer, E. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. The Journal of Behavioral Health Services & Research, 39(4), 374–396. References 55

National Center on Intensive Intervention 1000 Thomas Jefferson Street NW Washington, DC