1. Strengthening Intensive Intervention: Assessing and Improving Implementation of Data- Based Individualization Laura Berry Kuchle, Christopher Lemons, T. Chris Riley-Tillman, Louis Danielson This document was produced under U.S. Department of Education, Office of Special Education Programs, Award No. H326Q110005. Celia Rosenquist serves as the project officer. The views expressed herein do not necessarily represent the positions or policies of the U.S. Department of Education. No official endorsement by the U.S. Department of Education of any product, commodity, service or enterprise mentioned in this document is intended or should be inferred.

2. Session Overview  Introduction to Data-Based Individualization (DBI)  Why Do We Need to Assess DBI Implementation?  Developing a Rubric  DBI Implementation Rubric and Interview  Use in NCII Intensive Technical Assistance Sites  Lessons Learned: Strategies for Improving Technical Assistance and DBI Implementation  NCII Resources to Support Implementation

3. 3 National Center on Intensive Intervention (NCII) NCII’s mission is to build district and school capacity to support implementation of data-based individualization in reading, mathematics, and behavior for students with severe and persistent learning and behavioral needs.

4. 4 What is Intensive Intervention? Intensive intervention addresses severe and persistent learning or behavior difficulties. Intensive intervention should be:  Driven by data  Characterized by increased intensity (e.g., smaller group, expanded time) and individualization of academic instruction and/or behavioral supports

5. 5 What Intensive Intervention… Is…  Individualized based on student needs  More intense, often with substantively different content AND pedagogy  Comprised of more frequent and precise progress monitoring Is Not…  A single approach  A manual  A preset program  More of the same Tier 1 instruction  More of the same Tier 2 instruction

6. 6 Why Do We Need Intensive Intervention? Students with disabilities have… Lower academic achievement Higher dropout rates Higher arrest rates For more information: 2015 NAEP (http://www.nationsreportcard.gov/); Sanford et al., 2011; Planty et al., 2008; Aud et al., 2012http://www.nationsreportcard.gov/

7. 7 Why Do We Need Intensive Intervention? More Help Validated programs are not universally effective programs; 3 to 5 percent of students need more help (Fuchs et al., 2008; NCII, 2013). More Practice Students with intensive needs often require 10–30 times more practice than peers to learn new information (Gersten et al., 2008).

8. 8 Who Needs Intensive Intervention?  Students with disabilities who are not making adequate progress in their current instructional program  Students who present with very low academic achievement and/or high-intensity or high-frequency behavior problems (typically those with disabilities)  Students in a tiered intervention system who have not responded to secondary intervention programs delivered with fidelity

9. 9 What is NCII’s Approach to Intensive Intervention? Data-Based Individualization (DBI): A systematic method for using data to determine when and how to provide more intensive intervention:  Origins in data-based program modification/experimental teaching were first developed at the University of Minnesota (Deno & Mirkin, 1977).  It is a process, not a single intervention program or strategy.  It is not a one-time fix, but an ongoing process comprising intervention and assessment adjusted over time.

10. 10 DBI: Integrating data-based decision- making across academics and social behavior See interactive graphic at http://www.intensiveintervention.org/http://www.intensiveintervention.org/

11. 11 Why Do We Need to Assess DBI Implementation?

12. 12 What Is Fidelity?  Extent to which an intervention’s core components have been implemented as planned (Nelson et al., 2012)  For DBI, this includes the instructional platform, adapted iterations of intervention, ongoing progress monitoring, and decision-making procedures.  See IRIS module for additional information: http://iris.peabody.vanderbilt.edu/module/fid/ http://iris.peabody.vanderbilt.edu/module/fid/

13. 13 Levels of Fidelity  Student or interventionist level Are assessment and intervention being carried out as planned? May be assessed through teacher logs, observations, or record review  Systems level (school or team) Are essential components of DBI being implemented consistently? Are there systems-level problems that hinder DBI implementation? A systems-level rubric or interview might examine team meeting checklists, logs, or other records

14. 14 Why Does Fidelity Matter?  Fidelity assessment provides evidence that DBI is being implemented as intended Assessment, intervention, and decision-making processes Student and systems levels  Provides guidance on how to improve DBI implementation Does the interventionist need additional training or support? Is an intervention adaptation needed? Is there a systems-level problem? (e.g., scheduling, access to evidence- based instructional platforms)  Helps determine program effectiveness

15. 15 Developing a Rubric

16. 16 What Does It Take to Implement Intensive Intervention?  Studied knowledge development sites demonstrating positive outcomes for students with disabilities  Informed future technical assistance to partner sites http://www.intensiveintervention.org/resource /implementing-intensive-intervention-lessons- learned-field

17. 17 Lessons Learned What tends to make intensive intervention easier or harder to implement? Facilitators Multi-tiered system of support Family engagement Efficient use of staff Hindrances Inconsistent behavior progress monitoring (fewer tools) Unclear distinction between Tiers 2 and 3

18. 18 Developing a Systems-Level Rubric  Content: structures, resources, and practices needed for quality school-level implementation of DBI  Structure: Based on the Center on Response to Intervention’s RTI Essential Components Integrity Rubric http://www.rti4success.org/resource/essential-components- rti-integrity-rubric-and-worksheet http://www.rti4success.org/resource/essential-components- rti-integrity-rubric-and-worksheet

19. 19 DBI Implementation Rubric and Interview

20. 20 Measuring School-Level Implementation  NCII’s DBI Implementation Rubric and Interview: http://www.intensiveintervention.org/resource/dbi- implementation-rubric-and-interview http://www.intensiveintervention.org/resource/dbi- implementation-rubric-and-interview  Aligned with the essential components of DBI  Reflects the infrastructure needed for successful implementation in Grades K–6 (academics and behavior)

21. 21 Components/Infrastructure Required for DBI Implementation 1.System Features 2.Data and Decision Making 3.Intervention 4.DBI Process 5.DBI Evaluation

22. 22 1. System Features a)District Leadership b)School Leadership c)School Schedules d)Resources e)Cultural and Linguistic Responsiveness f)Communication With and Involvement of Parents g)Communication With and Involvement of All Staff h)Participation of Students With Disabilities

23. 23 2. Data and Decision Making a)Process to Identify Students in Need of Intensive Intervention b)Academic Progress-Monitoring Tools c)Behavior Progress-Monitoring Tools d)Progress-Monitoring Procedures e)Diagnostic Assessment f)Fidelity of Assessment Implementation

24. 24 3. Intervention a)Intensive Intervention b)Adaptation of Intervention c)Fidelity of Implementation of Intervention d)Interventionist Characteristics e)Relationship to Core Instruction

25. 25 4. DBI Process a)Team or Appropriate Support Mechanism b)Regular Meetings c)Data Review d)Documentation e)Decision Rules f)Overall Implementation of DBI Process

26. 26 5. DBI Evaluation a)Evaluation

27. 27 Rating Scale 1–5 points with anchors for  1 point = little or no implementation  3 points = partial or inconsistent implementation  5 points = complete and consistent implementation

28. 28 DBI Implementation Interview  Script and note-taking template for gathering information to evaluate rubric  Sample questions for each rubric item

29. 29 Use in NCII Intensive Technical Assistance Sites

30. 30 DBI Implementation “Pulse Checks”  Two facilitators led interview NCII staff and consultants, including NCII school coach  Co-rated for reliability checks  2014: 17 schools Ongoing NCII training and coaching for approximately 1–2 school years  2015: 16 schools Second interview for 14 schools First interview for 2 schools

31. 31 Revising the Rubric and Interview  Version 1 – 17 schools in 2014 Refined 2 items with lower interrater agreement Further revisions based on user feedback  Version 2 – 13 schools in 2015 Agreement improved for two targeted items Overall, interrater agreement was similar across the first two versions Further revisions based on user feedback  Version 3 – 3 schools in Fall 2015

32. 32 Version 2 Interrater Agreement: How Much Did Raters’ Scores Vary? When two facilitators scored the same item for the same school:  Mean item-level difference ranged from 0.15 to 0.85 points on the 5-point scale  Mean difference across items was 0.48 points

33. 33 How Often Did Both Raters Give the Same Score? Within One Point? Inter-rater Agreement on Item Ratings (Version 2) Across All ItemsRange Perfect Agreement 54%15%–85% Agreement Within 1 Point 97%85%–100%

34. 34 Version 3 – Preliminary Interrater Agreement  Only used by three schools (same two facilitators)  Mean difference between facilitators was 0.40 points  62% perfect agreement  98% agreement within one point  Stronger agreement when rating reading than behavior

35. 35 Scores For schools rated in both 2014 and 2015 20142015 Mean Rating Across All Items and Schools 3.053.31 Range of School Means Across All Items 2.29–3.581.83–4.29 Range of Item-Level Means2.14–4.072.36–4.27

36. 36 Changes in Scores  Varied across schools and items Most improved but a few declined Item mean changes ranged from -0.84 to 1.21 School mean changes ranged from -0.88 to 1.29  Why might some scores drop? Staff turnover – changes in team or leadership Change in area(s) of implementation being rated Better understanding of what strong implementation looks like

37. 37 Items With Highest Scores in 2015 (Returning Schools) ItemMean Academic Progress-Monitoring Tools4.27 Regular Meetings4.00 Relationship to Core Instruction3.81 Leadership3.80 Cultural and Linguistic Responsiveness*3.78 *Item showing greatest increase in mean score from 2014 to 2015

38. 38 These Scores Suggest Relative Strengths in the Areas of…  Rigorous, sensitive academic progress-monitoring tools  Regular DBI team meetings  Intervention aligned to student need and core expectations  District and school leadership support  Culturally and linguistically appropriate assessments and intervention programs

39. 39 Items With Lowest Scores in 2015 (Returning Schools) ItemMean Evaluation2.36 Fidelity of Implementation of Intervention*2.52 Behavior Progress-Monitoring Tools*2.67 Decision Rules2.71 Overall Implementation of DBI Process2.75 *Mean score decreased from 2014 to 2015

40. 40 Why Might These Items Have Relatively Low Scores? Evaluation  Schools may informally review implementation without setting specific goals  Evaluation may be difficult or a low priority if other pieces are not clearly articulated and in place Fidelity  Few schools systematically monitor

41. 41 Considering Low Scores, Continued… Behavior progress monitoring  Fewer validated tools available compared to academics  New to many schools Decision rules  May not be clearly articulated in one or more areas  Application may not be consistent Overall implementation  May be inconsistent or poorly documented

42. 42 Lessons Learned Strategies for Improving Technical Assistance and DBI Implementation

43. 43 Learning From Initial DBI Interviews  Cross-state coach meeting in June 2014 Discussed DBI Interviews and coaching activities throughout the year Identified essential elements of DBI implementation—implications for expanding to new schools  Increased support for coaches Coaches’ Corner website to share resources Professional Learning Community  Incorporated interview findings into district TA plans (now more individualized at school level)

44. 44 Essential Elements of DBI Implementation  Staff commitment  Student plans  Student meetings  Valid, reliable data  Inclusion of students with disabilities

45. Staff Commitment Key Element Flexibility Within Implementation Commitment of:  Principal  Intervention staff  Special educators  Specific intervention staff involved including staff who work with students with intensive needs in the area(s) of concern. (e.g., reading specialists, social workers)

46. 46 Student Plans Key Element Flexibility Within Implementation Student plans are developed and reflect:  Accurate and timely student data  Goal(s) for the intervention based on valid, reliable assessment tools  Timeline for executing and revisiting the intervention plan  Content area(s)  Number of student plans  Grade level(s)

47. 47 Student Meetings Key Element Flexibility Within Implementation  Student meetings are data driven  There is a regularly scheduled time to meet  Meetings are structured to maximize efficiency and focused problem solving  Frequency  Schedule  Team members

48. 48 Progress Monitoring Key Element Flexibility Within Implementation  Valid, reliable progress monitoring tools are used.  Data are graphed.  Data are collected at regular intervals.  Choice of tool  Use of progress- monitoring data at other tiers

49. 49 Students With Disabilities Key Element Flexibility Within Implementation  Students with disabilities must have access to intensive intervention.  Who delivers intervention for students with disabilities  Inclusion of students with and without IEPs

50. 50 NCII Resources to Support Implementation

51. 51 Addressing Common Barriers to Implementation  Systems level considerations—infrastructure and staff commitment Module to assess and support readiness to implement DBI  Identifying intervention and assessment tools for mathematics and behavior Tools charts Sample lessons and activities  Consistent procedures and documentation Student meeting tools DBI implementation logs and checklists

52. 52 Getting Ready to Implement Intensive Intervention: Infrastructure for DBI  Module introducing schools to DBI and infrastructure needed for implementation: http://www.intensiveintervention.org/resource/getting- ready-implement-intensive-intervention-infrastructure-data- based-individualization http://www.intensiveintervention.org/resource/getting- ready-implement-intensive-intervention-infrastructure-data- based-individualization  Handouts and activities to assess readiness and begin action planning

53. Tools Charts Academic Progress Monitoring http://www.intensiveintervention.org/char t/progress-monitoring http://www.intensiveintervention.org/char t/progress-monitoring Academic Intervention http://www.intensiveintervention.org/char t/instructional-intervention-tools http://www.intensiveintervention.org/char t/instructional-intervention-tools Behavioral Progress Monitoring http://www.intensiveintervention.org/char t/behavioral-progress-monitoring-tools http://www.intensiveintervention.org/char t/behavioral-progress-monitoring-tools Behavioral Intervention http://www.intensiveintervention.org/char t/behavioral-intervention-chart http://www.intensiveintervention.org/char t/behavioral-intervention-chart

54. 54 Mathematics: Sample Lessons & Activities http://www.intensiveintervention.org/resources/sample-lessons- activities/mathematics

55. 55 Implementing Behavioral Strategies: Considerations and Sample Resources http://www.intensiveintervention.org/implementing-behavioral-strategies- considerations-and-sample-resources

56. 56 Individual Student Meeting Tools http://www.intensiveintervention.org/tools-support-intensive-intervention- data-meetings

57. 57 Student-Level DBI Implementation Checklists http://www.intensiveintervention.org/resource/student-level-data-based- individualization-implementation-checklists

58. 58 DBI Implementation Log: Daily and Weekly Intervention Review http://www.intensiveintervention.org/resource/data-based-individualization- implementation-log-daily-and-weekly-intervention-review

59. 59 Intensive Intervention Implementation Review Log http://www.intensiveintervention.org/resource/intensive-intervention- implementation-review-log

60. 60 Learn More About DBI  NCII website  Examples of Standards-Aligned Instruction Across Tiers  DBI Training Series  Webinars  Connect to NCII

61. 61 Find Out What NCII Has to Offer www.intensiveintervention.org

62. 62 Examples of Standards-Aligned Instruction Across Tiers http://www.intensiveintervention.org/illustration-standards-relevant-instruction- across-levels-tiered-system

63. 63 DBI Training Series  Eight training modules focusing on components of DBI for academics and behavior  One module focused on readiness and action planning  Include Slides and speaker notes Activities Coaching guides http://www.intensiveintervention.org/content/dbi-training-series

64. 64 Webinars View archived webinars and look for announcements about the next live webinar: www.intensiveintervention.org

65. 65 Connect to NCII  Sign up on our website to receive our newsletter and announcements  Follow us on YouTube and Twitter YouTube Channel: National Center on Intensive Intervention National Center on Intensive Intervention Twitter handle: @TheNCII@TheNCII

66. 66 References Aud, S., Hussar, W., Johnson, F., Kena, G., Roth, E., Manning, E., Wang, X., & Zhang, J. (2012). The condition of education 2012 (NCES 2012-045). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubs2012/2012045.pdf http://nces.ed.gov/pubs2012/2012045.pdf National Center on Intensive Intervention. (2013). Implementing intensive intervention: Lessons learned from the field. Washington, DC: U.S. Department of Education, Office of Special Education Programs. http://www.intensiveintervention.org/sites/default/files/Lessons_Learned_From_Field_0.pdf http://www.intensiveintervention.org/sites/default/files/Lessons_Learned_From_Field_0.pdf Nelson, M. C., Cordray, D. S., Hulleman, C. S., & Sommer, E. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. The Journal of Behavioral Health Services & Research, 39(4), 374–396. Planty, M., Hussar, W., Snyder, T., Provasnik, S., Kena, G., Dinkes, R., KewalRamani, A., & Kemp, J. (2008). The condition of education 2008 (NCES 2008-031). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubs2008/2008031.pdf http://nces.ed.gov/pubs2008/2008031.pdf Sanford, C., Newman, L., Wagner, M., Cameto, R., Knokey, A.-M., & Shaver, D. (2011). The post-high school outcomes of young adults with disabilities up to 6 years after high school. Key findings from the National Longitudinal Transition Study-2 (NLTS2) (NCSER 2011-3004). Menlo Park, CA: SRI International. Retrieved from http://www.ies.ed.gov/ncser/pubs/20113004/pdf/20113004.pdfhttp://www.ies.ed.gov/ncser/pubs/20113004/pdf/20113004.pdf

67. National Center on Intensive Intervention 1000 Thomas Jefferson Street NW Washington, DC 20007-3835 866-577-5787 www.intensiveintervention.org ncii@air.org www.intensiveintervention.org ncii@air.org @TheNCII