Improving MTSS Implementation and Outcomes through Problem Solving

Session Outcomes Identify areas of concerns and develop a plan of action to improve system level implementation.

Intervention Levels/Tiers Primary (T1) Secondary (T2) Intensive (T3) Instruction/ Intervention Approach Comprehensive research-based curriculum Standardized, targeted small-group instruction Individualized, based on student data Group Size Class-wide (with some small group instruction) 3–7 students No more than 3 students Monitor Progress 1x per term At least 1x per month Weekly Population Served All students At-risk students Significant and persistent learning needs Review table. Highlight the differences between the tiers in terms of: Increasingly focused/tailored approach Decreasing group size Increasing frequency of progress monitoring Increasing student need

Critical Features of Tier I Instruction Section objectives: Explain the critical features of a high-quality Tier I system Delivery time: 40 min with activities

Critical Features of Tier I Instruction Uses research-based curriculum materials Articulation of teaching and learning (in and across grade levels) Consistent use of differentiated instruction Instruction aligned with state standards Inclusion of students with disabilities and those exceeding benchmark Refer participants to Handout 4.2. This is an animated slide. Alignment of behavior and academics is not described on the WY MTSS Fidelity of Implementation Rubric, but it is a critical feature of Tier 1 Instruction that will be highlighted in this module. As we talk about Tier I, we are referring to the alignment of core instruction and expectations for both behavior and academics. Each of these features is described in detail in Handout 4.2, which is an excerpt from the Wyoming MTSS Fidelity of Implementation Rubric. Briefly describe each component by reviewing the contents of the last column in the handout. Handout

Use Research-Based Curriculum Materials Across all subjects and school-wide behavior supports Components have been researched and found to be generally effective Curriculum materials have not been rigorously evaluated as a package NCII Encourage participants to follow along using Handout 4.2. This slide has animation. Tier I requires research-based curricula across subjects and in the design of the school-wide behavior supports. Research-based curricula have been researched and found to be generally effective, although the curriculum materials have not been rigorously evaluated as a package. If you’re wondering about resources available for determining the extent of the research-base for curriculum materials available to teachers, there are online resources available, or you can contact the vendor or publisher of the curricula available to you in your school or classroom. Click for animation. The What Works Clearinghouse is consider the “Gold Standard”. It review of studies conducted on curricula and instructional practices and provides Intervention Reports and Practice Guides, http://ies.ed.gov/ncee/wwc. Click for animation. The Best Evidence Encyclopedia also provide research reviews of commonly used curriculum material, http://www.bestevidence.org/. Click for animation. Some curricula may not have been reviewed by external resources such as these. In these cases, it is recommended to contact the vendor/publisher of curricula to inquire about available research reports.

Ensure Articulation of Teaching and Learning (In and Across Grade Levels) Teaching and learning objectives are well articulated from one grade to another Teaching and learning are well articulated within grade levels so that students have highly similar experiences, regardless of their assigned teacher Review slide. How can you determine the effectiveness of strategies (e.g., curriculum mapping, pacing guides) to ensure articulation of teaching and learning? We will take a look at some screening data together on the next slide.

Consistent Use of Differentiated Instruction Teachers in the school differentiate instruction for students on, below, or above grade level Most teachers in the school use student data to identify and address the needs of students Review participants to the final column for differentiated instruction in Handout 4.2. Review criteria. How do we know if we are consistently using differentiated instruction? The best way to assess this is to interview and observe staff. Lack of differentiated instruction is often attributed to inconsistent understanding of what is differentiated instruction and how do it look in practice.

Aligned with State Standards The core curriculum (reading and mathematics) is aligned with the Common Core or other state standards. https://edu.wyoming.go v/in-the- classroom/wyoming- standards/ Refer participants to the descriptor found in the final column for aligned with state standards in Handout 4.2. Note that there are guidance, resources, lesson plans, student writing samples, etc. on the Common Core State Standards site as well as the Wyoming standards page (screenshot here). In June 2012, Wyoming adopted the Common Core State Standards as its standards in English/Language Arts and Math. No amendments were made to the original Common Core State Standards. Wyoming standards: https://edu.wyoming.gov/in-the-classroom/wyoming-standards/ CCSS: http://www.corestandards.org/

Inclusion of Students Exceeding Benchmark Schools provide enrichment opportunities for students exceeding benchmarks Educators implement those opportunities consistently in all classes and grade levels Refer participants to the final row of Handout 4.2. Review the description in the last column.

Inclusion of Struggling Students and Students with Disabilities Schools provide differentiation of instruction, accommodations, and modifications to ensure struggling students, including those with disabilities, benefit from core instruction These supports are consistently provided across all classes and grade levels Struggling students, including students with disabilities, must be provided appropriate supports to ensure they have access to core instruction. Review slide.

Activity: Self-Evaluation of Tier I System Ask participants to locate Handout 6.1. and review the directions on Handout 6.1. Clipart: Creative Commons license; http://www.pezdisco.es/fido-normas/parametros-de-votacion Handout

Critical Features of Tier II System Section objective: Explain the critical features of a braided academic and behavior Tier II system Key ideas: Feature 1: Uses evidence-based intervention that support academic and behavior needs. Feature 2: Complements core academic and behavior instruction/program. Feature 3: Uses standardized interventions with appropriate dosage and grouping size delivered by trained personnel and implemented with fidelity. Feature 4: Scheduled in addition to Tier I support The HLPs introduced in Module 4: Tier I are intensified as the foundations for more targeted instruction and interventions Delivery time: 20 – 30 minutes (Slides 7 – 11)

Four Critical Features of Tier II Uses evidence-based interventions that support academic and behavior needs Complements core academic and behavior instruction/program In previous modules, you explored the WY MTSS implementation rubric. Item 4b describes four critical features of secondary-level, or Tier II. Let’s discuss those features now. The WY MTSS implementation rubric is excerpted over the next few slides. The full rubric is available for download here: http://edu.wyoming.gov/downloads/special-programs/symposium/2015/Bailey%20WY%20MTSS%20Fidelity%20of%20Implementation%20Rubric.pdf All Tier II interventions are evidence-based in content areas and grade levels where they are available and they support academic and behavior needs. Tools for identifying evidence-based interventions will be discussed later in the presentation. Tier II complements core instruction; specifically, it is well aligned with core instruction and incorporates foundational skills that support the learning objectives of core academic instruction and supports behavior programs. Clipart retrieved from: http://anglo-celtic-connections.blogspot.com/2014_08_01_archive.html?_sm_au_=iVVQMSTHvV6qjWS7; Creative Commons license

Four Critical Features of Tier II Uses standardized interventions with appropriate dosage and grouping size delivered by trained personnel with fidelity Standardized intervention uses research-based instructional programs provided in a specific manner to students typically includes a step-by-step sequence (Vaughn et al., 2012) Scheduled in addition to Tier I The third critical feature pertains to the use of instructional characteristics. Specifically, all three of the following conditions are met: (1) interventions are standardized; (2) secondary-level interventions are led by staff trained in the intervention according to developer requirements; and (3) group size and dosage are optimal (according to research) for the age and needs of students. The fourth critical feature is that secondary-level interventions supplement core instruction is scheduled in addition to Tier I. Invite participants to discuss how they approach scheduling Tier II interventions in addition to Tier I Clipart retrieved from: http://anglo-celtic-connections.blogspot.com/2014_08_01_archive.html?_sm_au_=iVVQMSTHvV6qjWS7; Creative Commons license

Activity: Self-Evaluation of Tier II System Ask participants to locate Handout 6.1. and review the directions on Handout 6.1. Clipart: Creative Commons license; http://www.pezdisco.es/fido-normas/parametros-de-votacion Handout

Essential Components of Tier III Section Objective: Explain the critical features of an MTSS Tier III system. Estimated time: 10-15 minutes

Essential Tier III Criteria: Data-based Individualization Tier III or intensive intervention Is more intensive than Tier II interventions. Is adapted to address individual student needs in a number of ways (e.g., increased duration or frequency, change in interventionist, decreased group size, change in instructional delivery, and change in type of intervention). Uses an iterative process based on student data. Read slide.

Essential Tier III Criteria: Instructional Characteristics Tier III or intensive intervention Is individualized. Led by well-trained staff experienced in individualizing instruction based on student data. Uses optimal group size (according to research) for the age and needs of students. Read slide.

Essential Tier III Criteria: Relationship to Core Decisions regarding student participation in both core instruction and intensive intervention are made on a case-by-case basis, according to student need. Intensive interventions address the general education curriculum in an appropriate manner for students. Read slide. Resources regarding alignment of standards-relevant instruction and multi-tiered systems of support (MTSS): http://www.intensiveintervention.org/standards-relevant-instruction-multi-tiered-systems-support-mtss-or-response-intervention

What’s the difference between Tier 2 and Tier 3? Tier 3 or DBI This graphic provides a closer look at the DBI process, and notes the places in which secondary interventions fit into the process. Before a team decides that a student requires intensive intervention in a certain content area, the team should consider whether or not the student has received a secondary intervention program with consistency and fidelity. The DBI process begins when data show that a student is making insufficient progress in response to a secondary intervention program that is evidence based and delivered with fidelity. The first step in this process is to implement the intervention with greater intensity (e.g., smaller group size, more time), and to progress monitor frequently. If the student continues to be non-responsive, diagnostic data are collected and analyzed to identify the specific skill deficits that need to be targeted. The results of the diagnostic assessment, in combination with the teacher’s analysis of what features of the intervention need to be modified to better support the student, help staff determine how to individualize the secondary intervention program to meet the individual student’s unique needs. Upon implementing the intervention adaptation, the teacher continues to progress monitor at regular intervals to help determine whether additional changes to the individualized intervention are required to support adequate student response. As you can see, secondary interventions play a significant role in the DBI process. A student’s responsiveness to secondary intervention determines whether or not intensive intervention is needed. The secondary intervention program is also the platform or starting point for intensive intervention, which is intensified and/or adapted to meet a student’s unique needs. For more information about the DBI process, please review or direct participants to the Introduction to Data-Based Individualization (DBI): Considerations for Implementation in Academics and Behavior module.

Activity: Self-Evaluation of Tier III System Ask participants to locate Handout 7.1 and review the directions For the activity described in Handout 7.1. Clipart: Creative Commons license; http://www.pezdisco.es/fido-normas/parametros-de-votacion Handout

Data Review Considerations Are 80% of students benefiting from core instruction? Are 80% of students responding to Tier 2 instruction? Are students making adequate growth?

Share Out What is the impact of MTSS on student outcomes? What are your top 2-3 areas of improvement?

Understanding Evidence-Base Instruction and Intervention

Intervention and Instruction Evidence-Based Intervention Fidelity Adherence Student Engagement Program Specificity Quality of Delivery Exposure Tier II interventions should be evidence-based and conducted with fidelity. Click for animation. First, let’s take a look at what it means for a secondary intervention to be evidence-based.

EBPs: What Are They? In simple terms, EBPs are practices that are supported by research But what is considered a practice? And how do I know if it is supported by research?

Challenge: Inconsistent Terminology Authors and organizations refer to EBPs in a variety of ways: Evidence-based practice Research-based practice Scientifically-based practice Promising practice Best practice Possible poll/comment request: ask people to throw out other terms they have heard

Varying Evidence Standards Research-Based Curricula Evidence-Based Intervention Recommended for Tier I across subjects Components have been researched and found to be generally effective Curriculum materials have not been rigorously evaluated as a package Recommended for Tier II and Tier III Materials evaluated using rigorous research design Evidence of positive effects for students who received the intervention There are varying levels of evidence for instruction and interventions delivered within an MTSS framework. On the left, you’ll see research-based curricula. This is recommended for Tier I, or core instruction, across subjects. This means that the components have been researched and found to be generally effective, although the curriculum materials have not been rigorously evaluated as a package. On the right side of your screen you’ll see evidence-based intervention. Evidence-based interventions are recommended for Tier II and Tier III. These materials are evaluated using rigorous research design, and there is evidence of positive effects for students who received the intervention. This means that programs have been rigorously evaluated as a whole, and that the specific intervention program was found to have positive effects for students receiving the intervention program.

Challenge: Inconsistent Criteria Authors and organizations define criteria for EBPs in different ways: Quality of evidence (study design) Direction and magnitude of results Quantity of evidence Source of evidence Some combination of all these things

Every Student Succeeds Act Requires use of “evidence-based interventions” in low performing schools Four categories of evidence: “strong evidence”: supported by at least one randomized study “moderate evidence”: supported by at least one quasi-experimental study “promising evidence”: at least one correlational study with pretests as covariates programs with a rationale based on high-quality research or a positive evaluation that are likely to improve student or other relevant outcomes Must have a statistically significant positive effect on relevant outcomes  

Limitations of Gold Standard Evidence… Time and fiscal resources that delay movement from the research lab into schools may mean few options in certain areas. Generalizability to other populations Why isn’t this working? Misalignment of “practice” with research design Generally effective isn’t universally effective… Some kids need more.

Is MTSS/RTI an EBP? In MTSS/RTI, the “practice” in “evidence-based practice” is different in each tier! Wouldn’t this mean that criteria for evidence should differ too?

Varying Evidence Standards (Examples) This graphic illustrates an example of what a school’s reading instruction may look like at each tier. As you’ll see, the evidence standards for the instruction vary by tier. At the primary level, of Tier I, reading instruction consists of research-based curricula. This means that the core curriculums are not evidence-based as a whole, but typically include research-based strategies and practices such as explicit instruction or partner reading. At the Tier II or secondary level, interventions should be evidence-based interventions, researched as a whole for their effect on student learning outcomes. Finally, at the Tier III or intensive level, schools should use adapted evidence-based interventions. In other words, Tier III depends on the adaptation of Tier II evidence-based interventions. Adaptations are be based on data to meet the unique needs of that student.

What to Look For When Examining the Published Evidence Base Type/Source Population Desired Outcomes Effects In order to ensure that a Tier II intervention program is evidence-based, teams should convene to review the existing research on that particular program. You can focus your efforts by looking at these areas when examining the evidence base: First, consider the type of information and source from which you are gathering this information. Is the information coming from the intervention vendor or a reputable website? Also ask yourself, what type of evidence is available? Did the study involve experimental design, where the intervention group was compared to an equivalent control group? Next, consider the population. For which populations has the program been researched and found effective? Is the sample described? Is the population similar to or representative of your student population? Are there different effects for different population groups? It is important to consider whether or not the desired outcomes assessed in a study are relevant to the outcomes you hope to achieve with an intervention. Finally, consider the effects. Were the effects of the study large enough to be meaningful? Consider established guidelines for effect sizes when reviewing evidence on effectiveness. Sites including NCII’s Interventions Tools Chart, the What Works Clearinghouse, and the Best Evidence Encyclopedia all offer guidance when interpreting effect sizes.

Resources for Evaluating Evidence Base of Published Tier II Interventions NCII Interventions Tools Chart http://www.intensiveintervention.org/chart/instructional-intervention-tools What Works Clearinghouse http://ies.ed.gov/ncee/wwc/findwhatworks.aspx Best Evidence Encyclopedia http://www.besteviden ce.org/ Model how to navigate at least the first two resources. If you are unable to model the site during the presentation, review the content of the resources on the next couple of slides. Participants may also wish to locate Handout 6.2, which outlines the features of the resources modeled on this slide. After modeling the resources, ask participants to share out at their tables, with a partner, or as a whole-group using prompts such as the following: Have you accessed any of these resources to identify of evaluate the Tier II interventions used at your site? If you haven’t accessed any of these resources, identify one that you will access and describe how you might use it. Handout

What If Evidence-Based Interventions Aren’t Available? Use them when available and consider augmenting current offerings, if feasible. Also consider: Remediation materials that came with your core program materials Expert recommendations (if evidence-based programs are not available) Standards-aligned materials In some content areas and grade levels, published evidence-based intervention programs may not be available. Schools should use evidence-based intervention programs when available, and consider augmenting current offerings if feasible. If evidence-based interventions aren’t available, consider using remediation materials that came with your core program materials, expert recommendations, or standards-aligned materials.

What Works Clearing House Scavenger Hunt Identifying and evaluating evidence base for Tier II interventions The purpose of this OPTIONAL activity is to provide individuals or teams an opportunity to explore a commonly used resource for identifying and evaluating the evidence base of Tier II interventions. Website link: https://ies.ed.gov/ncee/WWC/ Estimated time: 20-30 minutes Grouping: Partners or school teams. This activity is not recommended for individuals to complete during the session. If not completed during the session, encourage participants to complete the activity or utilize the tools to evaluate current interventions. A Find What Works Video Tour is available on the WWC website here: https://ies.ed.gov/ncee/wwc/Multimedia/27. The link is included on Handout 6.4. Clip art: Creative Commons Handout

Elements of Tier II Interventions Evidence-Based Intervention Fidelity Adherence Student Engagement Program Specificity Quality of Delivery Exposure Now let’s discuss what it means to deliver Tier II interventions with fidelity.

What Is Fidelity? Degree to which the program is implemented the way intended by program developer. Fidelity = Consistency and Accuracy Fidelity = Integrity Fidelity refers to how closely prescribed procedures are followed, and in the context of schools, the degree to which teachers implement programs the way they were intended by the program developers. It also relates to the quality of the implementation. This means that teachers are implementing the intervention with consistency and accuracy, and are adhering to the instructional plan with integrity. Note: Throughout discussions of fidelity it is important to ensure that teachers believe that they work in an open, non-threatening environment that values their skills and expertise and where they can learn from their colleagues. With a system of open communication and productive feedback, fidelity checks of classroom techniques and the essential components of multi-tiered systems of support can be a useful and supportive way for teachers to collaborate and become a stronger teaching network. This may be a useful discussion point for some groups.   (Gersten et al., 2005; Mellard & Johnson, 2007; Sanetti & Kratochwill, 2009)

Why Is Fidelity Important? Ensures that instruction has been implemented as intended Allows us to link student outcomes to instruction Helps in the determination of intervention effectiveness and instructional decision-making Positive student outcomes depend on level of fidelity of intervention implementation Why is fidelity important? If teachers aren't consistent and accurate in delivering Tier II interventions, they aren’t able to confidently explain a student's lack of response to an intervention. Did the student make insufficient progress because they require more intensive intervention? Or, did the student make insufficient progress because the secondary intervention wasn’t delivered with fidelity? Without practicing consistency and integrity in intervention delivery, we can't link, or attribute, student outcomes to the instruction provided. Fidelity allows us to evaluate the effectiveness of the secondary intervention and tells us when a student may require a more intensive level of intervention.   Furthermore, Pierangelo and Giuliani (2008) concluded that positive student outcomes are particularly dependent on aspects of fidelity within the framework of a tiered support system. One of these aspects is fidelity of implementation at the classroom or teacher level. (Pierangelo & Giuliani, 2008)

Five Elements of Fidelity Student Engagement: How engaged and involved are the students in this intervention or activity? Adherence: How well do we stick to the plan, curriculum, or assessment? Program Specificity: How well is the intervention defined and different from other interventions? Exposure/Duration: How often does a student receive an intervention? How long does an intervention last? Refer participants to Handout 6.3. Encourage participants to record examples of what this looks like in the third column of the handout. After presenting the slide content, provide participants 3-5 minutes to talk with their neighbors about other examples of these five elements in action. Have them record their responses. This graphic provides one example of a way to think about fidelity, and includes the elements of adherence, exposure, quality of delivery, program specificity, and student engagement. Schools should have procedures in place to monitor the fidelity of their implementation of secondary interventions. While these don’t have to be formal, it is important to consider whether or not they’re implementing programs the way that they are intended to be delivered. In the midst of all of the responsibilities of educators, small checks can make a big difference in keeping services for students on track. Note: The notes on each element on fidelity are animated to pop up with each click. Click ahead each time you discuss a new element of fidelity, and click again to close that element. (Click) When we discuss adherence we are focused on how well we stick to the plan/curriculum/assessment, or implementing the plan/curriculum/assessment as it was intended to be implemented based on research. For a secondary intervention, this may mean how well teachers implement all pieces of an intervention, in the way they were intended to be implemented. This doesn’t necessarily mean that teachers should follow a script word for word, but that covering certain content with appropriate pacing and relevant language and techniques are important. (Click) 2. (Click) Duration/Exposure refers to how often a student receives an intervention and how long an intervention lasts. When thinking about fidelity we are considering whether the exposure/duration being used with a student matches the recommendation by the author/publisher of the curriculum. In the case of secondary interventions, developers and researchers typically specify the required exposure/duration that is needed for the intervention to be effective for most students. If the intervention developer calls for the intervention 3 days a week for 45 minutes each day, is the student receiving this dosage? (Click) 3. (Click) Not only is it important to adhere to the plan/curriculum/assessment, but it is also import to look at the quality of delivery. This refers to how well the intervention, assessment, or instruction is delivered. For example, do you use good teaching practices? Quality instructional delivery also means that teachers are engaged in what they’re teaching, and animated in their delivery, not simply reading from a script. Providing teachers with constructive feedback on their instructional delivery is one way to improve the quality of delivery for secondary interventions. (Click) 4. (Click) Another component is program specificity, or how well the intervention is defined and how different it is from other interventions. Having clearly defined interventions/assessments allows teachers to more easily adhere to the program as defined. Is the intervention a good match for the student’s needs? Or does every low reader get the same intervention? (Click) 5. (Click) Just as quality of delivery is critical, it also is important to also focus on student engagement, or how engaged and involved the students are in the intervention or activity. Following a prescribed program alone is often not enough. Consider whether or not competing behaviors make it difficult for students to take part in the intervention as designed. During the delivery of secondary interventions, teachers may need to use behavior management strategies to manage student behaviors, including providing choice, adding elements of competition, and offering frequent opportunities to respond. (Click) Quality of Delivery: How well is the intervention, assessment, or instruction delivered? Do you deliver instruction using high leverage practices? (Dane & Schneider, 1998; Gresham et al., 1993; O’Donnell, 2008)

Monitoring Fidelity of Implementation – Examples Element Data Source Examples Adherence Self-report, observation checklist Duration/exposure Self-report, observation Quality of delivery Observation, reflection, self-report on techniques used Program specificity Intervention component checklist Student engagement Student progress, student survey Refer participants to the last column of Handout 6.3. Encourage participants to record examples of possible data sources presented on the slide. After presenting the slide content, provide participants 3-5 minutes to identify additional examples of what evidence could be used to assess if these elements of fidelity were addressed in the delivery of the Tier II intervention. Review slide. Sources: Dane & Schneider, 1998; Mellard & Johnson, 2008; O’Donnell, 2008

Discussion To what extent are we using evidence based practices at all levels of instruction? What changes do we need to make to improve students access to evidence based practices?

Next Steps

Communication Plan Purpose Audience Strategies

Professional Development Plan Content Schedule Audience Format