Making Progress Monitoring Work at Tier 2 February, 2014 Facilitated/Presented by: The Illinois RtI Network is a State Personnel Development Grant (SPDG) project of the Illinois State Board of Education. All funding (100%) is from federal sources. The contents of this presentation were developed under a grant from the U.S. Department of Education, #H325A However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. (OSEP Project Officer: Grace Zamora Durán) I-RtI Network Insert name(s) here

1.NAME: 2. DISTRICT NAME: 3. Progress Monitoring Tools I want to know more about: (Level and Content, e.g. Middle School Math) 4. PM Tools I could share with others about: (Level and Content, e.g. Elementary Reading) 5. Other questions or comments:

What’s happening in your district? questions do you have? What coaching Check-in Making connections Applying what we’ve learned

Review December’s Meeting Outcomes : Share and learn about Tier 2 systems being implemented in some Illinois districts Utilize the Tier 2 Logistics Checklist to find potential strengths & weaknesses in Tier 2 systems Evaluate utility of Tier 2 Logistics Checklist for their districts Troubleshoot logistics around Tier 2 interventions One of the best ways to remember something is to test yourself.

Logistics Checklist Illinois RTI Network, 2013

Tier 2 Topics Review SchedulingFidelity Decision Rules Communication with Parents & Students

Outcomes Participants will : Network around progress monitoring tools used for reading and math at the elementary, middle and high school levels. Use a checklist to apply research based best practices in progress monitoring to evaluate current tools. Evaluate progress monitoring processes and problem solve common errors.

Connection to Illinois Professional Teaching Standards This content is directly tied to Standard 7: Assessment  The competent teacher understands and uses appropriate formative and summative assessments for determining student needs, monitoring student progress, measuring student growth, and evaluating student outcomes.

PROGRESS MONITORING: COMMON UNDERSTANDING AND PURPOSE I-RtI Network Illinois RTI Network, 2013

What do we expect to see? Why do we spend time monitoring the progress of students receiving Tier 2 support?

What Should We See? Improved student performance – Effect Size: ½ to 1 Standard Deviation improvement Improvement among lowest achievers Teachers providing more effective feedback Teachers making better decisions Students who are more aware of their own performance

BASICS IN MONITORING STUDENT PROGRESS I-RtI Network

1.Providing high quality instruction/intervention matched to student needs and 2.Using learning rate over time and level of performance to 3.Make educational decisions. RtI is the Practice of:

Three tiers of support are stacked resources so that they are layered and aligned with each other Tier 1: ALL Tier 2: Some Tier 3: Few A Multi-tiered System of Supports

1.Start with core and standards based instruction 2.Identify prerequisite skills and examine where deficits exist 3.Then allocate resources to help close the achievement gap for all of our Learners Core learning outcomes Do deficits exist within individuals or groups of students? Adjust instruction Assess Problem Solve

A Coordinated Assessment System Purpose (within a problem solving system) – Screening – diagnostic – Monitoring progress of student learning – Program evaluation

Plan Development What will we do? What is the goal? Based on why, what will we do differently? What has the highest likelihood of success? How will we monitor progress & fidelity? What resources are needed? Plan Evaluation Did the plan work? Adequate progress? Was the goal met? Do we need to continue or modify the plan? Problem Analysis Why is it happening? When does it occur? With whom? Can’t do or Won’t do? Curricula, Instruction? Environment? Problem Identification What is the problem? What is expected? What is occurring? What is the context? Systematic Use of Progress Monitoring Data Illinois RtI Network

Progress Monitoring Data At Tier 2, Progress Monitoring data can be used to …

Uses of Progress Monitoring Data 1) To estimate the rates of improvement which allows for comparison to peers 2) To identify students who are not demonstrating or making adequate progress so that instructional changes can be made 3) To compare the efficacy of different forms of instruction – in other words, which instructional approach or intervention led to the greatest growth among students. National Center on Response to Intervention Website, 2010

Rates of Improvement

ID Students In Need of More Support

Compare Intervention Effectiveness

Summary of Uses As Part of a Tiered System of Supports Data for Problem Solving at Tier 2 To Identify Rates of Improvement To Identify Students In Need of More Support To Analyze the Effectiveness of Interventions

Time to Reflect Where do I see these uses present in my district? Where could my district or school strengthen our process? What is one thing I could do to begin to address this?

TOOLS AND EVALUATION I-RtI Network

Progress monitoring has a scientific base in assessment with over 30 years of research. Technical aspects of these tools make them adequate for monitoring the academic progress of students.

Progress Monitoring Standards Brief Valid, reliable, and evidence based Sufficient number of alternate, equivalent forms Sensitive to student growth Benchmarks of expected skill growth Specified rates of improvement

Psychometric Standards 1.Reliability 2.Validity 3.Sufficient number of alternate forms 4.Sensitivity to learning 5.Evidence of instructional utility 6.Specification of adequate growth 7.Description of benchmarks for adequate end- of-year performance or goal-setting process

Validated Forms of PM CBM differs from most approaches to classroom assessment in two important ways (Fuchs & Deno, 1991). 1.First, CBM is standardized so that the behaviors to be measured and the procedures for measuring those behaviors are prescribed and have been shown to be reliable and valid. 2.Second, with CBM, each weekly test is of equivalent difficulty and represents what the teacher wants the student to be able to do well at the end of the year.

Tools Chart

TOOL EVALUATION & NETWORKING I-RtI Network Illinois RTI Network, 2013

Progress Monitoring Rubric Header on cover page Descriptive info on each work group’s section

Iowa Department of Education’s Evaluation of PM Tools – # of forms and equivalence – Cost and time for student – Scientific adequacy: criterion measure, validity of level, reliability of level and ROI – Alignment with core – Training needed – Time for administration, scoring, data retrieval – Computer options

Checklist for Evaluating Tools Name of Measure: Academic Area : Level: Yes or No (Circle one) Y N 1. Does the measure provide a reliable score? Research evidence demonstrates that the measure is reliable (e.g. similar results are obtained when different people give the measure or when alternate forms of the measure are given at different times). Y N 2. Does the measure yield a reliable trend of student progress (reliability of slope)? The measure has been shown to accurately identify and differentiate the trend or rate of improvement of different students over time. Y N 3. Is the measure a valid indicator of the academic skill of interest? The measure provides meaningful information about the academic ‘construct’ being assessed (i.e. the cognitive skills or processes needed to complete the academic skill(s) of interest).

Using a Checklist to Evaluate Progress Monitoring Tools Choose a progress monitoring tool that you are currently using for PM Evaluate the tool using the 8 questions on the Checklist Write a one sentence summary of the strengths and weaknesses of the tool Share how the this evaluation process occurs in your district

What are You using?

Panel Discussion

CHALLENGES I-RtI Network

Problem: It doesn’t always fit PM

What to do?

What do instructors currently use to measure: Basic Skills: Basic Math, Reading, Writing Complex Skills: Reading Comprehension, Extended Writing Content Based Knowledge: Factual Information, Specialized Vocabulary Extending Knowledge, Skills/Problem Solving: Demonstrating High Level Reasoning

Psychometric Standards 1.Reliability 2.Validity 3.Sufficient number of alternate forms 4.Sensitivity to learning 5.Evidence of instructional utility 6.Specification of adequate growth 7.Description of benchmarks for adequate end- of-year performance or goal-setting process

Student Outcomes Progress Monitoring Interim Summative Student Growth Informal Formative Formal Formative High Quality Instruction

A Coordinated Assessment System Monitoring progress of student learning – PM – Formal Formative – Interim – Informal Formative – Summative

Progress Monitoring Standards Brief Valid Reliable Repeatable: a sufficient number of alternate, equivalent forms Sensitive to student growth Specified rates of improvement Necessary characteristics of any high quality assessments that measure student growth.

What makes our traditional research-based PM for RtI purposes so good? Then, make sure you build these features into formal formative assessments to the greatest degree possible. Validity -Does it measure what it is intended to measure? o Essential to have very clear learner outcomes prior to test construction o Individual items need to be pure o Need to be independent of other (unrelated) factors  Readability level  Cultural differences  Prior knowledge Reliability (can be very sensitive to error) -Sources of error: o Poorly designed scoring procedures. o Different scorers o Different administration o Different question formats  Language  Proportion -Necessary features o clearly defined and understood scoring rubrics o standardized administration o Repeatability -required features o parallel forms  question type  proportional weight to items o comparable conditions o standardized administration protocol

Time to Reflect How has this changed my understanding of monitoring progress of student learning?… One thing I would like to have clarified would be… One thing I could do to apply this learning is…

PROGRESS MONITORING PROCESS I-RtI Network Illinois RTI Network, 2013

Progress Monitoring Process  Choose appropriate PM tools  Create a year long schedule for the collection of PM data  Schedule regular meetings with agendas to evaluate the data  Establish a written data evaluation process with decision rules  Establish written practices to ensure fidelity of administration, scoring and interpretation

Take a Look at the Process To consider whether a practice has been implemented with fidelity, the practice must first be clearly defined (Century, Rudnick & Freeman, 2010). By clearly defining a practice, expectations are spelled out creating an understanding of what needs to be accomplished. From that understanding, educators are able to reflect on the integrity of their work and then plan for next steps to improve implementation (Fixsen, Blasé, Horner & Sugai, 2009).

ISSUES AND TROUBLE SHOOTING I-RtI Network Illinois RTI Network, 2013

Issue: Lack of Adequate Progress Look at Fidelity in several areas: 1.Research Based & Matched to Student Need 2.Delivered as Designed 3.Sufficient Time 4.Student Engagement 5.Training Provided to Interventionist 52

Issue: Middle and High School Reading: – PM using a 3 minute maze passage Writing: – PM using a 5 minute writing sample Content Area Knowledge: – PM using a 5 minute vocabulary matching of terms with definitions

Issue: Communication of Results Teachers Students Parents Stakeholders

Issue: Lack of PM Data Students are receiving instruction, but progress monitoring data are not being collected consistently. Data are being collected, but not being used appropriately. Decisions are not made using explicit decision rules.

Issue: Frequency Does the how often do you make decisions about changes to instruction or grouping? Does your data collection keep pace with this timeline? Does it go over? Does it not keep up?

Issue: PD and Support Do teachers and assessment personnel understand the PM process? Who provides training and ongoing support? What process is used to evaluate the tools and process systematically? Who decides what to add and want to eliminate as evidenced by data?

Issue: Goal Setting Three Methods – Standards Based Goals Indicates that a student who meets the expectation is likely to meet standards (either grade level or state test). – Norm Referenced Goals Indicates that a student who meets the expectation is likely to be successful working with in a group with typical students. – Growth Referenced Goals (critical for use in PERA decisions) Indicates that a student who meets the expectation is making progress relative to their own level of achievement, even though it may still be substantially below either normative information or standards. Setting goals is a value driven activity.

Issues Discussion Lack of Adequate Progress Middle School/High School Measurement Communication of Results Lack of PM Data Frequency of Assessment PD and Ongoing Support Goal Setting

Activity

Time to Reflect One thing I learned during this section is… One thing I would like to have clarified would be… One thing I could do to apply this learning is…

Key Ideas In Progress Monitoring Tool Evaluation 1.PM should result in greater student achievement. 2.PM tools, CBM and other types of tools, should adhere to scientific standards. 3.An RtI/MTSS system should include a process for evaluating tools.

Action Planning Three things I plan to do: