Demonstrating Growth for Low-Achieving Students With Disabilities Assessment Toolbox Demonstrating Growth for Low-Achieving Students With Disabilities Laura Kuchle, American Institutes for Research Lou Danielson, American Institutes for Research Pakethia Harris, American Institutes for Research Kristin Ruedel, American Institutes for Research Lynn Fuchs, Vanderbilt University 04.15.2016

Session Overview Need for Ongoing, Sensitive Assessment to Inform Intervention Why Progress Monitoring? Selecting Assessments Aggregating for Program Evaluation Monitoring Implementation Discussion

Drivers for This Work National Center for Systemic Improvement (NCSI) Supporting states with planning and implementing their state systemic improvement plans, part of OSEP’s Results Driven Accountability National Center on Intensive Intervention (NCII) Improving intervention for students with severe and persistent learning and behavioral difficulties

Who Is in the Room?

Need for Ongoing, Sensitive Assessment to Inform Intervention

Why Do We Need Data-Driven Intensive Intervention? Students with disabilities have… Lower academic achievement Higher dropout rates Higher arrest rates For more information: 2015 NAEP (http://www.nationsreportcard.gov/); Sanford et al., 2011; Planty et al., 2008; Aud et al., 2012

Who Needs Intensive Intervention? Students with disabilities who are not making adequate progress in their current instructional program Students who present with very low academic achievement and/or high-intensity or high-frequency behavior problems (typically those with disabilities) Students in a tiered intervention system who have not responded to secondary intervention programs delivered with fidelity

Types of Assessment Type When? Why? Summative After Assessment of Learning Diagnostic Before Identify skill strengths and weakness Formative During Assessment for Learning

Why Summative Assessments Are Not Enough Low sensitivity to improvement for the target population Assessment is too infrequent to inform timely intervention adaptations If using state tests, many don’t start until third grade

Why Progress Monitoring?

What Is Progress Monitoring? A standardized method of formative assessment tells us how well students are responding to instruction. Progress monitoring tools have the following characteristics: Brief assessments Repeated measures that capture student learning Measures of age-appropriate outcomes Reliable, valid, and evidence based

Progress Monitoring Informs Student-Level Intervention Decisions Data allow us to… Estimate the rates of improvement across time. Compare the efficacy of different forms of instruction. Identify students who are not demonstrating adequate progress. Determine when an instructional change is needed.

Approaches to Progress Monitoring What is the difference?

Benefits of Monitoring Progress With General Outcome Measures Number of Assessments/15 Weeks Effect Size (SD) Percentile Gain 1 .34 13.5 5 .53 20 10 .60 22.5 15 .66 24.5 .71 26 25 .78 28.5 30 .82 29 Source: Bangert-Drowns, Kulik, and Kulik (1991); similar results found by Fuchs and Fuchs (1986).

Common Reading Measures Recommended Grades Letter Naming Fluency (LNF) Letter Sound Fluency (LSF) Phoneme Segmentation Fluency (PSF) K Nonsense Word Fluency (NWF) Late K–1 Word Identification Fluency (WIF) 1 Passage Reading Fluency (PRF), also called Oral Reading Fluency (ORF) Late 1–4 Maze or Maze Fluency 4+

Common Mathematics Measures Domain Measures Grades Early numeracy Oral Counting Next Number Number Identification Quantity Discrimination Missing Number K–1 Computation M-CBM Math Computation Number Facts 1–8 Concepts and applications Math Concepts and Applications Concepts Concepts/Applications 2–8

Selecting Assessments

Review of Screening Tools Center on Response to Intervention http://www.rti4success.org/resources/tools-charts/screening-tools-chart

Review of Progress Monitoring Tools National Center on Intensive Intervention http://www.intensiveintervention.org/chart/progress-monitoring

Considerations When Selecting or Evaluating a Tool Skills to be measured—age and grade appropriate Cost and training requirements Administration and scoring time Data management Technical rigor (consider population)

Dimensions of Technical Rigor Reliability Validity Evidence of being sensitive to change Alternate/parallel forms: different versions of the assessment that are of comparable difficulty

Should We Ever Assess Off-Level… Should We Ever Assess Off-Level…? Consider the Purpose of the Assessment Screening to identify students at risk for poor learning outcomes should always occur at grade level and do the following: Determine students’ response to grade-level core instruction. Assess performance relative to grade-level expectations. Provide schoolwide data regarding the percentage of students in each grade level who are at or below benchmarks.

Should We Ever Assess Off-Level… Should We Ever Assess Off-Level…? Consider the Purpose of the Assessment (2) Progress monitoring should be done at grade level when possible, but the following is also applicable: It must also match a student’s instructional level. If a student’s performance is well below grade-level expectations, grade-level probes are unlikely to be sensitive to growth. Off-level assessment may be warranted in these cases.

More Information From NCII Academic Progress Monitoring http://www.intensiveintervention.org/resource/usi ng-academic-progress-monitoring-individualized- instructional-planning-dbi-training Academic Diagnostic Assessment http://www.intensiveintervention.org/resource/inf ormal-academic-diagnostic-assessment-using- data-guide-intensive-instruction-dbi-training

Aggregating for Program Evaluation

Thinking Beyond the Student Level Aggregate progress monitoring data to inform programmatic decisions. Sample question: Have our system changes (e.g., professional development or new service delivery framework) resulted in improved student performance?

How Can Progress Monitoring Data Inform Systems-Level Decisions? Aggregate progress monitoring results for program evaluation: Timely and sensitive outcome measure Already available as important part of improvement efforts targeting students with intensive needs: Information about both outcomes and implementation

Potential Challenges Whose data do we need? Sample considerations Tracking a sample across years How can we aggregate across different progress monitoring tools and measures?

Selecting a Sample For smaller populations, can you analyze all available progress monitoring data? If selecting program pilot sites, consider the feasibility of program implementation and collecting or accessing data. If sampling only a portion of program participants, consider the representativeness of your sample.

Tracking a Sample Across Multiple School Years Longitudinally examine performance for a group of students. What about students who enter or exit the program or target group? Consider tracking progress for a broader sample of students who are at risk. NCSI Brief: Advantages of Assessing SiMR Progress by Tracking a Sample Over Time http://ncsi.wested.org/news-events/ncsi-brief- advantages-of-assessing-simr-progress-by- tracking-a-sample-over-time/

Aggregating Across Different Measures Within a state or district, schools may use different progress monitoring tools. Within a school, students receive different measures according to grade and time of year. Within a grade, students’ progress may need to be monitored with different measures: Skill targeted by intervention Current level of performance

One Possible Approach: Percentage of Expected Slope How does the student’s slope or rate of improvement compare to the “expected” slope for students at the same instructional level? Consult norms provided by the tool developer: The NCII tool chart includes the reliability and predictive validity of the slope. Find the mean or median slope for the appropriate grade and time of year, for the measure used for progress monitoring: When available, find the norm for the student’s initial level of performance.

Calculating the Percentage for One Student Student’s slope divided by expected slope Example: Julia’s slope = 1 word per minute (wpm) per week Expected slope = 2 wpm/week Julia’s percentage of expected slope = 1 2 =.5=50%

Aggregating Across Students Find the mean percentage across all students. Example: Jim = 50% Mary = 200% Ted = 150% Linda = 100% Mean = 125%

Aggregation Example—Multiple Measures and Grades Student Measure Student’s Slope Expected Slope Percentage Jim Third-grade reading fluency 0.5 1 50% Mary First-grade math computation 1.5 0.75 200% Ted Sixth-grade math concepts and applications 0.3 0.2 150% Linda First-grade nonsense word fluency 100% Mean percentage across all students = 125% Literacy students = 75% Math students = 175% First-grade students = 150%

Using Mean Percentage for Program Evaluation Compare the program mean to the baseline or comparison program or group. Examples: Comparison groups: School A (new program) = 125% School B (business as usual) = 80% Pre/post for one group: Baseline (old program) = 90% New program = 125%

Monitoring Implementation

Importance of Evaluating Implementation Implementation will be challenging and occur over time. Early and ongoing (formative) evaluation of implementation will help to: Document early successes. Identify solutions that foster expected progress.

NCSI Implementation Evaluation Matrix Available under “SSIP Phase III” resources at: http://ncsi.wested.org/resources/tools-publications/

Evaluating District and School Frameworks If you are using a specific framework for assessment and intervention, you may find tools to help you assess the use of that framework or process. For example: RTI Essential Components Integrity Rubric and Worksheet http://www.rti4success.org/resource/essential- components-rti-integrity-rubric-and-worksheet Schoolwide PBIS evaluation tools https://www.pbis.org/evaluation/evaluation-tools

NCII Tools for Measuring Data-Based Individualization (DBI) Implementation DBI Implementation Rubric and Interview http://www.intensiveintervention.org/resource/dbi- implementation-rubric-and-interview Student-Level DBI Implementation Checklists http://www.intensiveintervention.org/resource/student-level- data-based-individualization-implementation-checklists DBI Implementation Log: Daily and Weekly Intervention Review http://www.intensiveintervention.org/resource/data-based- individualization-implementation-log-daily-and-weekly- intervention-review Intensive Intervention Implementation Review Log http://www.intensiveintervention.org/resource/intensive- intervention-implementation-review-log

Intervention Fidelity If you are using one or more specific interventions or programs, see if the publisher: Identifies intervention components considered essential for: Strong implementation Achieving intended student outcomes Provides a tool for assessing fidelity (e.g., checklist or rubric)

Questions and Discussion The learning continues! Give us your name and email address if you would like to know more about this issue. Connect with NCSI and NCII for additional resources and future webinars www.ncsi.wested.org NCSI@wested.org Twitter: @TheNCSI www.intensiveintervention.org ncii@air.org Twitter: @TheNCII

References Bangert-Drowns, R. L., Kulik, J. A., & Kulik, C.-L. C. (1991). Effects of frequent classroom testing. Journal of Educational Research, 85, 89–99. Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: A meta- analysis. Exceptional Children, 53(3), 199–208.

THANK YOU! For more information, please contact Kristin Ruedel, the NCSI Data Use Lead, at kruedel@air.org www.ncsi.wested.org NCSI@wested.org Twitter: @TheNCSI www.intensiveintervention.org ncii@air.org Twitter: @TheNCII

THANK YOU! http://ncsi.wested.org | @TheNCSI