Linking Progress Monitoring with High-Stakes Assessments: Setting Standards-Based Goals in Reading John Hosp & Kristen Missall The University of Iowa Pursuing the Promise June 12, 2012

Project Partners

Purpose of Project To identify research-based benchmarks for monitoring K-6 reading performance in standards-based goals that –Align with the Iowa CORE Content for Literacy –Accurately predict Iowa Assessments reading performance

Why is this Important? Despite being a high-achieving state, the gap between students with disabilities and their peers is large The role of special education is to support students in developing grade-level skills We need to decrease this gap to meet the needs of all students

Eliminate the Achievement Gap by 2020 Students with disabilities consistently perform below their peers on critical academic skills. The Iowa CORE provides standards of what all students need to learn to be successful in and out of school. To close the gap we need to determine what is a criterion of acceptable performance (CAP) for ALL students that is related to the standards, is meaningful, and is research based.

Eliminate the Achievement Gap by 2020-cont Iowa currently uses various ways to determine IEP goals (local norms, growth rates, DIBELS benchmarks) which may or may not be appropriate and lead to eliminating the achievement gap. Therefore, there is a need for Iowa-specific, cut- scores for setting reading goals and determining progress that predicts proficient performance on meaningful reading outcomes K-6.

Eliminate the Achievement Gap by 2020-cont An Iowa-Specific Focus will Provide: The Criterion of Acceptable Performance (CAP) based on technically adequate Progress Monitoring Tools and the Iowa Assessments Meaningful goals for standards-based IEPs that are based on research and are used universally across the state Identification of which Progress Monitoring Tools are most appropriate to use Better outcomes for Iowa students (a step toward eliminating the achievement gap)

National Assessment of Educational Progress (NAEP) Using state samples, annual comparisons of performance in reading and math for students in 4 th and 8 th grade Results reported based on “Basic” level which is roughly equivalent to Iowa Assessments (or NCLB) “Proficient” score –“Basic” scores represent criterion for acceptable performance or the lowest point necessary for grade-level success –Results can be examined for subgroups of children –We are interested in results for children with and without IEPs

Project Methods

Participating Sites and Students To provide a state-representative sample, at least 1 elementary school per AEA was included –3 middle schools to ensure 6 th grade –14 total schools >200 students per grade level –Match state level demographics

AEAs

Per Grade Proportions from State Enrollment IowaSample n%n% Total487, %1, % Female234, %n/a American Indian2,7840.6%221.2% Asian10,5432.2%15.9% Black28,3175.8%502.7% Hispanic33,9747.0%1226.6% White411, % % Multiracialn/a 372.0% Economic Disadvantage165, %73440% Limited English Proficiency20,3344.2%693.8% Students with Disabilities67, % %

Instruments

Progress Monitoring Tools Selection Criteria Technical Adequacy –Reliability & validity Cost Time/ease of administration/scoring Sensitivity to growth Alternate forms

Kindergarten AIMSweb –Letter Sound Fluency –Phoneme Segmentation Fluency –Nonsense Word Fluency DIBELS Next –Letter Naming Fluency –Nonsense Word Fluency –Phoneme Segmentation Fluency FAST –Letter Sound Fluency

First Grade AIMSweb –Phoneme Segmentation Fluency –Nonsense Word Fluency –Oral Reading Fluency DIBELS Next –Nonsense Word Fluency –Phoneme Segmentation Fluency –Oral Reading Fluency –Retell FAST –Sight Word Fluency –Oral Reading Fluency

Second Grade AIMSweb –Oral Reading Fluency –Maze DIBELS Next –Oral Reading Fluency –Retell FAST –Oral Reading Fluency

Third - Sixth Grade AIMSweb –Oral Reading Fluency –Maze DIBELS Next –Oral Reading Fluency –Retell –Maze FAST –Oral Reading Fluency

Outcome Instrument Iowa Assessments –K: Reading Profile Total Listening, Vocabulary, Word Analysis –Grades 1-6: Reading (parts 1 & 2)

Data Collection Within a 4-week window of Iowa Assessments administration, students were administered grade-appropriate Progress instruments All assessors were trained to reliability before data collection Team of assessors collected all data from a school over a 1-2 day period

Data Analysis Concurrent Validity –Bivariate correlation matrix Empirical Cut Scores –Receiver Operator Characteristic (ROC) and logistic regression Classification Accuracy –Sensitivity, specificity, and overall correct classification analyses

The Importance of Sufficient Sample Size

Overall Correct Classification (OCC) = (a+d)/(a+b+c+d) Kappa = (Observed – Chance)/(1-Chance) Observed = (a+d)/(a+b+c+d) Chance = (a+b)/(a+b+c+d) * (a+c)/(a+b+c+d) * (b+d)/(a+b+c+d) * (c+d)/(a+b+c+d) Sensitivity/True Positive Rate (TP) = a/(a+c) Specificity/True Negative Rate (TN) = d/(b+d) Positive Predictive Power (PPP) = a/(a+b) Negative Predictive Power (NPP) = d/(c+d) False Negative Rate (FN) = c/(a+c) or 1-Sensitivity False Positive Rate (FP) = b/(b+d) or 1- Specificity Outcome Instrument ProficientNonproficient Progress Proficient True Positive a False Positive b Instrument Nonproficient False Negative c True Negative d Adapted from Hosp, 2011

Results

Application for Standards- Based Goals

Standards for Comparison 1.Benchmarks –An empirically derived cut score on the screening or progress tool, that predicts proficiency/mastery on an outcome tool 2.Norms –Average performance of a group of similar students 3.Intraindividual –Future goal is set based on past performance

Do you only want students to be “successful” in your school? Broader education goals Shouldn’t we use Local Norms?

A Problem with Local Norms NationalLocal 5% +4.5% 0.5% 5% Example 1: Local Performance above National

A Problem with Local Norms NationalLocal 5% -60% 65% 5% Example 2: Local Performance below National

The same is true for using different norms/benchmarks for various subgroups of students –Do you really want to reduce expectations for a specific subgroup? –Reinforces the notion that difference = deficit Different Subgroup Norms/Benchmarks

Use of Standards-Based Goals Spring Fall 1%ile 41%ile 75%ile 99%ile Iowas Progress Tool

Questions and Discussion