1. Linking Progress Monitoring with High-Stakes Assessments: Setting Standards-Based Goals in Reading John Hosp & Kristen Missall The University of Iowa Pursuing the Promise June 12, 2012

2. Project Partners

3. Purpose of Project To identify research-based benchmarks for monitoring K-6 reading performance in standards-based goals that –Align with the Iowa CORE Content for Literacy –Accurately predict Iowa Assessments reading performance

4. Why is this Important? Despite being a high-achieving state, the gap between students with disabilities and their peers is large The role of special education is to support students in developing grade-level skills We need to decrease this gap to meet the needs of all students

5. Eliminate the Achievement Gap by 2020 Students with disabilities consistently perform below their peers on critical academic skills. The Iowa CORE provides standards of what all students need to learn to be successful in and out of school. To close the gap we need to determine what is a criterion of acceptable performance (CAP) for ALL students that is related to the standards, is meaningful, and is research based.

6. Eliminate the Achievement Gap by 2020-cont Iowa currently uses various ways to determine IEP goals (local norms, growth rates, DIBELS benchmarks) which may or may not be appropriate and lead to eliminating the achievement gap. Therefore, there is a need for Iowa-specific, cut- scores for setting reading goals and determining progress that predicts proficient performance on meaningful reading outcomes K-6.

7. Eliminate the Achievement Gap by 2020-cont An Iowa-Specific Focus will Provide: The Criterion of Acceptable Performance (CAP) based on technically adequate Progress Monitoring Tools and the Iowa Assessments Meaningful goals for standards-based IEPs that are based on research and are used universally across the state Identification of which Progress Monitoring Tools are most appropriate to use Better outcomes for Iowa students (a step toward eliminating the achievement gap)

8. National Assessment of Educational Progress (NAEP) Using state samples, annual comparisons of performance in reading and math for students in 4 th and 8 th grade Results reported based on “Basic” level which is roughly equivalent to Iowa Assessments (or NCLB) “Proficient” score –“Basic” scores represent criterion for acceptable performance or the lowest point necessary for grade-level success –Results can be examined for subgroups of children –We are interested in results for children with and without IEPs

11. Project Methods

12. Participating Sites and Students To provide a state-representative sample, at least 1 elementary school per AEA was included –3 middle schools to ensure 6 th grade –14 total schools >200 students per grade level –Match state level demographics

13. AEAs

14. Per Grade Proportions from State Enrollment IowaSample n%n% Total487,559100.0%1,876100.0% Female234,73748.1%n/a American Indian2,7840.6%221.2% Asian10,5432.2%15.9% Black28,3175.8%502.7% Hispanic33,9747.0%1226.6% White411,94184.5%157585.8% Multiracialn/a 372.0% Economic Disadvantage165,83034.0%73440% Limited English Proficiency20,3344.2%693.8% Students with Disabilities67,06513.8%22012.0%

15. Instruments

16. Progress Monitoring Tools Selection Criteria Technical Adequacy –Reliability & validity Cost Time/ease of administration/scoring Sensitivity to growth Alternate forms

17. Kindergarten AIMSweb –Letter Sound Fluency –Phoneme Segmentation Fluency –Nonsense Word Fluency DIBELS Next –Letter Naming Fluency –Nonsense Word Fluency –Phoneme Segmentation Fluency FAST –Letter Sound Fluency

18. First Grade AIMSweb –Phoneme Segmentation Fluency –Nonsense Word Fluency –Oral Reading Fluency DIBELS Next –Nonsense Word Fluency –Phoneme Segmentation Fluency –Oral Reading Fluency –Retell FAST –Sight Word Fluency –Oral Reading Fluency

19. Second Grade AIMSweb –Oral Reading Fluency –Maze DIBELS Next –Oral Reading Fluency –Retell FAST –Oral Reading Fluency

20. Third - Sixth Grade AIMSweb –Oral Reading Fluency –Maze DIBELS Next –Oral Reading Fluency –Retell –Maze FAST –Oral Reading Fluency

21. Outcome Instrument Iowa Assessments –K: Reading Profile Total Listening, Vocabulary, Word Analysis –Grades 1-6: Reading (parts 1 & 2)

22. Data Collection Within a 4-week window of Iowa Assessments administration, students were administered grade-appropriate Progress instruments All assessors were trained to reliability before data collection Team of assessors collected all data from a school over a 1-2 day period

23. Data Analysis Concurrent Validity –Bivariate correlation matrix Empirical Cut Scores –Receiver Operator Characteristic (ROC) and logistic regression Classification Accuracy –Sensitivity, specificity, and overall correct classification analyses

25. The Importance of Sufficient Sample Size

26. Overall Correct Classification (OCC) = (a+d)/(a+b+c+d) Kappa = (Observed – Chance)/(1-Chance) Observed = (a+d)/(a+b+c+d) Chance = (a+b)/(a+b+c+d) * (a+c)/(a+b+c+d) * (b+d)/(a+b+c+d) * (c+d)/(a+b+c+d) Sensitivity/True Positive Rate (TP) = a/(a+c) Specificity/True Negative Rate (TN) = d/(b+d) Positive Predictive Power (PPP) = a/(a+b) Negative Predictive Power (NPP) = d/(c+d) False Negative Rate (FN) = c/(a+c) or 1-Sensitivity False Positive Rate (FP) = b/(b+d) or 1- Specificity Outcome Instrument ProficientNonproficient Progress Proficient True Positive a False Positive b Instrument Nonproficient False Negative c True Negative d Adapted from Hosp, 2011

27. Results

28. Application for Standards- Based Goals

29. Standards for Comparison 1.Benchmarks –An empirically derived cut score on the screening or progress tool, that predicts proficiency/mastery on an outcome tool 2.Norms –Average performance of a group of similar students 3.Intraindividual –Future goal is set based on past performance

30. Do you only want students to be “successful” in your school? Broader education goals Shouldn’t we use Local Norms?

31. A Problem with Local Norms NationalLocal 5% +4.5% 0.5% 5% Example 1: Local Performance above National

32. A Problem with Local Norms NationalLocal 5% -60% 65% 5% Example 2: Local Performance below National

33. The same is true for using different norms/benchmarks for various subgroups of students –Do you really want to reduce expectations for a specific subgroup? –Reinforces the notion that difference = deficit Different Subgroup Norms/Benchmarks

34. Use of Standards-Based Goals Spring Fall 1%ile 41%ile 75%ile 99%ile Iowas 98 12 120 135 25 52 Progress Tool

35. Questions and Discussion