Making Data Driven Decisions: Cut points, Curve Analysis and Odd Balls Laura Lent IU 13 Pennsylvania

District-Wide RtI Development Essential Question: How do you use data to facilitate systemic paradigm shifts?

District Demographics 3200 students total 5 Elementary Buildings, 2 Secondary 4 K-4 buildings ranging from 110 to 500 students building of 630 students 25-35% FRL <15% minority Equivalent of 2.5 full-time school psychologists

PSSA Scores: Reading Warning=(W), School Improvement 1=SI 1 Grade (W)60 (SI 1)62 (SI 1)

Intensive Reform Focus 5 th /6 th grade building the identified patient due to School Improvement status. Building has 5 teacher teams of a pair of 5 th grade teachers with a pair of 6 th grade teachers. Two 5 th grade classrooms allocated as ELM or Essentials of Literacy and Math. Each team has a learning support teacher assigned to it who does inclusion during social studies and science. No universal screening. No consistent reading instructional practices.

Data Collection Use of CBMs for Universal Screening and Progress Monitoring including: Early Literacy, Early Numeracy, MAZE, R-CBM, Math Applications and Single Digit Computation. K Screening: Use of Individual Growth and Development Indicators (IGDIs)-Picture Naming and Rhyming and Early Numeracy in August to form heterogeneous classes.

Benchmark Data Analysis Move from DIBELS to AIMSweb allowed for determination of local benchmark criteria to be used in the reporting system. Question: How do we set benchmarks that are sensitive and specific to PSSA performance?

ROC Curve Analysis Receiver Operating Characteristic (ROC) Statistical Evaluation Process to identify benchmarks by identifying cut scores that are the most sensitive without sacrificing specificity. Sensitivity: Fewest False Negatives Specificity: Fewest False Positives

Scatterplot Interpretation of Screening Results False Positives/ Happy Surprises Adapted from Silberglitt (2009). True Negatives True Positives False Negatives/ Unhappy Surprises PSSA Proficient R-CBM Low Risk

Benefits of ROC Curve Analysis Re-set Benchmarks for Greater Classification Accuracy. Minimize the word caller hysteria. Fringe Benefits: Odd Ball or Outlier Score Investigation.

Acknowledgement Thanks to Dr. Edward Shapiro and Dr. Gini Hampton, Center for Promoting Research to Practice, Lehigh University, for sharing the proceeding slides of ROC curve analysis for this district.

ROC Curve Analysis R-CBM District Benchmarks- 25 th %tile GradeScore - WCPM

Probability Outcomes based on AIMSweb (25 th %tile) Spring Benchmarks Grade 4 = 104 Adv/Proficient Basic/Belo w BasicTotal RCBM BMK=104Low RiskCount % within RCBM BMK= %14.0%100.0% % of Total 68.3%11.1%79.4% Some RiskCount % within RCBM BMK= %57.1%100.0% % of Total 6.0%7.9%13.9% At RiskCount % within RCBM BMK= %94.1%100.0% % of Total.4%6.3%6.7% TotalCount % within RCBM BMK= %25.4%100.0% % of Total 74.6%25.4%100.0%

Probability Outcomes based on Reset R-CBM Spring Benchmarks Grade 4 = 117

Probability Outcomes based on AIMSweb (25 th %tile) Spring Benchmarks Grade 5 = 114 Adv/Proficien t Basic/Below BasicTotal RCBM BMK=114Low RiskCount % within RCBM BMK= %30.1%100.0% % of Total 58.0%25.0%83.0% Some RiskCount % within RCBM BMK= %74.1%100.0% % of Total 2.4%6.9%9.4% At RiskCount % within RCBM BMK= %90.9%100.0% % of Total.7%6.9%7.6% TotalCount % within RCBM BMK= %38.9%100.0% % of Total 61.1%38.9%100.0%

Probability Outcomes based on Reset R-CBM Spring Benchmarks Grade 5 = 140

Probability Outcomes based on AIMSweb (25 th %tile) Spring Benchmarks Grade 6 = 130 Adv/Proficient Basic/Below BasicTotal RCBM BMK=130Low RiskCount % within RCBM BMK= %18.1%100.0% % of Total 66.2%14.7%80.8% Some RiskCount % within RCBM BMK= %69.4%100.0% % of Total 4.1%9.4%13.5% At RiskCount 015 % within RCBM BMK=130.0%100.0% % of Total.0%5.6% TotalCount % within RCBM BMK= %29.7%100.0% % of Total 70.3%29.7%100.0%

Probability Outcomes based on Reset R-CBM Spring Benchmarks Grade 6 = 154 Adv/Proficient Basic/Below BasicTotal RCBM Bmk=154 Gr 6Low RiskCount % within RCBM Bmk=154 Gr %13.8%100.0% % of Total 54.1%8.6%62.8% Some/At RiskCount % within RCBM Bmk=154 Gr %56.6%100.0% % of Total 16.2%21.1%37.2% TotalCount % within RCBM Bmk=154 Gr %29.7%100.0% % of Total 70.3%29.7%100.0%

ROC Curve Analysis to Reset Cut Points Reset Spring Targets: R-CBM Proficient GradeSpring R-CBM Score Spring R-CBM Percentile th th nd th

What to do about those odd balls? To investigate the scores that were false positives and false negatives or happy surprises and unhappy surprises the following variables were examined: Current and historical PSSA performance Demographic characteristics including ELL, Economic Disadvantage (ED), IEP status, gender and teacher assignment.

False Positive Results Students who were at risk or below the 10 th percentile on the spring R-CBM but scored Proficient or above on PSSA. Grade3456 Number of Scores 0120

Extreme False Negative Results or Very Unhappy Surprises Students who were low risk on spring RCBM and Below Basic on PSSA. Grade3456 Number of False Positives 41176

Who are these students? 5 th Grade (n=17): –11 Male, 6 Female –7 with IEPs, 4 ED, 1 ELL –3 scored Proficient in 4 th grade –Teacher distribution: all had at least 1, three teachers held 3 or more. 6 th Grade (n=6): –5 Female, 1 Male –1 with IEP, 1 ELL –None scored Proficient in 5 th grade

Near False Negative Results Students who scored low risk on RCBM and Basic on PSSA. Grade3456 Number of False Positive Scores

Who are these kids? Grade 3: –Evenly distributed across schools (n=3, n=5, n=5) –3 students ED, 3 students IEP, 1 ELL Grade 4: –Evenly distributed across schools (n=7, n=10, n=8) –5 students retainees, 5 ED *12/23 were students who previously scored Proficient!

5 th Grade: 12/55 retentions, 12 ED (1 overlap), 2 IEP 31 fell from Proficient to Basic 2 fell from Advanced to Basic *33/47 or 70% were students who previously scored Proficient!

6 th Grade: 6 were retainees, 5/ED, 4/IEP, 0 ELL 11/33 or 33% fell from Proficient to Basic!

Teacher Distribution: 5 th Grade Total # False Negatives =72 Of the 12 teachers, F had the most false positives outside of the combined ELM classes. TABCDEFGHELM Total # False Neg I 5 J 4

Teacher Distribution: 6 th Grade Total Number of False Negatives=33 TABCDEFGHI # False Neg

Local Impact of Odd Ball Analysis Consensus-Building: Administration conceded that major infrastructure and implementation changes were needed. Staff agreed. Infrastructure: 5/6 teams were restructured into 5 th and 6 th only teams. Weaker teams paired with stronger and 5 th grade classrooms moved closer to the office. Implementation: 6 full days professional development on core reading instruction. Administrators attend all inservice trainings and all team meetings. R-CBM targets re-set for to attempt to better identify students who need targeted intervention. Targeted intervention provided daily for 45 minutes by all classroom teachers and interventionists. Groups matched to identified need.

Questions for Further Analysis Do the False Negatives fail to identify those with comprehension problems or fail to identify those who are failing to receive adequate instruction? Would MAZE identify the same sample of False Negatives as at risk? Would a diagnostic or benchmark measure based on grade level standards have identified these students as at risk?