Presented to: Oklahoma State Board of Education August 20, 2013 Oklahoma Interruption Investigation Arthur Thacker

Introduction to HumRRO  HumRRO is a 62 year old non-profit research company  Education clients include: –NAEP/NAGB –SMARTER Balanced –PARCC –State education agencies (OK, FL, KY, MN, ND, CA, PA, UT)  Services provided include: –Psychometric consulting and processing –Validity studies –Alignment studies –Standards setting –Quality assurance

Overview  Some students completing the OK assessments in spring 2013 experienced computer delays/interruptions.  The focal disruptions occurred on 4/29 and 4/30, although other disruptions were recorded on other days.  The methodology and interpretation of results was conducted independently of the testing contractor.

Overview (cont.)  The purpose of this investigation was to determine if computer disruptions affected student scores  Specifically, the concern lies in disrupted student scores being lower than expected  We investigated multiple groups of interrupted students (grades 6-8 and high school) using multiple methods in an attempt to “converge” on a bias, if any could be detected.

Challenges to the Investigation  Students were not interrupted randomly or by design  Computer interruptions are not that uncommon, even when there is no identified issue during testing to be discovered  Individual students may have responded very differently to the interruption

Structure  Four “Cohorts” were investigated –Cohort A – All students who had an interruption in the Grades 6-8 dataset, regardless of day –Cohort B – Only examined those with interruptions on 4/29 and 4/30 –Cohort C – EOI Data for Algebra scores –Cohort D – EOI Data for English scores

Propensity Score Matching –Propensity scores used to “mimic” an equivalent sample for comparison to the interrupted group –Matched the interrupted sample with individuals in the non- interrupted group who were similar on all available variables that relate to 2013 scores, including: Prior year scores School-level scores Ethnicity Gender Free/Reduced Lunch

Algebra and English II EOI  Two sets of analyses were conducted for each EOI exam  Algebra –Grade 9 students in 2013 with Algebra scores matched to their Grade 8 Math scores from 2012 –Grade 7 and 8 students in 2013 with Algebra scores matched to their 2013 math (and reading) scores  English II –Grade 10 students in 2013 with English II scores matched to their 2013 Grade 10 US History scores –Grade 10 students in 2013 with English II scores matched to their 2011 Grade 8 reading (and math) scores

Analyses  All analyses performed on the matched Disrupted and Non-Disrupted groups 1.Mean differences on 2013 scores (Statistical, Meaningful) 2.R 2 differences when predicting 2013 scores separately 3.R 2 change when combining groups and adding dichotomous disruption variable 4.Applying Non-Disrupted regression equation from step 2 on Disrupted group as well as 5 th, 10 th, 90 th, and 95 th percentile cuts

Conclusion  The evidence shows that the effect of disruption on students’ scores was neither widespread nor large and the conclusions were not consistent across methods.  In addition, the results show that the effects did not always disadvantage the disrupted students, but at times the disrupted students did better than expected.  The most significant disruption impact seems to be for Algebra students in Grades 7 and 8, but the impact was relatively small and inconsistent across the distribution of students. Lower performing students tended not to perform as well as predicted.