Human Resources Research Organization (HumRRO) 66 Canal Center Plaza, Suite 700 Alexandria, Virginia 22314-1578 | Phone: 703.549.3611 | Fax: 703.549.9661.

Slides:



Advertisements
Similar presentations
Using Growth Models to improve quality of school accountability systems October 22, 2010.
Advertisements

School Performance Variation on COMLEX Level 1 Linjun Shen, Ph.D., MPH National Board of Osteopathic Medical Examiners.
Teacher Effectiveness in Urban Schools Richard Buddin & Gema Zamarro IES Research Conference, June 2010.
Comparison of Half- and Full-Day Kindergarten on Kindergarten Achievement Jack B. Monpas-Huber, Ph.D. Director of Assessment and Student Information.
Presented to: Oklahoma State Board of Education August 20, 2013 Oklahoma Interruption Investigation Arthur Thacker.
Standardized Tests What They Measure How They Measure.
Designing Accessible Reading Assessments Reading Aloud Tests of Reading Review of Research from the Designing Accessible Reading Assessments Projects Cara.
1 Performance of English Language Learners on the 2008 Grades 3-8 ELA Tests David Abrams Assistant Commissioner for Standards, Assessment, and Reporting.
Dallas ISD’s Value-Added Model School Effectiveness Index (SEI) Classroom Effectiveness Index (CEI) Data Analysis, Reporting, and Research Services.
TEACHER EFFECTIVENESS INITIATIVE VALUE-ADDED TRAINING Value-Added Research Center (VARC)
Districts and States Working with VARC Minneapolis Milwaukee Racine Chicago Madison Tulsa Atlanta New York City Los Angeles Hillsborough County NORTH DAKOTA.
Post-Test Data Analysis Workshop. Agenda I. Analyze 2014 TerraNova Results II. Comparative Data Analysis: TerraNova & Explore Test Results III. Using.
UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Keeping Kids in School:
Ch 11 – Probability & Statistics
Different Skills? Identifying Differentially Effective Teachers of English Language Learners Ben Master, Susanna Loeb, Camille Whitney, James Wyckoff 5.
Regression Part 3 of 3 Parts Multiple Regression Overview Examples Hypothesis Tests MR ANOVA Table Interpretation Indicator Variables Assumptions Homework.
Overview of the MAP Assessment Implementation Appoquinimink School District Board Meeting 10/13/09.
Annotated Bibliography Presentation April 17, 2013 Carol Redmond ED 521: Educational Research and Analysis Flipping Literacy and Improved Reading Levels.
The Learning Behaviors Scale
KCCT Kentucky’s Commonwealth Accountability Testing System Overview of 2008 Regional KPR.
Identifying the gaps in state assessment systems CCSSO Large-Scale Assessment Conference Nashville June 19, 2007 Sue Bechard Office of Inclusive Educational.
Using Data to Make Decisions Regarding Professional and Personalized Learning: WV General Summative Assessment – Part 1 County Chief Instructional Leadership.
1 New York State Growth Model for Educator Evaluation 2011–12 July 2012 PRESENTATION as of 7/9/12.
The Impact of Including Predictors and Using Various Hierarchical Linear Models on Evaluating School Effectiveness in Mathematics Nicole Traxel & Cindy.
What Was Learned from a Second Year of Implementation IES Research Conference Washington, DC June 8, 2009 William Corrin, Senior Research Associate MDRC.
SAT 10 (Stanford 10) 2013 Nakornpayap International School Presentation by Ms.Pooh.
Slide 1 Estimating Performance Below the National Level Applying Simulation Methods to TIMSS Fourth Annual IES Research Conference Dan Sherman, Ph.D. American.
State Charter Schools Commission of Georgia SCSC Academic Accountability Update State Charter School Performance
Research and Statistics in Psychology Wadsworth, a division of Thomson Learning.
Thinking Mathematically Statistics: 12.5 Problem Solving with the Normal Distribution.
Counseling Research: Quantitative, Qualitative, and Mixed Methods, 1e © 2010 Pearson Education, Inc. All rights reserved. Basic Statistical Concepts Sang.
Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison.
Online Course Evaluations Is there a perfect time? Presenters: Cassandra Jones, Ph.D., Director of Assessment Michael Anuszkiewicz, Research Associate.
The Nation’s Report Card: U.S. History National Assessment of Educational Progress (NAEP)
The Nation’s Report Card Science National Assessment of Educational Progress (NAEP)
35th Annual National Conference on Large-Scale Assessment June 18, 2005 How to compare NAEP and State Assessment Results NAEP State Analysis Project Don.
Precision Gains from Publically Available School Proficiency Measures Compared to Study-Collected Test Scores in Education Cluster-Randomized Trials June.
Copyright © 2010, SAS Institute Inc. All rights reserved. How Do They Do That? EVAAS and the New Tests October 2013 SAS ® EVAAS ® for K-12.
A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.
The Nation’s Report Card 4th-Grade Reading SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP),
CREP Center for Research in Educational Policy SES Student Achievement Methods/Results: Multiple Years and States Steven M. Ross Allison Potter The University.
School-level Correlates of Achievement: Linking NAEP, State Assessments, and SASS NAEP State Analysis Project Sami Kitmitto CCSSO National Conference on.
Race to the Top Assessment Program: Public Hearing on Common Assessments January 20, 2010 Washington, DC Presenter: Lauress L. Wise, HumRRO Aab-sad-nov08item09.
Mini-Project #2 Quality Criteria Review of an Assessment Rhonda Martin.
Beginning in the school year, a student scoring at the lowest achievement level in reading on the 3 rd Grade Summative Assessment will not be.
Copyright © 2010, SAS Institute Inc. All rights reserved. How Do They Do That? EVAAS and the New Tests October 2013 SAS ® EVAAS ® for K-12.
Descriptive Statistics Research Writing Aiden Yeh, PhD.
Evidence for Impact on Student Learning Tony Norman Associate Dean, CEBS In P. R. Denner (Chair), Evidence for Impact on Student Learning from the Renaissance.
The Normal Distribution and Norm-Referenced Testing Norm-referenced tests compare students with their age or grade peers. Scores on these tests are compared.
Effectiveness of Selected Supplemental Reading Comprehension Interventions: Impacts on a First Cohort of Fifth-Grade Students June 8, 2009 IES Annual Research.
Florida Department of Education’s Florida Department of Education’s Teacher Evaluation System Student Learning Growth.
Evaluation Results MRI’s Evaluation Activities: Surveys Teacher Beliefs and Practices (pre/post) Annual Participant Questionnaire Data Collection.
Arizona LEARNS: Overview of the Achievement Profiles.
“The retained student” Ethnic minorities: black/Hispanic Gender: male Low SES Late birthday Parents normally a dropout; little or no involvement Behavior.
Psychology’s Statistics Appendix. Statistics Are a means to make data more meaningful Provide a method of organizing information so that it can be understood.
1 Main achievement outcomes continued.... Performance on mathematics and reading (minor domains) in PISA 2006, including performance by gender Performance.
1 New York State Growth Model for Educator Evaluation June 2012 PRESENTATION as of 6/14/12.
Examining Achievement Gaps
Preliminary Review of the 2012 Math SOL Results
Inference and Tests of Hypotheses
Preliminary Analysis of EOG/EVOS Data – Greene County 2009,2010,2011
CMSD Fall Data Check In November 3, 2017.
Module 8 Statistical Reasoning in Everyday Life
Myers EXPLORING PSYCHOLOGY (6th Edition in Modules)
Validating Student Growth During an Assessment Transition
Correlates of District-Level Performance
Student Mobility and Achievement Growth In State Assessment Mohamed Dirir Connecticut Department of Education Paper presented at National Conference.
POWER UP! James took the National Career Assessment Examination. When the results were handed to him, he saw that he got 95 percentile. James wonders if.
Developing Honors College Admissions Rubric to Ensure Student Success
Impact of Interruptions on Test Scores in Indiana
Presentation transcript:

Human Resources Research Organization (HumRRO) 66 Canal Center Plaza, Suite 700 Alexandria, Virginia | Phone: | Fax: | A Statistical Investigation of the Effects of Computer Disruptions on Student and School Scores Presented at: National Conference on Student Assessment New Orleans, LA Presenters :Bethany H. Bynum (HumRRO) Gene Hoffman (HumRRO) Art Thacker (HumRRO) Matt Swain (James Madison University) June 25, 2014

Computer Disruptions ● In 2013, several States experienced computer problems during student testing. –Minnesota’s Comprehensive Assessment –Kentucky's high school end-of-course exams –Indiana’s ISTEP exam –Oklahoma Student Testing Program ● Thousands of students were effected by wide-spread computer disruptions –Kicked off the test –Long load times ● States were concerned that these disruptions could detrimentally impact student scores 2

Research Question ● Minnesota and Oklahoma: –Commissioned a study to examine the impact of computer disruptions on student scores. –The type of disruption was different, but we used the same statistical approach. ● “On average, would student scores have been different if computer disruption did not occur?” ● Things to consider: –Minnesota allowed students the opportunity to review items or retake the test, but did not keep record of who took advantage of this opportunity. –Oklahoma allowed students that were kicked off to retake the test. 3

Defining Disruption ● Oklahoma: Computer issues occurred on April 28 and April 30 –Students kicked off of the test in the middle of the exam. –The Vendor had record of every student that was kicked off. ● Minnesota: Computer issues occurred on April 16 and 23 –Long item latencies – the amount of time it take an online page to load –Abnormal restarts – students unexpectedly logged out and required to log back in to resume test –Administrative Pauses – test paused by teacher or test proctor –No clear record of who was or was not interrupted 4

Defining Disruption 5 Page Latency in Seconds DateLatency RecordsMeanMedian90th95th99th April 16129, April 231,072, April 302,176, Percentage of Students with Indicated Latencies DateN> 30 sec20-30 sec15-20 sec10-15 sec5-10 sec April 16 23, April 23 65, April 30 95,

Research Approach ● Basic approach: –Based on the premise that student scores are consistent over time. –Match disrupted students to similarly performing non-disrupted students. Using Propensity Matching (Prior year scores, Gender, Ethnicity, FRP Lunch, LEP, School-level FRP Lunch, School-level achievement) –Examine differences in 2013 test scores between the two groups. ● Student-level Statistical Investigation: 1.Are there mean differences in scores between the disrupted and non- disrupted groups? 2.Does disruption add to the prediction of the 2013 tests scores? 3.Are there differences in prediction between the disrupted and non- disrupted groups? 4.Would disrupted students have done differently under normal testing conditions? 5.Compare the distribution of differences between the disrupted and non- disrupted samples. 6

Student-level Statistical Investigation 1. Are there mean differences in scores between the disrupted and non-disrupted groups? 7

Student-level Statistical Investigation 2. Does disruption add to the prediction of the 2013 tests scores? 8

Student-level Statistical Investigation 3. Are there differences in prediction between the disrupted and non-disrupted groups? 9

Student-level Statistical Investigation 4. Would disrupted students have done differently under normal testing conditions? –For each disrupted student, apply the non-disrupted prediction equation and compute the predicted score. –Take the difference between the predicted score and observed score (score obtained under the disrupted conditions) –For, the non-disrupted sample, computed the difference between predicted scores and observed scores ● Large numbers of students with notable differences between observed and predicted scores provides another piece of evidence about the impact of the computer disruptions. 10

Student-level Statistical Investigation 5. Compare the distribution of differences between the disrupted and non-disrupted samples. –Determine the difference in observed and predicted scores at the 5 th, 10 th, 90 th and 95 th percentile for the non-disrupted group –Apply these cuts to the disrupted group and determine what percent of students fall at or above the 90 th and 95th cuts and what percent of students that fall at or below the 5 th and 10 th cuts. –Larger than 5 and 10 percent of the sample, would provide evidence that the computer disruption may have impacted scores. 11

Student-level Statistical Investigation Percent of Disrupted Students with Predicted and Observed Score Differences at the 5 th, 10 th, 90 th and 95 th Percentile of Non-Disrupted Students 12

School-Level Score Differences ● School-level Statistical Investigation: 1.Would school-level scores differ if disrupted students were excluded? 2.Does disruption add to the prediction of 2013 school-level means? 3.Would there be differences in % proficient if predicted scores were used in place of observed scores? 13

School-Level Score Differences Would school-level scores differ if disrupted students were excluded? ● All students included (baseline) ● Students in disrupted sample excluded –School-level means increased slightly (d = -.02 to.22) ● Students that test on April 16 or 23 excluded –School-level means dropped on average by.09 theta points (d = ) ● Predicted score are used in place of the observed score for students that were in the disrupted sample –School-level means for some grades increased and for other grades decreased (d = ) 14

School-Level Score Differences Does disruption add to the prediction of 2013 school-level means? ● Multiple Regression: –2012 school-level means –Percent of students disrupted –Interaction between 2012 school-level mean and % of students disrupted ● Results: –% of students disrupted and the interaction term were not significant predictors –ΔR² was small 15

School-Level Score Differences Would there be differences in % proficient if predicted scores were used in place of observed scores? 16

Conclusions Student-level Summary: “The evidence shows that there were disruption effects; however, the effects on students’ scores were neither widespread nor large. In addition, the evidence shows that disruptions were not consistently detrimental, but at times were beneficial.” School-Level Summary: “School-level analyses suggest that there may be a small impact on school-level means for schools that experienced disruption, but the direction of the impact is not consistent and adjusting school-level scores based on this data would be beneficial for some schools and detrimental to others.” 17