Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 1 Evaluation of Distributed Scoring with the Texas Assessment of Knowledge and Skills (TAKS) Laurie Laughlin Davis, Leslie Keng, & Shelley Ragland

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 2 Overview of Studies Part of a set of 4 research studies designed to iteratively build on and extend the results of the previous studies to inform a decision by TEA to move to distributed scoring –Study 1: 2008 Online Training Demonstration –Study 2: 2009 Online Training Pilot –Study 3: 2008 Distributed Scoring Research –Study 4: 2009 Distributed Scoring Research in TX context

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 3 Study Features Use of experienced Texas scorers Use of potential “regional” scorers for both regional and distributed scoring conditions –Mitigates influence of scorer demographic characteristics on study outcome 2009 TAKS Grade 11 ELA essay prompt Two populations of Texas students –Online retesters who type their essays –Paper primary testers who hand write their essays

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 4 Study Features 3,999 handwritten primary student essay responses selected to be representative of statewide population 1,291 online retest student essay responses 2 scoring conditions –Online Training with Distributed Scoring –Stand-Up Training with Regional Scoring All essay responses scored through both conditions

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 5 Study Features Both scoring conditions replicated (week 1 and week 2) –Week 1 Regional (41 scorers) –Week 1 Distributed (41 scorers) –Week 2 Regional (37 scorers) –Week 2 Distributed (38 scorers) How different are Distributed and Regional Scoring from each other given the normal variation we might see within a scoring condition due to scorer effects, etc?

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 6 Study Features Use of operational scoring practices to the extent feasible –157 total scorers –Each essay received scores from at least 2 scorers –Resolution and adjudication readings were applied –Not all scorers scored all essays –However, analytics not conducted Evaluation of impact of scoring model to classification decisions

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 7 *Percentages may not sum to 100% due to rounding **Essay scores from study did not include specialists or analytics

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 8 *Percentages may not sum to 100% due to rounding **Essay scores from study did not include specialists or analytics

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 9

10 *Percentages may not sum to 100% due to rounding **Essay scores from study did not include specialists or analytics

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 11 *Percentages may not sum to 100% due to rounding **Essay scores from study did not include specialists or analytics

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 12 Conclusions Perfect agreement across distribution of final essay scores for primary testers Near perfect agreement across distribution of final essay scores for retesters 95-98% consistency of student classification at the total test level between regional and distributed scoring in the study

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 13 Conclusions This study continues to support the use of distributed scoring These results show a high degree of similarity between regional and distributed scoring methods Similarity of results could be even stronger if analytics implemented –Especially true for retesters where higher percentage of students earn essay scores of ‘1’

Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 14 Limitations Study was conducted on a sample of student essay papers (for primary testers) Study duration was short relative to operational scoring projects –Some metrics (i.e. inter-rater reliability, Rate of Scoring, etc.) may be artificially suppressed Scoring process during study was similar to operational context, but did not perfectly replicate it. –i.e. did not include specialists or analytics