Writing and scoring test questions based on Ohio’s Academic Content Standards Scoring Welcome and Introductions 1

Writing and scoring test questions based on Ohio’s Academic Content Standards Agenda Large Group Session: 8:00 – 9:00 a.m. –Objectives –Handscoring Overview Content-area Break-out Sessions –Modeling Activities –Try-it Exercises –Wrap-up 2

Writing and scoring test questions based on Ohio’s Academic Content Standards Objectives By the end of workshop, participants will be able to: –Recognize the value of utilizing open-ended items for evaluating student understanding –Apply rubrics and scoring guides to unscored responses in a reliable manner –Utilize rubrics and scoring guides to establish the spectrum of scores and select exemplars from an unscored pool of student responses 3

Writing and scoring test questions based on Ohio’s Academic Content Standards Handscoring Overview 1.Item Development 2.Field Testing 3.Rangefinding 4.Creating Training Materials 5.Training and Qualifying Scorers 6.Monitoring Scoring 4

Writing and scoring test questions based on Ohio’s Academic Content Standards Handscoring Overview: Rangefinding 5

Writing and scoring test questions based on Ohio’s Academic Content Standards Handscoring Overview: Rangefinding Goal – teacher input on how to score DRC selects responses Committees meet by content area Teachers’ consensus score responses DRC and ODE facilitate discussion and take notes After rangefinding – DRC and ODE organize responses into training sets 6

Writing and scoring test questions based on Ohio’s Academic Content Standards Handscoring Overview: Train and Qualify Scorers Rubric – define the score points Anchor set – exemplify the score points Training sets – practice scoring Qualifying sets – demonstrate accuracy 7

Writing and scoring test questions based on Ohio’s Academic Content Standards Handscoring Overview: Monitoring Scoring Inter-rater reliability Score point distribution Validity sets Team Leader read-behinds 8

Writing and scoring test questions based on Ohio’s Academic Content Standards Break Time We will reconvene in 15 minutes in content-area specific small group sessions 9

Writing and scoring test questions based on Ohio’s Academic Content Standards Breakout Activity 1 Establish the spectrum of scores Select an item Study the Scoring Guide Model consensus scoring (10 responses) Discuss results 10

Writing and scoring test questions based on Ohio’s Academic Content Standards Breakout Activity 2 Model a training session Continue with selected item 10-paper set –Individually read and score –Review responses as a group 5-paper set –Score responses one-by-one –Use flash cards –Discuss response Discuss the results 11

Writing and scoring test questions based on Ohio’s Academic Content Standards Breakout Activity 3 Establishing the spectrum of scores Sit at tables by grade-band Distribute scoring guides specific to each grade- band Each table will –Review the Scoring Guide –Select a Table Leader –Consensus score 10 responses –Identify responses as exemplars Report back on the experience 12

Writing and scoring test questions based on Ohio’s Academic Content Standards Breakout Activity 4 Do training exercise Distribute training sets specific to each grade-band Score responses one-by-one Use flash cards to monitor agreement Discuss the responses Report back on the experience 13

Writing and scoring test questions based on Ohio’s Academic Content Standards Breakout Activity 5 Time permitting! Establish the spectrum of scores Sit at tables by grade-band Distribute scoring guides specific to each grade- band Each table will –Review the Scoring Guide –Select a Table Leader –Consensus score 10 responses –Identify responses as exemplars Report back on the experience 14

Writing and scoring test questions based on Ohio’s Academic Content Standards Breakout Activity 6 Time Permitting! Do training exercise Distribute training sets specific to each grade-band Score responses one-by-one Use flash cards to monitor agreement Discuss the responses Report back on the experience 15

Writing and scoring test questions based on Ohio’s Academic Content Standards What Does This Mean to Me? Overall takeaways Classroom/School/District Implications 16

Writing and scoring test questions based on Ohio’s Academic Content Standards Review Objectives Recognize the value of utilizing open- ended items for evaluating student understanding Apply rubrics and scoring guides to unscored responses in a reliable manner Utilize rubrics and scoring guides to establish the spectrum of scores and select exemplars from an unscored pool of student responses 17

Writing and scoring test questions based on Ohio’s Academic Content Standards Thank you! 18