Scoring Open-Ended Literacy Items

Goals for the Session Understand the importance of open-ended literacy items Analyze prompts and rubrics for these items Scoring tips Relating Scoring to Classroom Instruction Practice scoring open-response reading questions Practice scoring writing samples Share best practices

Importance of Open-Ended Literacy Items Reading Open-ended Questions ◦1 Open Response question for each passage ◦Total of 3 for ACTAAP tests grades 3-8 ◦Total of 6 for Grade 11 Literacy Exam ◦Account for 50% of score (24 points out of 48 for grades 3-8)

Importance of Open-Ended Literacy Items Writing Open-ended Tests ◦2 questions (prompts) requiring essay- length responses ◦Account for 40 out of 48 points ◦That is 83% of total writing score in grades 3-8

Importance of Open-ended Literacy Items Open-ended questions account for 67% of the total literacy score Students need to earn significant points in these items in order to achieve a proficient score It’s critical that we score these items accurately so that the testing experience and data will be relevant in helping students improve

Few Similarities Reading open response questions and writing prompts have little in common Frustrating because teachers want students to express themselves well in all written responses Difficult for teachers and students to accommodate the differences

Reasons to Score The two reasons to score open-ended items are often at cross-purposes 1.To upload scores quickly to get data back  Scoring should be as accurate to the ACTAAP scoring as possible  With practice this can be done with a minimum of marks and time 2.To give meaningful feedback to students and parents  This scoring is more detailed, in-depth, and takes much longer

Scoring Decisions District, or at least building, must decide whether to score interim assessments for quick, accurate scores to be uploaded for data purposes only or for meaningful in- depth scoring for feedback and reteaching To do both will require delayed uploading, double scoring, or some creative time use

Scoring Requirements Writing: 5 domains on the state scoring rubric are used to assess writing. Content Style Sentence Formation Usage Mechanics Can the student express himself in written form? Reading: Content-specific rubric is used to assess the reading open response question. Scored for content only Did the student read and understand?

Reading Open-ended Items

Open Response Reading Questions Each reading passage has 1 open response question The question will generally require more than one thinking skill, even at the lower grades: ◦Analysis ◦Inference ◦Evaluation ◦Comparison

Open Response Reading Questions Question is tied to the passage and requires information from the passage as support It is a more in-depth question than can be covered by a multiple-choice format question. Short answer Scored on a 0 to 4 scale TLI continues to use a B for Blank

Examples of Open Response Questions Based on the information that you read in the passage, describe how Molly and her mother used Molly’s old crayons to make new crayons. Then, describe Molly’s new crayons. Include details from the passage.  Grade 3, 2006

Examples of Open Response Questions Use information from the passage to describe two ways in which Jesse Owens influenced people. Be sure to use details from the passage to support each of your ideas or conclusions.  Grade 5, 2006

Examples of Open Response Questions Compare and contrast Sylamore Horse Trail and Ozark Bicycle Trail. Use specific examples from the passage to support your response.  Grade 11, 2006

Rubrics Each open response reading question is scored by a specific rubric ◦Can be subjective  Was the response thorough or only adequate? ◦A variety of tasks requested

Analysis of Open Response Reading Questions Based on the information that you read in the passage, describe how Molly and her mother used Molly’s old crayons to make new crayons. Then, describe Molly’s new crayons. Include details from the passage. What does “describe” really mean? How much description is enough? How many details do we include?

Analysis of Open Response Reading Questions Use information from the passage to describe two ways in which Jesse Owens influenced people. Be sure to use details from the passage to support each of your ideas or conclusions. Describe or tell? How many details should support each idea? What conclusions?

Sample Rubric Score 4: The response clearly and thoroughly describes at least two ways in which Jesse Owens influenced people and supports each with relevant information from the passage. The response demonstrates comprehensive understanding of the task.

Sample Rubric Score 4: The response clearly and thoroughly describes at least two ways in which Jesse Owens influenced people and supports each with relevant information from the passage. The response demonstrates comprehensive understanding of the task.

New Open Response Format With the 2007 test a new format was begun by the state Old format is still in use Most questions in 2007 and 2008 did follow the new format Continue to use old format in practice TLI now uses the 4 point format

New 4-point Open Response Format These questions are written so that there are 4 tasks to perform The rubrics show score points equal to the number of tasks correctly performed 4—all 4 tasks are completed correctly 3—any 3 of the tasks are completed correctly 2—any 2 of the tasks are completed

Analysis of New Format Example Explain why it is important to guard against germs in the kitchen. Identify three ways to fight against infectious diseases in relation to food and its preparation. ◦Explain why ◦Identify way #1 ◦Identify way #2 ◦Identify way #3

Variations in Rubrics Most rubrics seem to accept any 3 tasks to score a 3, any 2 tasks for a 2 Occasionally a rubric will be specific about the tasks. ◦For a 4, the response explains and gives 3 examples ◦For a 3, the response must explain and give 2 examples (3 examples without explanation may not be acceptable)

Sample Open Response Items

OR Short Answers Encouraged Fragments OK Charts or Diagrams OK Lists OK “Don’t be creative.” --ADE Just answer the questions Some questions have parts that should be labeled. Generally at lower grades. Some questions have charts. The missing information should be listed in the answer box.

Respond to Match the Question Some questions are difficult to answer with lists If the question suggests discussion, sentences may be a better approach Complete paragraph construction is never necessary Scoring should never take anything into consideration except what is specifically called for in the rubric (either positively or negatively)

Scoring Tips for Open Response Items Scorers should be very familiar with the passage and the question Scorers should work the “problem” themselves, preferably more than 1 way TLI provides possible answer information Follow the rubric exactly, even if the response is unexpected. Give credit if tasks are accurately performed.

Scoring Tips for Open Response Items Scan the response, looking for answer information Put a check-mark in the margin for each task performed Tasks do not have to be placed in the order requested If question divides answer and requests labeling, no points should be removed if student does not label

Scoring Tips for Open Response Items Score of B—blank, no attempt to answer Pull these from the stack and score ◦Analyze why  Did student run out of time?  Did student waste time?  Was student unaware of question?  Did student refuse to attempt?  Does this reveal a hidden problem?

Score 0—Student attempted an answer but earned no points Analyze why ◦Ran out of time and jotted something down ◦Did not understand the question ◦Was uncertain of format

For all responses that earn points, look quickly at why points were not earned ◦Only 3 tasks performed? Why not the other? ◦4 tasks performed, but some incorrectly? ◦What could student do to earn just one more point?

Grade 3 Question For Blaise, one of the best things about bobsledding is going fast. Give three details from the passage that describe bobsledding as a fast sport. Not a four-point format Question includes important reading clues What skill is this testing?

Grade 3 Sample Response

Score and share

More OR Scoring Tips Do you know that open response question and rubric templates are on the TLI Web site?

The Scoring-Teaching Connection DO teach students to analyze the Open Response prompt DO teach the 4 types of Open Response Skills Do expect “analysis” to be the underlying skill DO expect 4 tasks on each question DO teach students to score their own work—great practice DON’T score for mechanics, sentence formation, and the like

The Scoring-Teaching Connection DO teach students to complete tasks in order DO encourage labeling/numbering tasks DO work with students to find ways to increase the score by 1 point DO encourage students to look for clues within the question DO encourage students to mark the passage to locate answer information for the question

The Scoring-Teaching Connection DO teach students to anticipate an OR question DO teach students to write their own OR questions and rubrics DO help students understand the difference between the reading open response and a writing assignment: Short box= short answer= score for content only

The Scoring-Teaching Connection DO teach students to consider their OR answers to be “prewriting;” do not need to develop the answer past this stage DO teach students to use information from the passage; prior knowledge is rarely requested DO encourage reading the questions first with students for whom that might work ◦Search for answers as they read the text

Tips to Share