Michigan Assessment Consortium Common Assessment Development Series Common Assessment Development Series Rubrics and Scoring Guides

Developed by… Bruce R. Fay, PhD Assessment Consultant Wayne RESA Wayne RESA

Support The Michigan Assessment Consortium professional development series in common assessment development is funded in part by the Michigan Association of Intermediate School Administrators in cooperation with the Michigan Department of Education, Michigan State University, Ingham and Ionia ISDs, Oakland Schools, and Wayne RESA. The Michigan Assessment Consortium professional development series in common assessment development is funded in part by the Michigan Association of Intermediate School Administrators in cooperation with the Michigan Department of Education, Michigan State University, Ingham and Ionia ISDs, Oakland Schools, and Wayne RESA.

What you will learn Why and when you need rubrics Why and when you need rubrics Different kinds of rubrics Different kinds of rubrics How to develop a rubric How to develop a rubric How to use a rubric How to use a rubric What scoring guides are What scoring guides are How to use scoring guides How to use scoring guides

Subjectivity in Scoring No such thing No such thing If it’s truly subjective, it’s just someone’s opinion, and is of little or no value to the person being assessed If it’s truly subjective, it’s just someone’s opinion, and is of little or no value to the person being assessed

The problem is Bias There are sources of bias in all assessment methods There are sources of bias in all assessment methods Some are common to all methods Some are common to all methods Others are unique to each method Others are unique to each method All of them must be minimized in order for assessments with scored items to be fair All of them must be minimized in order for assessments with scored items to be fair

So, what is a rubric? “…guidelines, rules, or principles by which student responses, products, or performances are judged. They describe what to look for in student performances or products to judge quality.” (p. 4) Scoring Rubrics in the Classroom Judith Arter and Jay McTighe Corwin Press

Assessment methods that require a rubric / scoring guide Written response Written response Performance/observation Performance/observation Interactive/conversation Interactive/conversation Portfolio Portfolio

Where Do Rubrics Fit? Classroom or large-scale assessment Classroom or large-scale assessment Free-response written methods Free-response written methods  Short- and extended-response items Performance observation (somewhat) Performance observation (somewhat) Across assessment targets, especially complex, hard to define ones, such as problem-solving, writing, and group processes Across assessment targets, especially complex, hard to define ones, such as problem-solving, writing, and group processes Across content areas and grade levels Across content areas and grade levels

Assessment Myth #1 True or False? Selected-response tests are more objective that free- response tests? True or False? Selected-response tests are more objective that free- response tests? Answer? Answer? False! The only thing truly “objective” about selected-response tests is the scoring. False! The only thing truly “objective” about selected-response tests is the scoring.

Assessment Myth #2 True or False? Scoring (or grading) of free-response items is inherently subjective? True or False? Scoring (or grading) of free-response items is inherently subjective? Answer? Answer? False! On two accounts: False! On two accounts:  1) Scoring and grading are not the same thing  2) “Subjective scoring” isn’t scoring at all, it’s just your opinion

Assessment of Open-ended Work Checklists Checklists Performance lists Performance lists Scoring Rubrics Scoring Rubrics Scoring Guides Scoring Guides

Checklists Simple criteria for simple tasks Simple criteria for simple tasks Checklist format Checklist format Assess presence or absence only Assess presence or absence only No judgment of quality No judgment of quality

Performance Lists More sophisticated than checklists More sophisticated than checklists Criterion-based Criterion-based Product, task, or performance broken down into relatively simple, discrete pieces Product, task, or performance broken down into relatively simple, discrete pieces Each piece scored on a scale Each piece scored on a scale Scale for each piece can be different Scale for each piece can be different

Scoring Rubrics & Guides Judgments regarding complex tasks Judgments regarding complex tasks Written criteria Written criteria Score points defined/described Score points defined/described Represents the essence of quality work Represents the essence of quality work Reflects the best thinking in the field Reflects the best thinking in the field Scoring guides often include exemplars in the form of annotated anchor papers Scoring guides often include exemplars in the form of annotated anchor papers

Benefits of Rubrics for Teaching, Learning, and Assessing Focuses instruction/assessment Focuses instruction/assessment  Clarifies instructional goals  Clarifies assessment targets  Defines quality work Integrates assessment and instruction Integrates assessment and instruction Develops shared/consistent vocabulary and understanding Develops shared/consistent vocabulary and understanding Provides consistency in scoring Provides consistency in scoring  Across students by a scorer  Across multiple scorers  Across time

Rubric Basics Types Types Uses Uses Scoring ranges Scoring ranges

Types of Rubrics Holistic Trait Analytic GenericAB Task Specific CD

USES OF RUBRICS

Holistic Rubrics Strengths Strengths  Provide a quick, overall rating of quality  Judge the “impact” of a product or performance  Use for Summative or large-scale assessment Limitations Limitations  May lack the diagnostic detail needed to  Plan instruction  Allow students to see how to improve  Students may get the same score for vastly different reasons

Trait-Analytic Rubrics Strengths Strengths  Judge aspects of complex work independently  Provide detailed/diagnostic data by trait that can better inform instruction and learning Limitations Limitations  More time consuming to learn and apply  May result in lower inter-rater agreement when multiple scorers are used (without appropriate procedures)

Generic Rubric Strengths Complex skills that generalize across tasks, grades, or content areas Complex skills that generalize across tasks, grades, or content areas Situations where students are doing a similar but not identical task Situations where students are doing a similar but not identical task Help students see “the big picture”, generalize thinking Help students see “the big picture”, generalize thinking Promote/require thinking by the student Promote/require thinking by the student Allow for creative or unanticipated responses Allow for creative or unanticipated responses Can’t give away the answer ahead of time Can’t give away the answer ahead of time More consistency with multiple raters (only one rubric to learn, so you can learn it well) More consistency with multiple raters (only one rubric to learn, so you can learn it well)

Generic Rubrics Limitations Difficult to develop and validate Difficult to develop and validate Takes time and practice to learn, internalize, and apply consistently Takes time and practice to learn, internalize, and apply consistently Takes time to apply Takes time to apply Takes discipline to apply correctly Takes discipline to apply correctly Requires a scoring procedure to ensure consistent scores when multiple raters are involved Requires a scoring procedure to ensure consistent scores when multiple raters are involved

Task-specific Rubric Stengths Specialized tasks Specialized tasks Highly structured assignments Highly structured assignments Specific/detailed assessment goals Specific/detailed assessment goals Provide detailed feedback to student on work Provide detailed feedback to student on work Situations requiring quick but highly consistent scoring from multiple scorers with less training and/or inter-rater control procedures Situations requiring quick but highly consistent scoring from multiple scorers with less training and/or inter-rater control procedures

Task-specific Rubric Limitations Can’t show to students ahead of time as they give away the answer Can’t show to students ahead of time as they give away the answer Does not allow the student to see what quality looks like ahead of time Does not allow the student to see what quality looks like ahead of time Need a new rubric for each task Need a new rubric for each task Rater on autopilot may miss correct answers not explicitly shown in the rubric Rater on autopilot may miss correct answers not explicitly shown in the rubric

Scoring Ranges Minimum of 3 levels Minimum of 3 levels Maximum of 3 to 7 levels (typically, beyond 8 it’s hard to apply and understand) Maximum of 3 to 7 levels (typically, beyond 8 it’s hard to apply and understand) Even vs. Odd – Odd point scales (3, 5, 7, etc.) allow a middle ground that is psychologically attractive for the rater (which you may want to avoid) Even vs. Odd – Odd point scales (3, 5, 7, etc.) allow a middle ground that is psychologically attractive for the rater (which you may want to avoid) 5-point scales tend to look like A-F grading scheme (which you may also want to avoid) 5-point scales tend to look like A-F grading scheme (which you may also want to avoid)

Distinguish Quality 4 or more points typically needed to distinguish levels of quality 4 or more points typically needed to distinguish levels of quality 4 – 7 points is typical 4 – 7 points is typical Depends on being able to distinguish levels of quality Depends on being able to distinguish levels of quality The more open-ended/complex the task, the broader the range of points needed The more open-ended/complex the task, the broader the range of points needed

A Meta-rubric

A rubric for evaluating rubrics Trait 1: content/coverage Trait 1: content/coverage Trait 2: clarity/detail Trait 2: clarity/detail Trait 3: usability Trait 3: usability Trait 4: technical quality Trait 4: technical quality

MRT 1: Content/coverage Aligned to curriculum and instruction Aligned to curriculum and instruction Includes everything that is quality Includes everything that is quality Does not include trivial things Does not include trivial things Reasonable explanations for what is included and excluded Reasonable explanations for what is included and excluded Reflects best thinking and practice Reflects best thinking and practice Rarely find work that can’t be scored Rarely find work that can’t be scored

MRT 2: Clarity/detail Different users likely to interpret the rubric in the same way – language is not ambiguous, vague, or contradictory Different users likely to interpret the rubric in the same way – language is not ambiguous, vague, or contradictory Use of rubric supports consistent scoring across students, teachers, and time Use of rubric supports consistent scoring across students, teachers, and time Examples of student work illustrate each level of quality on each trait Examples of student work illustrate each level of quality on each trait

MRT 3: Usability/Practicality Can be applied in a reasonable amount of time when scoring Can be applied in a reasonable amount of time when scoring Can easily explain/justify why a particular score was assigned Can easily explain/justify why a particular score was assigned Student can see what to do differently next time to earn a better score Student can see what to do differently next time to earn a better score Teacher can see how to alter instruction for greater student achievement Teacher can see how to alter instruction for greater student achievement

MRT Trait 4: Technical Quality Evidence of reliability (consistency) – across students, teachers, and time Evidence of reliability (consistency) – across students, teachers, and time Evidence for validity (appropriateness) – students and teachers agree that it supports teaching and learning when used as intended Evidence for validity (appropriateness) – students and teachers agree that it supports teaching and learning when used as intended Evidence of fairness and lack of bias – does not place any group at a disadvantage because of the way the rubric is worded or applied Evidence of fairness and lack of bias – does not place any group at a disadvantage because of the way the rubric is worded or applied

Develop Your Own Rubrics Form a learning team Form a learning team Locate/acquire additional resources Locate/acquire additional resources Modify existing rubrics Modify existing rubrics Understand the development process Understand the development process Help each other out Help each other out When you are comfortable with the process, introduce it to your students When you are comfortable with the process, introduce it to your students

The Meta-rubric outline WOWMostSomeNone Trait 1 Trait 2 Trait 3 Trait 4

Possible Development Process Gather samples of student work Gather samples of student work Sort student work into groups and write down the reasons for how it is sorted Sort student work into groups and write down the reasons for how it is sorted Cluster the reasons into traits Cluster the reasons into traits Write a value-neutral definition of each trait Write a value-neutral definition of each trait Find samples of student work that illustrate a possible range of score points on each trait Find samples of student work that illustrate a possible range of score points on each trait Write value-neutral descriptions of each score level for each trait, if appropriate Write value-neutral descriptions of each score level for each trait, if appropriate Evaluate your rubric using the Meta-rubric Evaluate your rubric using the Meta-rubric Test it out and revise it as needed Test it out and revise it as needed

Acknowledgments This module is based on material adapted from: Scoring Rubrics in the Classroom By Judith Arter and Jay McTighe Experts in Assessment Series Corwin Press, Thousand Oaks, CA and Material provided by Edward Roeber of Michigan State University, East Lansing, Michigan