Scoring Rubrics

The Outcomes To describe scoring rubrics (SR) To explain the usefulness of SR To differentiate types of SR To discuss process of developing SR

What is Scoring Rubrics? A descriptive scoring scheme developed by assessors to guide the analysis of the products or processes of students’ efforts (Brookhart, 1999). Basically for performance assessment. Usually in a matrix form. Similar to Likert scales. It contains a mutually agreed upon negotiated contract or criteria for success.

What is Scoring Rubrics? Employed for a judgment of quality. Used to assess a broad range of subjects and activities. It focuses on stated objectives and the educational standards. As active participants, students are aware of the standards. The assessment is transparent, and fair.

Example 1: The Cookie Task: Make a chocolate chip cookie that I would want to eat. Criteria: Texture, Taste, Number of Chocolate Chips, Richness Range of performance: Delicious(14-16 pts) Tasty(11-13 pts) Edible(8-10 pts) Not yet edible(0-7 pts)

Assess The Cookie Overall score By criteria Delicious Tasty Edible Not yet edible By criteria Number of chips Texture Taste Richness

The Rubric Delicious 4 Tasty 3 Edible 2 Not yet edible 1 # chips Chips in every bite 75% chips 50% chips Less than 50% chips texture Consistently chewy Chewy middle, crispy edges Crunchy Like a dog biscuit color Even golden brown Brown with pale center All brown Or all pale Burned richness Buttery, high fat Medium fat Low-fat flavor Non-fat flavor

Holistic Example Cookie with Delicious level (4) Chips in every bite Consistently chewy Even golden brown Buttery, high fat

Example of Scoring Rubrics An assessment of writing sample Criteria: linguistic structure and/or persuasiveness of the argument? Four score levels; Meet Expectation, Adequate, Needs Improvement, and Inadequate Each describes the characteristics of a response that would receive the respective score. The description will increase raters’ reliability. The description helps students understand the basis for their scores and what they need to do to improve their future performances.

SR for Report Writing L4: Meet Expectation L3: Adequate Easily followed Effective transitions Professional format Supportive graphics explained Appropriate grammar Clear and concise L3: Adequate Basic transitions Structured format Supportive graphics not clearly explained Minimal distractions on flow, presentation, grammar L2: Needs Improvement Difficult to follow Inadequate transitions Rambling format Insufficient/irrelevant information Ambiguous graphics Numerous distractions L1: Inadequate No organization Difficult to read and understand

Learning Outcomes Cognitive Affective Psychomotor Social

Bloom’s Cognitive Domain Level Learning Outcomes Evaluation Making judgment based on a pre-established set of criteria Synthesis Producing something new or original from component parts Analysis See interrelationships/ hierarchy of ideas Application Using a concept or principle to solve a problem Comprehension Explaining/interpreting the meaning of material Knowledge Remembering facts, terms, concepts, definitions, principles

Krathwohl’s Affective Domain Level Learning Outcomes Characterization Behave in a way that is consistent with the values being adopted Organization Organizing ideas into a value system Valuing Judge & express commitment to the idea Responding Active participation in the activity Receiving Awareness of stimulus activity

Harrow’s Psychomotor Domain Level Learning Outcomes Non-discursive communication Communication through movement e.g. Posture nonverbal, expressive & interpretive movement Skilled Movements The result of learning, often complex learning, resulting in efficiency in carrying complex task Physical Abilities Individual’s physical self, enable smooth and efficient movements, e.g. endurance, strength Perceptual Abilities All the abilities that send input to the brain for interpretation, which affects motor movements Basic-Fundamental Movements Inherent in more complex or skilled motor movements e.g. manipulative movements Reflex Movements Involuntary movements, evident at birth or develop with maturation

Simpson’s Psychomotor Domain Category Learning Outcomes Perception The ability to use sensory cues to guide motor activity. This ranges from sensory stimulation, through cue selection, to translation Set Readiness to act. Includes mental, physical, and emotional sets. These three sets are dispositions that predetermine a person’s response to different situations – sometimes called mindsets. Guided Response The early stages in learning a complex skill that includes imitation and trial and error. Adequacy of performance is achieved by practicing. Mechanism The intermediate stage in learning a complex skill. Learned responses have become habitual and the movements can be performed with some confidence and proficiency

Simpson’s Psychomotor Domain Category Learning Outcomes Complex Overt Response The skillful performance of motor acts that involve complex movement patterns. Proficiency is indicated by a quick, accurate, and highly coordinated performance, requiring a minimum of energy. Includes performing without hesitation, and automatic performance. E.g. Players utter sounds of satisfaction (grunts) as soon as they hit a ball. Adaptation Skills are well developed and the individual modify movement patterns to fit special requirements. Responds effectively to unexpected experiences. Origination Creating new movement patterns to fit a particular situation or specific problem. Learning outcomes emphasize creativity based upon highly developed skills. Constructs a new theory.

Social Domain (Leise, 2004) Category Learning Outcomes Communication Receiving, preparing, delivering a message Teamwork Involves relating culture by accepting constraints, living in society, and demonstrating cultural competence. It also involves relating with others by inviting interaction, relating for meaning, performing a team and performing in an organization Management Managing people, building and maintaining teams, managing communication, and managing resources Leadership Envisioning, building a following, maintaining commitment, and empowering

Are Scoring Rubrics Appropriate? The purpose of evaluation: Learning Instruction Grading Selection Placement Counseling & Guidance Program or Curriculum Administrative Policy

Benefits of SR Support the “Examination” - ensured that the specified criteria has been reached. Establish clear benchmarks for success. Provide feedback on how to improve students performances. Because they are aware of the standards.

Types of SR Analytic versus Holistic General versus Task-specific

Analytic vs Holistic Analytic: Like a checklist, allows for a separate assessment of each of the factors – the criteria Example, SR on quality of written sample based on linguistic structure If our objective is for persuasiveness of the argument, a separate set of criteria is needed Each criterion is scored on a different descriptive scale

Analytic vs Holistic Holistic: Used when there is an overlap between the criteria set for the different factors The criteria are considered together on a single descriptive scale Holistic SR supports broader judgment concerning the quality of the process or product

General vs Task-Specific General SR: For the assessment of a broader tasks category E.g. If the purpose is to develop a student’s oral communication skills, a general SR is used to evaluate each of the given oral presentations The student will use the feedback to improve their next presentation

General vs Task-Specific Used to evaluate student’s performances on a single assessment event E.g. Evaluating student’s knowledge on a different historical event

SR may contain both general and task-specific An analytic SR that contains both general and task specific components could be designed to evaluate students’ oral presentation skills and their knowledge of the historical event being discussed.

Developing SR 1. Identify the qualities – the criteria - that need to be displayed. Refers to the learning outcomes 2. Define the criteria for lowest and top-level performance; resulting in three score levels 3. If greater distinctions are desired, comparisons can be made between the criteria for each existing score level, till the desired number of score levels is reached or no further meaningful distinctions can be made.

Developing SC 4. Each score category should be defined using descriptions of the work (quantifiable) rather than judgments about the work – “Student’s grammar contain no errors” is preferable over “Student’s grammar are good” 5. Ask other evaluators to use the SR to evaluate a subset of student responses; inter-raters reliability 6. Differences will suggest how the rubric may be further clarified

Thank You