Creating Analytic Rubrics April 27, 2017 Cathy Sanders Director of Assessment Office of Assessment & Accreditation

Workshop Objectives Introduce you to analytic and holistic rubrics Go through steps in creating an analytic rubric Provide you with the opportunity to create a simple analytic rubric Explain how to use an analytic rubric in conjunction with SLO assessment data

What is a Rubric? A scoring tool that uses a set of evaluation criteria that are directly tied to the student learning outcome to assess student learning Used with subjective assessments Promotes inter-rater reliability When the content of the rubric is communicated prior to students completing the work, the grading process is very clear and transparent to everyone involved.

Two Types of Rubrics Analytic rubric provides cell descriptors for each evaluation criterion for each level of performance. Holistic rubric provides 1 cell descriptor for all evaluation criteria for each level of performance. UNC Charlotte uses analytic rubrics More useful in pinpointing areas of strengths and weaknesses in student learning Aids in deciding needed improvements in student learning

Sample Analytic Rubric Evaluation Criteria 1 = Needs Improvement 2 = Acceptable 3 = Accomplished Criterion Score Ability to develop a rubric scale Is unable to develop a minimum of a 3 point scale that progresses from least proficient to most proficient Is able to develop a minimum of a 3 point scale that progresses from least proficient to most proficient Is able to develop a scale that is greater than 3 points that progresses from least proficient to most proficient 3 Ability to develop evaluation criteria Is unable to identify at least 3 relevant evaluation criteria Is able to identify at least 3 relevant evaluation criteria Is able to identify more than 3 relevant evaluation criteria 2 Ability to develop cell descriptors   Some of the cell descriptors are missing and cell descriptors(s) present do not clearly differentiate the levels of proficiency. Cell descriptors are present for all evaluation criteria but some do not clearly differentiate the levels of proficiency. Cell descriptors are present for all evaluation criteria that clearly differentiate the levels of proficiency. 1

Elements of an Analytic Rubric Rubric Scale Evaluation Criteria Cell Descriptors Criterion Score

Sample Analytic Rubric Evaluation Criteria 1 = Needs Improvement 2 = Acceptable 3 = Accomplished Criterion Score Ability to develop a rubric scale Is unable to develop a minimum of a 3 point scale that progresses from least proficient to most proficient Is able to develop a minimum of a 3 point scale that progresses from least proficient to most proficient Is able to develop a scale that is greater than 3 points that progresses from least proficient to most proficient 3 Ability to develop evaluation criteria Is unable to identify at least 3 relevant evaluation criteria Is able to identify at least 3 relevant evaluation criteria Is able to identify more than 3 relevant evaluation criteria 2 Ability to develop cell descriptors   Some of the cell descriptors are missing and cell descriptors(s) present do not clearly differentiate the levels of proficiency. Cell descriptors are present for all evaluation criteria but some do not clearly differentiate the levels of proficiency. Cell descriptors are present for all evaluation criteria that clearly differentiate the levels of proficiency. 1

Prior to Creating an Analytic Rubric Develop the Student Learning Outcome (SLO) statement The knowledge, skills, abilities, behaviors, and dispositions that students will demonstrate as a result of the educational program Decide the tasks that students will perform and the artifact they will create to demonstrate proficiency on the SLO Identify the criteria that will be used to evaluate proficiency on the SLO

Examples Students will demonstrate competence in programming skills. Students will demonstrate oral communication skills Artifact: An oral presentation Criteria: Body language, eye contact, introduction and closure, pacing, poise, voice, and organization. Artifact: A programming project Criteria: Understands requirements, uses appropriate data structures, uses appropriate algorithms, modularizes solution, designs appropriate user interface, tests program for correctness, and documents program

Steps in Creating an Analytic Rubric Determine a progressive scale and insert it in the columns across the top of the rubric template Insert the evaluation criteria in the rows down the left side of the rubric template At the intersection of each row and column, insert a description of what proficiency on each criterion at each scale level looks like It may be easier to first describe what proficiency at the highest and lowest scale levels look like Use a meta-rubric to evaluate the rubric. Norm the rubric: multiple scorers use the rubric to score student artifacts. Revise the cell evaluation criteria and/or cell descriptors to reflect needed changes

Workshop Exercise: Creating an Analytic Rubric

Instructions for Analytic Rubric Exercise Develop an analytic rubric for an oral presentation that includes: a 3-point rubric scale 3 evaluation criteria that you feel are the most important cell descriptors for each criterion at all 3 scale levels (a total of 9 cell descriptors)

SLO Assessment Plan Template Student Learning Outcome 1: the knowledge, skill, ability, behavior, or disposition that students will demonstrate proficiency in. Effectiveness Measure: Student artifact that will be used to gauge acquisition of the student learning outcome. Methodology: A description of how, what, when, and where… Performance Outcome: the percentage of students that will demonstrate proficiency and the level of proficiency expected.

Using the Analytic Rubric with Assessment Data Identify SLOs on which students did not meet performance expectations Compare assessment data with the Performance Outcome Example: PO states “ 80% of students will score a 2 “Acceptable” or higher on each of the rubric criteria” but only 75% did. That is an unmet SLO Look at the average criterion scores on the rubric associated with the unmet SLO Identify the evaluation criteria on which students did not meet the Performance Outcome (e.g., did not score 2 “Acceptable” or higher) Those are the criteria to focus improvement efforts on

Analytic Rubric 3 2 1 1 = Needs Improvement 2 = Acceptable  Criteria 1 = Needs Improvement 2 = Acceptable 3 = Accomplished Average Criterion Score Ability to develop a rubric scale Is unable to develop a minimum of a 3 point scale that progresses from least proficient to most proficient Is able to develop a minimum of a 3 point scale that progresses from least proficient to most proficient Is able to develop a scale that is greater than 3 points that progresses from least proficient to most proficient 3 Ability to develop evaluation criteria Is unable to identify at least 3 relevant evaluation criteria Is able to identify at least 3 relevant evaluation criteria Is able to identify more than 3 relevant evaluation criteria 2 Ability to develop cell descriptors   Some of the cell descriptors are missing and cell descriptors(s) present do not clearly differentiate the levels of proficiency. Cell descriptors are present for all evaluation criteria but some do not clearly differentiate the levels of proficiency. Cell descriptors are present for all evaluation criteria that clearly differentiate the levels of proficiency. 1

Collaborative Discussion by Program Faculty After Problem Areas of SLO Have Been Identified how the SLO is taught whether there is sufficient time to teach the SLO how assignments lead to acquisition of the SLO if assignments need to be revised or replaced if additional assignments are needed to reinforce the learning if a different assessment is needed (to achieve better alignment between what is being taught and what is being assessed).

Questions