T HE R OLE OF C ALIBRATION IN A DVANCING F ACULTY L EARNING A BOUT S TUDENT L EARNING Terry Rhodes & Ashley Finley AAC&U Institute on Integrative Learning and the Departments Portland, OR “O UR ” C OMMON G ROUND :

Criteria The Anatomy of a VALUE Rubric Levels Performance Descriptors

T HE C ALIBRATION T RAINING P ROCESS Scoring Steps: Review rubric to familiarize yourself with structure, language, performance levels Ask questions about the rubric for clarification or to get input from others regarding interpretation Read student work sample Connect specific points of evidence in work sample with each criterion at the appropriate performance level (if applicable) Calibration Steps: Review scores Determine common score(s) Hear from outliers Discuss Determine final score

T HE G ROUND R ULES This is not grading. Think globally about student work and about the learning skill. Think beyond specific disciplinary lenses or content. We are not changing the rubric (today). Our work is time sensitive. Go with your instinct. Start with 4 and work backwards. Pick one performance benchmark per criterion. Avoid “.5”. Zero does exist. Assign “0” if work does not meet benchmark (cell one) performance level. N/A exists. Assign “not applicable” if the student work is not intended to meet a particular criterion.

S IGNATURE A SSIGNMENTS Assignment should enable attainment of criteria Break down criteria to determine key components for assignment What should students do with content to meet criteria? E.g. What are the pieces to be analyzed, compared, integrated? Will the assignment be used for more than one outcome? What are the types of assignments that will be most helpful for allowing students to demonstrate competency?

E XAMPLE OF P ROCESS Step 1: All Gen Ed Courses reported as addressing and assessing Info. Tech. Literacy identified as potential courses from which to request artifacts. (54 courses) Step 2: Of courses identified, approx. 20% were randomly selected for sample (10 courses, 36 total sections) Step 5: Artifacts submitted to Director of Learning Outcomes for scoring. (66 artifacts) Step 4: Start of semester, department chairs notified of courses in from which artifacts were to be requested. Chairs worked with individual faculty to fulfill request. Step 3: Within each selected course, 2 students randomly selected by roster # to submit artifacts (74 artifacts) Step 6: Faculty scoring team met at the close of spring semester for a norming session and scoring. (62 artifacts) From: Carroll Community College Flow chart of sequential steps in the request, submission, and scoring of student artifacts for Learning Goal 4: Information and technology literacy.

C AMPUS E XAMPLE OF O UTCOMES A SSESSMENT U SING R UBRIC DATA Dimension% of students who scored 2 or higher % of students who scored 3 of higher Explanation of Issues Interpreting & Analysis Influence of Context and Assumptions Student’s position Conclusions and related outcomes From: UNC-Wilmington, Critical Thinking Rubric

B UILDING THE E VIDENTIARY B ASE : U NIVERSITY OF K ANSAS Percent of Ratings Critical Thinking: Issues, Analysis, and Conclusions Inter-rater reliability = >.8

B UILDING THE E VIDENTIARY B ASE : U NIVERSITY OF K ANSAS Percent of Ratings Critical Thinking: Evaluation of Sources and Evidence

B UILDING THE E VIDENTIARY B ASE : U NIVERSITY OF K ANSAS Percent of Ratings “VALUE added” for 4 years - writing