Developing and Testing a Rubric: Comprehensive Exams Christine Savi, PhD

Assessing Student Learning Assessment vs. Evaluation Forms and Tools Examination Spotlight – Question dissection – Answer criteria Authentic Assessment – Inter-rater reliability Handouts Reflection/Summation

Assessment vs. Evaluation Assessment – Focuses on learning, teaching, and outcomes – Provides information for improving learning and teaching. – An interactive process between students and faculty that informs faculty how well their students are learning what they are teaching. Evaluation – Focuses on grades or scores. – May reflect classroom components other than course content and mastery level.

Forms of Assessment Case studies Direct observation Examinations – Multiple Choice – Short answer – Essay – Oral questioning Knowledge surveys Simulations Portfolios Presentations Self assessments Performance projects

Spotlight: Graduate Examinations Short answer Essay (vignettes, scenarios, case studies) Performance (demonstration, presentation)

Dissecting the Scenario/Essay Question 1. Identify the ‘what’ you want the students to know in each question: - Underline the key words/concepts: only nouns and/or adjectives A report from Parkland Hospital identifies a rise in admissions and subsequent limb amputation in one zip code in Dallas. As the newly appointed health educator for the city of Dallas, you have been asked to develop a new diabetes program for middle aged African American women in a housing development within that zip code. Please answer the following questions using correct grammar and spelling: 1. Develop a logic model that would describe the emphasis of the program. a. Do we need to revise the question/s at all? Then, identify key words? 1. Develop a logic framework using the essential components that would describe the emphasis of the program.

List Descriptors – Question #1 2. List what these key words refer to in each group Descriptors of your key words: Question 1: What are the Essential Components of a Logic Framework? a. Purpose b. Resources c. Activities d. Results e. Short-term outcomes f. Long-term outcomes

Characterize IDEAL Performance for Each Question Using 1 sheet of white paper, divide into 3 vertical sections In the ‘3 rd or last’ column, list the qualities that need to be displayed in a student's work to demonstrate ideal performance – The student was able to develop a logic framework using 5-6 the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

Characterize MINIMAL Performance In the ‘1 st or Beginning’ column, list the qualities that need to be displayed in a student's work to demonstrate minimal or ‘best of the worst’ performance – The student was only able to develop a logic framework using 1-2 the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

Characterize ACCEPTABLE Performance In the ‘2 nd or middle’ column, list the qualities that need to be displayed in a student's work to demonstrate acceptable performance – The student was able to develop a logic framework using 3-4 of the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

3-scale Leveling Content ComponentDescriptors of key concepts Beginning 1 Developing 2 Exemplary 3 Component #1 Develop a logic framework using the essential components to describe the goal of the program. Logic Framework Essential Components Purpose Resources Activities Results Short-term outcomes Long-term outcomes The student was only able to develop a logic framework using 1-2 the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes The student was able to develop a logic framework using 3-4 of the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes The student was able to develop a logic framework using 5-6 the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

Break

Cookie Survey 1.Eat or taste the cookie. 2.Complete the survey. DeliciousGoodNeeds ImprovementPoor Number of Chips Texture Color Taste Richness

Cookie Rubric DeliciousGoodNeeds ImprovementPoor Number of Chips Chocolate chip in every bite Chips in about 75% of bites Chocolate in 50% of bites Too few or too many chips TextureChewy Chewy in middle, crisp on edges Texture either crispy/crunchy or 50% uncooked Texture resembles a dog biscuit ColorGolden brown Either light from overcooking or light from being 25% raw Either dark brown from overcooking or light from undercooking Burned TasteHome-baked taste Quality store-bought taste Tasteless Store-bought flavor, preservative aftertaste – stale, hard, chalky Richness Rich, creamy, high- fat flavor Medium fat contentsLow-fat contentsNonfat contents

Assessing Learning Rubric: – A scoring tool listing criteria for varying levels of performance on a task A series of narrative statements describing the levels of quality of a product/performance Defines qualitative differences in levels of targeted performance Contains levels which apply to indicators for each strategy – For each indicator, there is a brief written description of the different levels of performance based on performance criteria. – Raters use descriptions to determine the level of accomplishment on each indicator, e.g.: » Ratings of “Emerging”, “Developing”, and “Accomplished”

Why Use Rubrics? Used to: – Define quality within the area being rated – Articulate same target goals for improvement for everyone (consistency) – Track change or improvement over time – Provide a common set of definitions across all outcomes – Lead self assessment and planning

Authentic Assessment “Task-based assessment scored by a rubric”, Facione, 2010 Rubric types: – Analytic Targets > 1 content area; separate evaluation of multiple factors Used to identify and assess components of a finished product – Holistic Targets 1 area Used to assess student work as a whole

Analytic & Holistic Examples Analytic Rubric Criteria4 - Accomplished 3 – Developing 2 - Emerging 1 - Beginning PlotBoth plot parts are fully developed. One of the plot parts is fully developed and the less developed part is at least addressed. Both plot parts are addressed but not fully developed. Neither plot parts are fully developed. SettingBoth setting parts are fully developed One of the setting parts is fully developed and the less developed part is at least addressed. Both setting parts of the story are addressed but not fully developed. Neither setting parts are developed. Characte rs Main characters are fully developed with much descriptive detail. Reader has a vivid image of the characters. Main characters are developed with some descriptive detail. Reader has a vague idea of the characters. Main characters are identified by name only. None of the characters are developed or named. Holistic Rubric Fiction Writing Content Rubric 5The plot, setting, and characters are developed fully and organized well. The who, what, where, when, and why are explained using interesting language and sufficient detail. 4Most parts of the story mentioned in a score of 5 above are developed and organized well. A couple of aspects may need to be more fully or more interestingly developed. 3Some aspects of the story are developed and organized well, but not as much detail or organization is expressed as in a score of 4. 2A few parts of the story are developed somewhat. Organization and language usage need improvement. 1Parts of the story are addressed without attention to detail or organization.

3-scale Rubric Content ComponentDescriptors of key concepts Beginning 1 Developing 2 Exemplary 3 Component #1 Develop a logic framework using the essential components to describe the goal of the program. Logic Framework Essential Components Purpose Resources Activities Results Short-term outcomes Long-term outcomes The student was only able to develop a logic framework using 1-2 the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes The student was able to develop a logic framework using 3-4 of the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes The student was able to develop a logic framework using 5-6 the essential components related to the given scenario. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

Rubric Examples Varied scales, uses, descriptions – Dimension generation aide (If/Then scenarios): Compare, contrast, compliment – Open Ed Practices Resources – Presentation/Decision Making rubrics: html html

Inter-rater Reliability = Consistency Ensure all grading participants agree with rubric Verify using sample papers from pilot group Review and/or revise rubric from pilot response Test/revise for proper maintenance

Use the rubric below to evaluate the following presentations:

Inter-rater Verification

Questions?

University of North Texas Health Science Center Center for Learning and Development Assessment Contact Information Christine Savi, PhD Location: LIB-448 Phone: Website: