1. Rubrics: Using Performance Criteria to Evaluate Student Learning PERFORMANCE RATING PERFORMANCE CRITERIABeginning 1 Developing 2 Accomplished 3 Content Diagnosis (Application & Analysis) PERFORMANCE DESCRIPTORS Explanation (Knowledge & Comprehension) PERFORMANCE DESCRIPTORS Treatment (Evaluation) PERFORMANCE DESCRIPTORS Management (Creation) PERFORMANCE DESCRIPTORS Format AMA Style PERFORMANCE DESCRIPTORS

2. Agenda Assessment vs. Evaluation Forms of assessment Graduate examinations Item dissection Performance criteria, descriptors and rating levels Inter-rater reliability

3. Assessment vs. Evaluation Assessment Focuses on learning, teaching, and outcomes Provides information for improving learning and teaching. An interactive process between students and faculty that informs faculty how well their students are learning what they are teaching. Evaluation – Focuses on grades or scores. – May reflect classroom components other than course content and mastery level.

4. Forms of Assessment Case studies Direct observation Examinations Multiple Choice Short answer Essay Oral questioning Knowledge surveys Simulations Portfolios Presentations Self assessments Performance projects

5. Spotlight: Graduate Examinations Short answer Essay (vignettes, scenarios, case studies) Performance (demonstration, presentation)

6. Dissecting the Scenario/Essay Question 1. Identify the ‘what’ you want the students to know in each question: - Underline the key words/concepts: only nouns and/or adjectives A report from Parkland Hospital identifies a rise in admissions and subsequent limb amputation in one zip code in Dallas. As the newly appointed health educator for the city of Dallas, you have been asked to develop a new diabetes program for middle aged African American women in a housing development within that zip code. Please answer the following questions using correct grammar and spelling: 1. Describe and support an appropriate logic model to emphasize this program. 2. Discuss the essential components of the logic framework with regard to its implementation.

7. Characterize IDEAL Performance for Question #1 Using 1 sheet of white paper, divide into 3 vertical sections In the ‘3 rd or last’ column, list the qualities that need to be displayed in a student's work to demonstrate ideal performance Q1.). Describe and support an appropriate logic model to emphasize this program. IDEAL Describes the study and supports an appropriate logic model

8. Characterize MINIMAL Performance In the ‘1 st or Beginning’ column, list the qualities that need to be displayed in a student's work to demonstrate minimal or ‘best of the worst’ performance Q1.). Describe and support an appropriate logic model to emphasize this program. MINIMAL Does not describe the study or support an appropriate logic model

9. Characterize ACCEPTABLE Performance In the ‘2 nd or middle’ column, list the qualities that need to be displayed in a student's work to demonstrate acceptable performance Q1.). Describe and support an appropriate logic model to emphasize this program. ACCEPTABLE Either describes the study or supports an appropriate logic model, but not both

10. 3-scale Leveling - Question #1 Content ComponentBeginning 1 Developing 2 Exemplary 3 Criteria #1 Describe and support an appropriate logic model to emphasize this program. Does not describe the study or support an appropriate logic model Either describes the study or supports an appropriate logic model, but not both Describes the study and supports an appropriate logic model

11. List Descriptors – Question #2 Q2). Discuss the essential components of the logic framework with regard to its implementation. List what these key words refer to in each group Descriptors of your key words: Essential Components of a Logic Framework? a. Purpose b. Resources c. Activities d. Results e. Short-term outcomes f. Long-term outcomes

12. Characterize IDEAL Performance for Each Question Using 1 sheet of white paper, divide into 3 vertical sections In the ‘3 rd or last’ column, list the qualities that need to be displayed in a student's work to demonstrate ideal performance Discusses 5-6 the essential components of the logic framework with regard to its implementation. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

13. Characterize MINIMAL Performance In the ‘1 st or Beginning’ column, list the qualities that need to be displayed in a student's work to demonstrate minimal or ‘best of the worst’ performance Discusses 1-2 the essential components of the logic framework with regard to its implementation. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

14. Characterize ACCEPTABLE Performance In the ‘2 nd or middle’ column, list the qualities that need to be displayed in a student's work to demonstrate acceptable performance Discusses 3-4 of the essential components of the logic framework with regard to its implementation. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

15. 3-scale Leveling Content ComponentBeginning 1 Developing 2 Exemplary 3 Criteria #2 Discuss the essential components of the logic framework with regard to its implementation. Discusses 1-2 of the essential components of the logic framework with regard to its implementation. Purpose Resources Activities Results Short-term outcomes Long-term outcomes Discusses 3-4 of the essential components of the logic framework with regard to its implementation. Purpose Resources Activities Results Short-term outcomes Long-term outcomes Discusses 5-6 of the essential components of the logic framework with regard to its implementation. Purpose Resources Activities Results Short-term outcomes Long-term outcomes

16. Assessing Learning Rubric: A scoring tool listing criteria for varying levels of performance on a task A series of narrative statements describing the levels of quality of a product/performance Defines qualitative differences in levels of targeted performance Contains levels which apply to indicators for each strategy For each indicator, there is a brief written description of the different levels of performance based on performance criteria. Raters use descriptions to determine the level of accomplishment on each indicator, e.g.: Ratings of “Emerging”, “Developing”, and “Accomplished”

17. Why Use Rubrics? Used to: Define quality within the area being rated Articulate same target goals for improvement for everyone (consistency) Track change or improvement over time Provide a common set of definitions across all outcomes Lead self assessment and planning

18. Authentic Assessment “Task-based assessment scored by a rubric”, Facione, 2010 Rubric types: Analytic Targets > 1 content area; separate evaluation of multiple factors Used to identify and assess components of a finished product Holistic Targets 1 area Used to assess student work as a whole

19. Analytic & Holistic Examples Analytic Rubric Criteria 4 -Accomplished3 – Developing2 -Emerging1 - Beginning Plot Both plot parts are fully developed. One of the plot parts is fully developed and the less developed part is at least addressed. Both plot parts are addressed but not fully developed. Neither plot parts are fully developed. Setting Both setting parts are fully developed One of the setting parts is fully developed and the less developed part is at least addressed. Both setting parts of the story are addressed but not fully developed. Neither setting parts are developed. Characters Main characters are fully developed with much descriptive detail. Reader has a vivid image of the characters. Main characters are developed with some descriptive detail. Reader has a vague idea of the characters. Main characters are identified by name only. None of the characters are developed or named. Holistic Rubric Fiction Writing Content Rubric 5 The plot, setting, and characters are developed fully and organized well. The who, what, where, when, and why are explained using interesting language and sufficient detail. 4 Most parts of the story mentioned in a score of 5 above are developed and organized well. A couple of aspects may need to be more fully or more interestingly developed. 3 Some aspects of the story are developed and organized well, but not as much detail or organization is expressed as in a score of 4. 2 A few parts of the story are developed somewhat. Organization and language usage need improvement. 1 Parts of the story are addressed without attention to detail or organization.

20. Rubric Examples Varied scales, uses, descriptions Dimension generation aide (If/Then scenarios): http://webquest.sdsu.edu/rubrics/rubrics.html http://webquest.sdsu.edu/rubrics/rubrics.html Compare, contrast, compliment Open Ed Practices Resources http://openedpractices.org/resources Presentation/Decision Making rubrics: http://www.bgsu.edu/offices/assessment/page31468.h tml http://www.bgsu.edu/offices/assessment/page31468.h tml

21. Inter-rater Reliability = Consistency Ensure all grading participants agree with rubric Verify using sample papers from pilot group Review and/or revise rubric from pilot response Test/revise for proper maintenance

22. More Information… Rubrics http://digitalcommons.hsc.unt.edu/rubric/ Christine Savi, PhD CLD & QEP: Assessment 817.735.2942