1. Developing Rubrics Using a Collaborative Interview Naomi L. Lacy & Anna M. Arroyave Paul L Foster School of Medicine Texas Tech University Health Science Center 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

2. Basic plan for session  What is a rubric?  Our interview process  Discussion 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

3. Expanding on what exists  Session will touch on principles for rubric creation  Online resources  The online journal called practical assessment, research & evaluation has a wonderful set of articles on rubrics. I consider the Moskal article below to be essential reading:  Moskal, Barbara M. (2000). Scoring rubrics: what, when and how?. Practical Assessment, Research & Evaluation, 7(3).  http://pareonline.net http://pareonline.net  Other sources used for today  http://www.carla.umn.edu/assessment/vac/Evaluation/p_7.html http://www.carla.umn.edu/assessment/vac/Evaluation/p_7.html  http://www.teachervision.fen.com/teaching-methods-and- management/rubrics/4522.html http://www.teachervision.fen.com/teaching-methods-and- management/rubrics/4522.html  http://www.middleweb.com/rubricsHG.html http://www.middleweb.com/rubricsHG.html  http://www.uen.org/rubric/know.shtml http://www.uen.org/rubric/know.shtml 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

4. What is a rubric?  A preplanned grading criteria  What areas will you grade  What are the grades you will give  Criteria for grade in each area  Purposes:  Provide student with feedback on what it would take to improve his/her grade.  Support grade based on quality (rather than quantity) of performance  Increase inter-rater concordance  Typically shown as a grid 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

5. The context behind development of our technique  Dr. Arroyave was the course director for Society, Community, and the Individual.  SCI involved clinical experiences which needed evaluations to:  Provide students with feedback  Support the clinical component of the grade  Dr. Lacy was the newly hired Director of Evaluation and Assessment. 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

6. The Interview  A conversational and collaborative process  Questions meant to determine (generally in this order):  Do I need a rubric  What areas do I want to score  What kind of grading scale do I need  What criteria distinguish performance 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

7. Question 1- What are you having the students do?  They are going to go to the clinic and shadow a physician. Ideally, they are going to observe [this] type of interaction.  Do they need to just show up or is there some other criteria that they have to meet?  They need to behave professionally, I hope that they will practice their Spanish, and eventually they will be conducting an H&P  Based on this, we conclude that  a rubric is a useful tool and that the minimum criteria areas include professionalism, Spanish practice, and skills  Part of it will be used for all the clinic activities and part of it will be task specific 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

8. Q2 – how do you know the students, and therefore the course, succeeded?  This is an area where your interview really helps – Take Notes!  Possible follow up questions in this area:  What does that look like?  When would you consider a student a failure?  Could you tell me more about how those differ?  I’m not sure I understand  The Difference between…  Importance of …  What you really want them to… 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

9. Reflection time  Using my notes, we then distilled out what were the most important criteria for performance.  Professionalism was expanded to specific elements (dress, timeliness, and respectful conduct)  We kept Spanish and clinical skills  We added communication skills  These became the side columns on a grid 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

10. The first look Criterion Timeliness Dress Respectful conduct Spanish Skills Communication with you, staff & patients Professionalism 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

11. Q3 - Does that cover it all?  Looking at our grid, we realized that it did not truly cover it all. We had pulled out professionalism issues that were previously highlighted by the preceptors, but there were other professionalism issues that we wanted the students evaluated on.  We revised the table to show subcategories 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

12. The second look 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews Criterion Timeliness Dress Conduct Courtesy Respect Vocabulary Spanish Skills Communication with you, staff & patients ProfessionalismConfidentiality Professional boundaries Honesty One idea to a line!!!

13. What kind of grades?  Naomi’s first principle – unless you have a very good reason, never have more categories than the course.  Pass – Fail course gets a pass-fail rubric,  Honors, Pass, Fail – get these three categories.  A, B, C, D, Fail – get these  Naomi’s second principle – name the categories the same. 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

14. Q4 - What grades  We use pass fail for this course.  Is there any reason that you would need more than two categories?  Well, I want more than two  Would you please explain why?  … and we discussed how some behaviors might need to trigger immediate remediation.  Conclusion – base of categories is pass/fail but we would divide some into 2 parts. 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

15. 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews Criterion PassRemediate Timeliness Dress Communication Courtesy Respect Vocabulary Spanish Skills Conduct ProfessionalismConfidentiality Professional boundaries Honesty Have to pass to pass class to be able to advance = remediate rather than fail

16. Now all we had to do was fill in the descriptions  Back to the interview notes:  The comparison questions about what was a successful versus unsuccessful student yielded descriptions that we then just put back into the rubric.  At each one we asked, is there a compelling reason why we should split this. 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

17. The first criteria 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews Criterion PassRemediate Timeliness On timeLate Stayed full shift Left early

18. A criteria requiring more grading groups 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews Criterion PassRemediate Communication Courtesy Courteous DiscourteousInsulting Respect RespectfulDisrespectfulPatronizing Vocabulary Appropriate Vocabulary Uses Jargon with patients These require an immediate remediation

19. Additional feedback  For some items, you need more information on how the individual decided on the descriptor.  On-time is pretty self-explanatory  Insulting needs more information  We add comment fields to most of our rubrics  Please comment  Adds to LCME narrative assessment data 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

20. Questions & Group Discussion Practice often helps develop questions so I would like to run a group exercise. 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

21. Exercise (15-25 minutes)  3-5 minutes: Pick a topic (suggested small groups, or one of the ACGME areas)  10 minutes: Group interview  5-7 minutes: Discussion of process and Questions 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews

22. 29Apr2011 STFM Annual Meeting – Developing Rubrics Using Collaborative Interviews