Case Description Papers – Utilizing a New Scoring Rubric Peggy Cyr, MD, David Little, MD, Martha Seagrave, PA-C, Julie Schirmer, MSW, Kahsi Smith, PhD, India Broyles, EdD

Rationale Importance of Assessing medical student writing skills Process for improving evaluation tools Need a lot of input for evaluating such a complex performance

Methodology (Plan, Study, Do, Assess) 1.Review and revise original evaluation instrument 2.Trial the new instrument for 9 months 3.Elicit faculty feedback to improve instrument 4.Measure outcomes –Faculty perspective: focus group –Student perspective: 1:1 interviews, survey –Instrument quality: blinded faculty evals

Faculty Perspective Improved ability to rate differences in performance criteria Improved sense of concordance between raters Improved confidence and skills in assessing particular assessment criteria –ex: biopsychosocial determinants, topic relevance

Family Medicine Clinical Core Clerkship Case Description Paper Evaluation NAME:________________________ DATE:________________ TOPIC:______________________ Criteria Inadequate (0) Needs Improvement (1) Meets Expectations (2)Exceeds Expectations(3) Writing Conventions Poor organization and sentence structure impedes comprehension. Errors distract the reader. Needs editing, several grammatical and/or punctuation errors. Only 2-3 minor errors in grammar or punctuation. Writing style and grammar are very high quality. Minimal editing needed in punctuation. Logical Sequencing Completely unorganized case presentation 2-3 errors in sequencing or 2-3 sections missing. 1 error in sequencing or 1 section missing No errors in sequencing. All relevant information is present. Topic Relevance Topic is uncommon and not translated into relevant Family Medicine care issues. Common presentation, but focus is not relevant to Family Medicine care issues. Common presentation and focus adequately translated into Family Medicine care issues. Common presentation and translates expertly into Family Medicine care issues. Depth of Knowledge No analysis or discussion of differential diagnosis. Incorrect medical information. Brief analysis and limited discussion of differential diagnosis Topic well developed and adequate analysis of differential diagnosis Excellent discussion and in depth analysis of differential diagnosis Biopsychosocial Determinants of Health Errors in and/or no descriptors that make this patient unique. Paper is culturally insensitive. Describes 1-2 psychosocial aspects that make this patient unique. Describes 3-4 psychosocial aspects that make this patient unique. Describes 5 psychosocial aspects that make this patient unique.(family, living situation, impact of disease on life, perspective on their illness, and state of psychological health) Cost Issues No description of how cost influences medical decision making. Cost issues described, but incomplete awareness of how cost influences medical decision- making. Full discussion of cost issues and well linked to medical decision-making. Captures nuances of how cost influences medical decision making. References No references in the paper 3 years old and/or references do not support the conclusions of the paper. 3-4 references that are all current (last 3 years) and adequately support the conclusions of the paper. 5 or more references that are all current (last 3 years), include a review article and expertly support the conclusions of the paper. COMMENTS: TOTAL SCORE: ________ FACULTY SIGNATURE_______________________________________

Writing Conventions Criteria ScaleInadequateNeeds Improvement Meets Expectations Exceeds Expectations Before> 5 errors in spelling, grammar or punctuation. 3-5 errors in spelling, grammar or punctuation. 2-3 errors in spelling, grammar or punctuation. No errors in spelling, grammar or punctuation. AfterPoor organization and sentence structure impedes comprehension. Errors distract the reader. Needs editing, several grammatical and/or punctuation errors. Only 2-3 minor errors in grammar or punctuation. Writing style and grammar are very high quality. Minimal editing needed in punctuation.

Logical Sequencing Criteria ScaleInadequateNeeds Improvement Meets Expectations Exceeds Expectations BeforeCompletely unorganized case presentation 2-3 errors in sequencing or 2-3 sections missing 1 error in sequencing or 1 section missing No errors in sequencing and all relevant information presented AfterCompletely unorganized case presentation. 2-3 errors in sequencing or 2-3 sections missing. 1 error in sequencing or 1 section missing. No errors in sequencing. All relevant information is present.

Topic Relevance Criteria ScaleInadequateNeeds Improvement Meets Expectations Exceeds Expectations BeforeNo relevance to Family Medicine. Common diagnosis but focus is on non relevant management issues. Focus on common problem and management issues in Family Medicine. Topic is appropriate and focus is exemplary for scope of this assignment. AfterTopic is uncommon and not translated into relevant Family Medicine care issues. Common presentation, but focus is not relevant to Family Medicine care issues. Common presentation and focus adequately translated into Family Medicine care issues. Common presentation and translates expertly into Family Medicine care issues.

Depth of Knowledge/Focus Criteria ScaleInadequateNeeds Improvement Meets Expectations Exceeds Expectations BeforeSuperficial coverage of the topic or lack of focus. Lacking some of the essential components or focus. Appropriate depth of knowledge or focus. Topic is developed extremely well and fits scope of the assignment expertly. AfterNo analysis or discussion of differential diagnosis. Incorrect medical information. Brief analysis and limited discussion of differential diagnosis. Topic well developed and adequate analysis of differential diagnosis. Excellent discussion and in depth analysis of differential diagnosis.

Biopsychosocial Determinants of Health Criteria ScaleInadequateNeeds Improvement Meets Expectations Exceeds Expectations BeforeNo discussion of biopsycho- social issues. Some discussion of biopsycho- social issues and not well integrated. Biopsycho- social issues are adequately integrated. Extensive awareness of biopsychosocial issues. AfterErrors in and/or no descriptors that make this patient unique. Paper is culturally insensitive. Describes 1- 2 psychosocial aspects that make this patient unique. Describes 3-4 psychosocial aspects that make this patient unique. Describes 5 psychosocial aspects that make this patient unique (family, living situation, impact of disease on life, perspective on their illness, and state of psychological health).

Cost Issues Criteria ScaleInadequateNeeds Improvement Meets Expectations Exceeds Expectations BeforeNo description of how cost influences medical choices. Partial awareness of how costs influence medical choices. Full awareness of how costs influence medical choices. Captures nuances of how costs influence medical choices. AfterNo description of how cost influences medical decision making. Cost issues described, but incomplete awareness of how cost influences medical decision- making. Full discussion of cost issues and well linked to medical decision- making. Captures nuances of how cost influences medical decision making.

References Criteria ScaleInadequateNeeds Improvement Meets Expectations Exceeds Expectations BeforeNo citations. Only 1-3 citations and not original articles i.e. Just from Up to Date. 3-5 original references that are within the last 3 years. >5 references that are all up to date and include a review article. AfterNo references in the paper. 3 years old and/or references do not support the conclusions of the paper. 3-4 references that are all current (last 3 years) and adequately support the conclusions of the paper. 5 or more references that are all current (last 3 years), include a review article and expertly support the conclusions of the paper.

Were you aware of how you were going to be evaluated during the family medicine clerkship? 0=Not at all Aware 1=Somewhat Aware 2=Completely Aware Did you look at the two evaluation tools for the written case report? 0=Not at all 1=Briefly 2=Thoroughly Did the scoring rubric add to your understanding of the assignment? 0=Not at all 1=Somewhat 2=Very Much Do the categories in the rubric seem appropriate? 0=Not at all Appropriate 1=Somewhat Appropriate 2=Very Appropriate Do you have any suggestions for improving the wording of the rubric? Do you have any suggestions of categories that should be added or removed? Is there any way this assignment could have been made more interesting for you?

Student Perspective Students very willing to give feedback Some sections harder for them to do such as biopsychosocial model and cost issues

Instrument Quality We have checked the rubric for inter-rater reliability both using the old form and the new form

Summary Assessing and giving feedback on students’ writing skills is important Use scoring rubrics whenever possible Involve your faculty and students in improving your scoring rubrics

References 1.Chur-Hansen, A., Medical students' essay-writing skills: criteria-based self- and tutor-evaluation and the role of language background. Med Educ, (3): p Chur-Hansen, A. and J. Vernon-Roberts, The evaluation of undergraduate students' written English language skills. Med Educ, (8): p Kogan, J.R. and J.A. Shea, An assessment measure to evaluate case write-ups in a medicine core clerkship. Med Educ, (11): p Kogan, J.R. and J.A. Shea, Psychometric characteristics of a write-up assessment form in a medicine core clerkship. Teach Learn Med, (2): p

References 5.Molenaar, W.M., et al., Written case reports as assessment of the elective student clerkship: consistency of central grading and comparison with ratings of clinical performance. Med Teach, (4): p O'Brien, C.E., A.M. Franks, and C.D. Stowe, Multiple rubric- based assessments of student case presentations. Am J Pharm Educ, (3): p Schuwirth, L.W. and C.P. van der Vleuten, Different written assessment methods: what can be said about their strengths and weaknesses? Med Educ, (9): p Smith, J., D. Neely, and R. Hirschtick, Achieving inter-rater reliability in evaluation of written documentation. Med Educ, (5): p Improvement in the scoring rubric was created with the help of faculty scorers and medical student input.