1. Seth D. Goldstein, MD 1, Brenessa Lindeman, MD 1, Jorie Colbert-Getz, PhD 3, Trisha Arbella 1, Robert Dudas, MD 2, Anne Lidor, MD, MPH 1, Bethany Sacks, MD, MEd 1 Johns Hopkins Hospital Departments of Surgery 1 and Pediatrics 2 Johns Hopkins School of Medicine Office of Medical Education Services 3

3. High Correlation Low Correlation Surgery Emergency Medicine Internal Medicine RadiologyPediatrics Objective and Subjective Knowledge Correlation

4.  Investigate convergent validity between subjective ratings of surgery student clinical knowledge and scores on the NBME subject examination  Inter-rater reliability of faculty and resident evaluations of global clinical knowledge

5. 12345 UnacceptableNeeds Improvement At Expected Level Above Expectations Outstanding Unable to apply preclinical knowledge to understand basic medical problems. Inconsistent understanding of patient problems. Limited differential diagnosis. Knows basic differential diagnoses of major/active problems in patients. Understands team’s choice of therapy. Knows expanded differential diagnoses, including recognition of emergencies. Can discuss therapeutic options. Knows nuances of differential diagnosis, including disease prevalence and anticipated history & exam findings. Able to independently formulate a management plan. Able to assign prognoses. Subjective Student Evaluation Scale Retrospective, cross-sectional study from July 2009 – June 2011

6. Inter-rater Reliability Intraclass Coefficient Attending evaluations (n=216) Resident evaluations (n=207) Convergent Validity Correlation of: Average evaluation ratings NBME exam scores Quartile Comparison ANOVA Average resident and attending ratings

7. RatingAttendings % (n) Residents % (n) 1 N/A 2<1% (4)1% (6) 320% (178)28% (244) 457% (498)47% (402) 522% (193)24% (211) 0.760.81

8. ρ 2 : Attendings <0.01 Residents 0.01

9. Medical Student Ratings by NBME Percentile Group QuartileAttending n (SD) Resident n (SD) Bottom 25 th 3.96 (0.45) 3.81 (0.48) Middle 50 th 4.00 (0.35) 3.95 (0.43) Top 25 th 4.08 (0.40) 4.10 (0.49) Bottom 25th Middle 50th Top 25th

10. Other aspects of student performance are perhaps being considered as a proxy for knowledge Faculty members form a one-dimensional view of student performance when assigning grades 1 Faculty may conceptualize a generalized global assessment of student performance from which they base all of their ratings 1 Pulito AR, Donnelly MB, Plymale M, et al. What do faculty observe of medical students' clinical performance? Teach Learn Med 2006;18:99-104.

11. Contextual and discipline-specific trends may contribute to how faculty and residents perceive student performance The technical domain of surgeons may skew the ability to assess other areas of performance Faculty and residents would benefit from dedicated training in knowledge assessment