1. A Step Ahead Assessing the Baseline Competency of Incoming Interns Andrea Wendling, M.D. Phillip Baty, M.D. Grand Rapids Family Practice Residency Program Michigan State University College of Human Medicine

2. A Step Ahead Assessing the Baseline Competency of incoming Interns Outline: Background Description of the Intern Clinical Judgment Evaluation Testing our system Using the information Discussion

3. Goals of Evaluation System Essential question: Can this intern safely take call? Understand baseline abilities and expectations Identify issues early

4. Defining Competencies Define competency  Recognize a patient is acutely ill  Understand help is needed  Communicate acuteness of situation appropriately

5. Designing the Intern Clinical Judgment Evaluation Use aspects of established testing systems Design stressful patient situations Test to be certain we were measuring our goals

6. Testing Room Observation Room Workstation

9. Chest pain

12. Chest Pain

14. Call for Help

15. Chest Pain- Summary Slide

16. Summary Reports

17. Testing our System To test our system, we compared three things:  Summary impressions from ICJE  Faculty prediction survey  End of year summary

18. Results ICJE  Kappa 0.80 Faculty prediction  Kappa 0.54

19. Diagnosis +Diagnosis - Test +132 Test -115 Sensitivity0.93 Specificity0.88 PPV0.87 NPV0.94 LR +7.9 LR -0.08 ICJE vs. Program Director

21. Diagnosis +Diagnosis - Test +81 Test -617 Sensitivity0.57 Specificity0.94 PPV0.89 NPV0..73 LR +10.3 LR -0.4 Faculty vs. Program Director

23. Results Resident feedback  Overwhelmingly positive  View it as a teaching experience

24. Using the information Personalized plans developed  Faculty, PGYIII and or Chief resident  Focus on self-reflection issues  Mentoring

25. Using the Information FMS continually changing  More resident supervision Plan on continuing due to  Additional information  Timely information  Cost

26. Other options Tried Comparing to OSCE  Mechanics vs. thought process  Timeliness Comparing to IM H and P  Mechanics vs. thought process Comparing to ERAS  No idea of resident ability to self- reflect

27. A Step Ahead Assessing the Baseline Competency of Incoming Interns Summary Handouts Question and Answer Session