1. ACGME 2016 Annual Education Conference SES052 How to Use Milestones to Provide an Evidence-based Early Warning System for Identifying the Academically Vulnerable Resident February 26, 2016 Ann Dohn, MA, DIO and Nancy Piro, PhD

2. Conflict of Interest Ann Dohn, MA - No Conflicts of Interest to Declare Nancy Piro, PhD - No Conflicts of Interest to Declare

3. Agenda - Criticality of Early Identification Results of a national survey of DIOs and program directors on Milestones Milestones as an early identification of resident academic vulnerability – Evaluations, Safety (SAFE) reports, in service exam scores – Interview behaviors, letters of reference, and summative evaluations from prior training programs. Mining the Milestone Data and Other Data to Identify at Risk Residents – Making it easy Tools which summarize resident Milestones as well as other data to facilitate collaborative discussion and remediation planning Using learning resources including educational testing and consultation with appropriate specialists Electronic resource toolkit to “Take Home”

4. Results of a 2015 National Survey of Program Directors on Milestones “Milestone evaluations increase our (PDs) documentation of trainee weaknesses/areas for improvement”

5. Results of a 2015 National Survey of DIOs on Milestones 33.33% 25% 41.67% “Milestone evaluations increase our (DIOs) documentation of trainee weaknesses/areas for improvement” All Agreed

6. This would never happen… “What do you mean I have to put my resident on probation? I want to terminate them at the end of the month!” 6/1/15

7. Evaluations Competency based faculty evaluations of residents (rotations) 360 evaluations (Techs, Allied health Professionals, nurses, patients, families, coordinators, etc)

8. ‘Data Mining’ Beyond Evaluations - Other Data Sources SAFE reports In service exam scores Interview, meeting and other observed behaviors Letters of reference Summative evaluations from prior training programs – Awareness of sources of potential evaluator bias

9. Leveraging Resident Management System (RMS)Tools - if Available RMSs – becoming more feature rich – CCC Online Functionality – Assign Files, Upload all required data for Preliminary Review, etc – Curriculum Goals and Objectives and learning outcomes by rotation Teaching and Assessment methodologies – Evaluation tool development Sharing between programs and institutions

10. Leveraging Resident Management System (RMS)Tools - if Available Conference attendance statistics – Core competencies linked to specified conferences – Attaching conference materials for later reference Procedures and levels; linked procedure evaluations

11. Resident portfolio tools QI participation and outcomes Scholarly Activity logs Leveraging Resident Management System (RMS)Tools, if Available

12. Leveraging Your Program Coordinator

13. Creating a Resident Performance Profile Step 1 – Defining what to track (Example from Program)

14. Creating a Resident Performance Profile Step 1 – Defining what to track

15. Creating a Resident Performance Profile Step 1 – Defining what to track

16. Creating a Resident Performance Profile Step 1 – Defining what to track

17. Creating a Resident Performance Profile Step 1 – Defining what to track

18. Creating a Resident Performance Profile Step 1 – Defining what to track

19. Patient Communication Data – Questions and Scale 19 Resident Assessment “How well did the resident physician…” Very PoorPoorFairGoodVery Good N/A 1. Greet you in a way that made you feel comfortable?12345 N/A 2. Treat you with respect?12345 N/A 3. Show interest in your ideas about your health?12345 N/A 4. Understand your main health concerns?12345 N/A 5. Pay attention to you (look at you, listen carefully)?12345 N/A 6. Let you talk without interruptions?12345 N/A 7. Give you as much information as you wanted?12345 N/A 8. Talk in terms you could understand?12345 N/A 9. Check to be sure you understood everything?12345 N/A 10. Encourage you to ask questions?12345 N/A 11. Involve you in decisions as much as you wanted?12345 N/A 12. Discuss next steps, including any follow-up plans?12345 N/A 13. Show care and concern?12345 N/A 14. Spent the right amount of time with you?12345 N/A Open-ended questionsOpen Response 15. What did you like about your resident’s communication? 16. How can this resident improve? 17. Do you have any other comments, questions, or concerns?

20. Creating a Resident Performance Profile Step 2 - Link data sources to milestones

21. Milestone Data Translation to Numerical Data

22. Creating a Resident Performance Profile Step 3: CCC defines performance ranges At or Above Expectation: 2.8 and higher Below Expectation: 1.7 – 2.7 Remediation: Below 1.7 Example: For all aggregate milestone evaluation scores for a PGY 3, the CCC defines these ranges: Hypothetical CCC Defined Ranges:

23. Creating a Resident Performance Profile Step 4 – Set conditional formatting

24. Example: Aggregate milestone evaluation data cells Highlight cells to apply the conditional formatting Creating a Resident Performance Profile Step 4 – Set conditional formatting

25. Click on “Conditional Formatting” “Highlight Cells Rules” Creating a Resident Performance Profile Step 4 – Set conditional formatting

26. Select, “Greater Than” “Less Than” or “Between” to Set Value Ranges Choose the corresponding fill color (e.g., red, yellow, green) Creating a Resident Performance Profile Step 4 – Set conditional formatting

27. Creating a Resident Performance Profile Compiling and centralizing data

28. Resident Performance Profile: Step 5: Enter in data

29. Creating a Resident Performance Profile Visual Trends and Detailed Data

30. When Remediation is Evident

31. Employing Resident Management System (RMS)Tools - if Available to Demonstrate Performance in a Peer Context Aggregate reporting and graphic summaries Peer or departmental average, individual average, minimum and maximum scores, standard deviation or listing of all scores

32. What To Do with an Underperforming Resident? 1.Look at your tracker for low scores… probe…

33. What can you do? 1. Engage the resident at the first sign of troubling results 2. Formulate an action plan (talk with your faculty) Assign a mentor Are institutional resources needed (DIO?) 3. PD must commit to close monitoring and follow-up

34. Case Study #1 (Disclaimer-all names and Depts and Details have been altered) SITUATION: – Neurosurgery resident – Low in-service exam scores – 1 % – Unable to keep up with ‘paper work’ and electronic charting - esp. discharge summaries – Great interpersonal skills – Outstanding surgical skills – Former concert musician ACTIONS: RESULTS

35. Case Study #2 - (Disclaimer-all names and Depts and Details have been altered) SITUATION: – Advanced Radiology Fellow – Nursing complaints about his responsiveness – Good clinical skills – Excellent test scores – Widely divergent evaluations (Faculty vs Nursing) ACTIONS: RESULTS

36. Using other resources to assist in remediation Educational testing Consultation with appropriate specialists Culture and Linguistic Specialists Well Being Resources Mentors

37. Electronic Toolbox for You! http://med.stanford.edu/gme/GME_Community.html Our Video on Conditional Formatting The Practice Excel File that Accompanies the Video (Refer to tab PGY3 for practice): Resident Performance Profile ToolResident Performance Profile Tool

38. Tools Can Be Downloaded @ www.gme.stanford.edu GME CommunityTemplates

39. Questions & Contact Information Ann Dohn: adohn1@stanford.eduadohn1@stanford.edu Nancy Piro: npiro@stanford.edunpiro@stanford.edu