1. Orientation, Milestones, and Initial Assessment of New Interns Webinar Presentation (June 11, 2014): Cindy Works MD, Brian Veauthier MD Content: Brian Veauthier MD, Beth Robitaille MD University of Wyoming Family Medicine Residency - Casper

2. Objectives Review benefits of evaluating new interns including ACGME Requirements Describe a process we started to evaluate our new interns based on the Family Medicine Milestones Share our successes and pitfalls Present a case study of a current intern Open the floor for interactive discussion

3. Why Evaluate New Interns? Because the ACGME says we need to Avoid the December shock factor of a poorly performing resident Variance in medical school education - assess strengths and weaknesses

4. What does the ACGME require? V.A.2.b) The program must: V.A.2.b).(1) provide objective assessments of competence in patient care and procedural skills, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice based on the specialty-specific Milestones; (Core) V.A.2.b).(1).(a) This assessment must involve direct observation of resident-patient encounters. (Detail) V.A.2.b).(1).(b) Each resident should be assessed in each of the six competency areas on entrance into the program. (Detail)

5. Our Process Each new intern evaluated using Beta Testing Milestone Form Gave recommendations for areas of concern CCC Committee met 2 weeks after orientation Further input given for evaluations IEPs finalized CCC Committee shared results with faculty Reviewed the Milestones in general Provided IEPs where appropriate Faculty advisors met individually with interns

6. The Tools General Interaction During Orientation USMLE/COMLEX Scores Mock In-training ExamOSCE Cases Milestone Evaluation

7. The Tools General Interaction During Orientation Useful to evaluate some of the Milestones under Professionalism and Communication (Prof-2; Com-3)

8. The Tools USMLE/COMLEX Scores Another piece of data for evaluating Medical Knowledge Poor test takers tend to be poor test takers

9. The Tools Mock In-training Exam Give complete ITE exam from previous year Score exam compared to previous intern class Helps with MK-1

10. The Tools OSCE Cases Use one afternoon and rotate the interns through eight OSCE cases - 30 minutes per case Conducted by core faculty

11. The Tools OSCE 1-CHEST PAIN CASE PATIENT CARE PC-1 Care of acutely ill patient MEDICAL KNOWLEDGE MK-2 Applies critical thinking skills SYSTEMS-BASED PRACTICE SBP-2 Emphasizes patient safety

12. The Tools OSCE 2-DIABETES CHRONIC CARE PATIENT CARE PC-2 Care for patient with chronic illness MEDICAL KNOWLEDGE MK-2 Applies critical thinking skills SYSTEMS-BASED PRACTICE SBP-1 Provides cost conscious care SBP- 4 Coordinates team based care

13. The Tools OSCE 3-DEPRESSED PATIENT PATIENT CARE PC-1 Cares for acutely ill MEDICAL KNOWLEDGE MK-2 Applies critical thinking skills SYSTEMS-BASED PRACTICE SBP-4 Coordinates team based care

14. The Tools OSCE 4-OB/INFANT FEVER CASE PATIENT CARE PC-1 Care of acutely ill patient MEDICAL KNOWLEDGE MK-2 Critical thinking skills

15. The Tools OSCE 5-WELL WOMAN CASE PATIENT CARE PC-3 Disease prevention and health promotion MEDICAL KNOWLEDGE MK-2 Applies critical thinking SYSTEMS-BASED PRACTICE SBP-2 Emphasizes patient safety SBP-4 Coordinates team based care

16. The Tools OSCE 6-SUTURING CASE PATIENT CARE PC-5 Procedures

17. The Tools OSCE 7-X-RAYS AND ECGS MEDICAL KNOWLEDGE MK-2 Applies critical thinking

18. The Tools OSCE 8-MEDICAL INFORMATION MASTERY PRACTICE BASED LEARNING AND IMPROVEMENT PBLI-1 Locates, appraises, and assimilates evidence from scientific studies related to their patients’ health problems

19. Sample Evaluations

20. Intern A “The Rock Star"

21. Intern A- Comments/Recommendations “Excellent test scores 440. Mean from last year 386 with std 74” “Professionally appropriate interaction” “Faculty advisor feels like she won the lottery”

22. Intern B “Should be OK"

23. Intern B- Comments/Recommendations “Would benefit from showing more confidence” “Needs significant work on suture skills” “Performed lower than expected on the mock ITE. Score 300. Mean from 2012- 386 with std 74. consider early IEP” “Professional interactions” “Very quiet and not interactive during orientation-is she fitting in ok?”

24. Intern C “Help"

25. Intern C- Comments/Recommendations “Give literature on common issues that is clinical” “Give specific info on common issues” “Needs organized approach to EKG- Wicks handout” “Scored 220 on mock ITE- mean last year was 386 with STD 74. Strongly recommend early IEP” “Has demonstrated appropriate professional interaction so far” “Some concerns with language barriers” “Interacting well with peers”

26. INTERN C MILESTONE IMPLEMENTATION INTO AN INDIVIDUAL EDUCATION PLAN

27. BIG REVEAL: ORIENTATION MILESTONE EVALUATION Significant deficits in her medical education leading to obvious need for intervention Milestone deficiencies most obvious in: Medical Knowledge Patient Care Communication

28. JULY: ROUTINE ADVISOR MEET & GREET Reviewed Milestone deficits Outlined her initial improvement plan/IEP Outlined close monitoring and follow-up

29. OCTOBER: ROUTINE ADVISOR MEETING AND EVALUATION Some progress noted, but insufficient to become commensurate with her peers Confirmed continued deficits and need for formal intervention and remediation IEP General consensus: she was functioning at early MS-4 level

30. OCTOBER: ADVISOR & PROGRAM DIRECTOR MEETING Intern C formally notified of deficiencies Remediation IEP developed based on the Milestones, concentrating on the most relevant competencies: Medical Knowledge Patient Care Communication Practice-based Learning and Improvement Specific goals: to progress from Level 0 to Level 1 by January in the identified sub-competencies Key away rotation of inpatient pediatrics in the interim

31. JANUARY: ADVISOR & PROGRAM DIRECTOR MEETING Intern C reassessed on the Milestones Core program rotational evaluations Video evaluation Field notes ITE Verbal feedback from away pediatric rotation attending Senior preparedness OSCE Demonstrated improvement in several areas but did not reach goal of Level 1 on all identified sub-competencies Key float rotation in February/March provided next opportunity to achieve the progression required to proceed on as a PGY-2: progress towards Level 2 on identified sub-competencies

32. MARCH: ADVISOR & PROGRAM DIRECTOR MEETING Intern C reassessed on the Milestones Field notes Evaluations Verbal feedback from faculty Written feedback from senior float resident Other IEP requirements (ITE and board preparation) Demonstrated enough continued progress in the identified sub-competencies to continue at our residency Remains on remediation IEP

33. MEDICAL KNOWLEDGE

34. PATIENT CARE

35. COMMUNICATION

36. PRACTICE-BASED LEARNING AND IMPROVEMENT

37. CONCLUSIONS Orientation Milestone evaluation via OSCE and mock ITE helped to identify deficiencies early leading to early implementation of improvement plan Monitoring Milestone progression throughout the year helped to define specific and objective deficiencies The Milestone based IEP provided the appropriate detail and language to set specific goals and plans of action The Milestones played an important role in this successful (although ongoing) remediation

38. How did it work? Met ACGME requirement for evaluating new interns Introduced the new interns to the Milestones Helped identify a weak intern and start early remediation Only a snapshot of the intern Unable to evaluate certain Sub-competency and Milestones Time consuming- at least 30 minutes per intern Positives Negatives

39. How did it work? Practical Points Evaluate all new interns one question at a time Try to have all the data from the tools easily accessible and organized

40. How did it work? Areas for Improvement/Changes Hopefully use New Innovations for Milestone evals with OSCEs mapped Better organization of data from tools Continue to “learn” the Milestones Continue to “fine tune” OSCEs and evaluation tools

41. Summary Overall found the process valuable and will continue the process Meets ACGME requirement for evaluating incoming interns Mechanism for introducing the Milestones Helped us identify a very weak resident early

42. Discussion