1. FMEC Milestones Learning Community Jennifer L. Middleton, MD, MPH, FAAFP Sherrilyn M. Sethi, D.MH Pamela Vnenchak, MD Sara Beth Karp, MD, FAAFP Allen Shaughnessy, PharmD, MMEd Netra Thakur, MD, MPH Laurence Bauer, MSW

6. Milestones! ACGME: Milestone Project Introduced in 2013 Role of the Milestones Clinical Competency Committee 1) Purpose 2) Membership 3) Process RESIDENTSPROGRAMSSOCIETY

7. COMPETENCY SUB-COMPETENCY Milestone

8. ACGME CCC Literature The Family Medicine Milestone Project http://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/FamilyMedicineMilestones.pdf_ Clinical Competency Committees: A Guidebook for Programs http://www.acgme.org/acgmeweb/Portals/0/ACGMEClinicalCompetencyCommitteeGuidebook.pdf

9. Survey Results Response from 104 programs Various program sizes 98% use residency management software

10. Logistics WHO: Size & Membership Program Director Assistant/Assoc PD* Depart Chair FM / Non FM Faculty Institutional GME Director Nursing Staff Residency Staff Behavioral Scientist Hosp Admin Director Dept Rep Residents (Chief, Reviewee) Fellows

11. CCC Composition Residency PD Asst/Assoc Director Dept Chair Fam Med Faculty Instit’l GME Director Non Fam Med Faculty Patient Rep Nursing staff Residency staff

12. Logistics WHEN: From “As Needed” to every 6 months Majority meet every 6 months HOW LONG: Majority: < 2hrs

13. Clinical Competency Committee End-of- Rotation Evaluations Peer Evaluations Self Evaluations Case Logs Student Evaluations Patient/ Family Evaluations Operative Performance Rating Scales Nursing and Ancillary Personnel Evaluations Assessment of Milestones Clinic Workplace Evaluations Mock Orals OSCE ITE Sim Lab Unsolicited Comments

14. Input to CCC Rotation assessments/evals 100% Preceptor evals ITE reports Direct observation Procedure logs Staff assessments Resident advisor assessments Resident peer assessments 70%

15. Input to CCC Resident self assessment 66% Resident practice data/metrics Patient eval of residents Field notes OSCE Simulation assessments Other 15.5%

16. Changes in Assessment post Milestones 85% mapped/ing evaluations to milestones 32% changed frequency of observations (esp preceptors) 51% added new evaluations

17. High Frequency EventsLow Frequency EventsSpecific Areas Shift cardsJournal clubResearch/Scholarly activity Field notesOther presentationsTransitions of care “on the fly” Advisor Meeting Patient Safety checklists Procedure evals Orientation PCMH Behavioral health evalsChronic disease

18. Advisors: 66% Program Directors, other faculty, CCC chair and/or support staff: 25% No one: 9% Who does the prep?

19. 44% 31% 6% prep time:

20. After the CCC, how do residents get feedback? advisors program directors >

21.  Identifying Milestones in curriculum  Spreadsheets of milestones & curricular elements  Rotation goals and objectives  Revised wording/questions of evaluations  Using evaluation software (New Innovations)  Adding new curriculum components Mapping Sub-competencies &/or Milestones

22. INCOMING RESIDENTS 11.522.533.544.55 Patient Care80.4%16.1%3.6% Medical Knowledge81.8%12.7%5.5% Systems Based Practice87.3%9.1%3.6% Practice Based Learning & Improvement 85.5%10.9%3.6% Professionalism73.2%12.1%12.5%1.8% Communication78.2%12.7%7.3%1.8%

23. END OF PGY-1 11.522.533.544.55 Patient Care2.9%16.4%71.6%7.5%1.5% Medical Knowledge4.5%14.9%70.2%8.9%1.5% Systems Based Practice3.1%15.4%75.4%4.6%1.5% Practice Based Learning & Improvement 1.5%15.4%76.9%4.6%1.5% Professionalism1.5%13.6%68.2%9.1%6.1%1.5% Communication1.5%16.6%66.7%10.6%6.1%1.5%

24. END OF PGY-2 11.522.533.544.55 Patient Care03.1%14.1%75%6.3%1.6% Medical Knowledge4.7%14.1%71.9%7.8%1.6% Systems Based Practice1.6% 19.7%70.5%4. 9%1.6% Practice Based Learning & Improvement 1.6% 15.9%74.6%4.8%1.6% Professionalism1.6%14.3%69.8%9.5%4.8% Communication1.6%12.7%71.4%11.1%3.2%

25. END OF PGY-3 11.522.533.544.55 Patient Care5.6%12.7%77.5%4.2% Medical Knowledge7%11.3%77.5%4.2% Systems Based Practice1.4%7%12.7%74.7%4.2% Practice Based Learning & Improvement 1.5%5.8%14.5%73.9%4.4% Professionalism5.7%12.9%72.9%7.1%1.4% Communication5.7%11.4%74.3%8.6%

26. Top 5 Challenging Sub-competencies PC-4: address issues…that remain over time w/o clear dx SBP-1: provides cost-conscious medical care SBP-3: advocates for individual and community health PBLI-1: locates, appraises, assimilates evidence PBLI-3: improves systems in which physician provides care

27. “best practices” - infrastructure Using an electronic format to assess milestones: Milestone-based rotation evaluations Milestone reports from electronic platform Identification of expected levels for each class

28. “best practices” – education Faculty development Resident education Self-assessments

29. “best practices” - prep Review of resident data prior to CCC By coordinator, advisor, faculty member, etc. Meetings prior to CCC to review data Pre-Identification of which member to review which resident

30. “best practices” - @ the CCC CCC Milestone assessment by PGY Class Using excel worksheets/handouts/and other tools to organize data & projecting data on screen at meetings

31. Do you have any best practices to share?

32. Jennifer Middleton, MD, MPH, FAAFP Associate Program Director Riverside Methodist Hospital Family Medicine Residency Sherrilyn M. Sethi, D.MH Assistant Program Director Worcester Family Medicine Residency Pamela Vnenchak, MD Deputy Program Director Lancaster General Family Medicine Residency Sara Beth Karp, MD, FAAFP Family Medicine Faculty Mt. Sinai Beth Israel Family Medicine Residency

33. PROGRAM 8 members APD: Chair PD Program Administrator FM faculty GME director 6 Semi-Annual Meetings 60 Minute Meetings 1 meeting for each PGY class twice a year approx. 10 min/resident. Ongoing gathering of resident data by Med Ed makes prepping for CCC meetings (fairly) painless CCC Membership Best practices CCC Meeting CCC Membership CCC Process 6-6-6 1 PCMH community health center on the campus of a large tertiary care hospital Pre-Prep: Data to advisors: Rotation evals Pt #s/metrics Procedure log Advisor assigns prelim milestones Presentation: Post-Process Review prelim milestones; discussion Report to PD -> discusses with resident @ biannual review

34. 3 Semi-Annual Meetings 90 Minute Meetings Arranged by PGY Class Each Resident reviewed in 7-10 mins. CCC Membership CCC Meeting 9 Members (6 reviewers) APD: Chair PD Program Administrator 3 Health Center Education Dirs. 3 Associate PDs CCC Membership CCC Process PROGRAM 12-12-12 3 Community Based Health Centers BEST PRACTICES Pre-Prep: (3wks) Presentation Post-Process Presentation Schedule Data: - Dashboard - Evalue Reports Milestone Packet Data projected Milestone Packet Discussion Data Collection Milestone Report Milestone Self- Assessment CCC Education Mapping Milestones to Curriculum Milestone Packet EValue Milestone Summary Report

35. BEST PRACTICES CCC Education Mapping Milestones to Curriculum EValue Milestone Summary Report

36. PROGRAM 5 Faculty members plus 1 administrative support person No formal chair Minimum of 3 must be present at meeting Res & Advisor meet Fall & Spring for Semi-annual Review that includes Milestones Summary, Res self rating & advisor rating submitted to CCC The final ratings of CCC are shared with resident & advisor. mtgs over 1-2 weeks ~35 hrs total Begin with R1s CCC members review SAR summary, Res & Adv ratings ahead, full res file @ mtg NewInnovations 20-45 min /res Resident self assessment Key role of advisor CCC standardizes ratings criteria CCC Membership Best Practices CCC Process CCC Membership CCC Meeting 13-13-13 Community Based 1 Rural Family Health Center & 1 Urban Family Health Center (residents rotate through both) New Innovations Residency Suite for evals, Milestones

37. PROGRAM 4 core Members FM Fac : Chair PD Behavioral Health Program Administrator Excel spreadsheet with color coding “met/not met” can be viewed by resident or by class Intensive prep and support of advisors as they synthesize milestones CCC Membership BEST PRACTICE CCC Meeting CCC Membership CCC Process 9-9-10 Urban FQHC Pre-Prep: Presentation: Post-Process Staff and CCC prep of files Each advisor synthesizes milestones and other data Advisor presents synthesis Faculty discussion Advisor meets with resident to review CCC “cleans up” Cluster of weekly meetings around Semi- Annual Evals 4 weeks in a row 2 hours Meetings attended by all faculty

38. BEST PRACTICES

39. NEXT STEPS finish survey data analysis disseminate findings define and test “best practices” What do you think we should be doing next?

40. JOIN US! It’s not too late to participate in our survey! www.surveymonkey.com/r/MilestonesSurvey Or to join our learning community!

41. Questions/comments?