FMEC Milestones Learning Community Jennifer L. Middleton, MD, MPH, FAAFP Sherrilyn M. Sethi, D.MH Pamela Vnenchak, MD Sara Beth Karp, MD, FAAFP Allen Shaughnessy, PharmD, MMEd Netra Thakur, MD, MPH Laurence Bauer, MSW

Milestones! ACGME: Milestone Project Introduced in 2013 Role of the Milestones Clinical Competency Committee 1) Purpose 2) Membership 3) Process RESIDENTSPROGRAMSSOCIETY

COMPETENCY SUB-COMPETENCY Milestone

ACGME CCC Literature The Family Medicine Milestone Project Clinical Competency Committees: A Guidebook for Programs

Survey Results Response from 104 programs Various program sizes 98% use residency management software

Logistics WHO: Size & Membership Program Director Assistant/Assoc PD* Depart Chair FM / Non FM Faculty Institutional GME Director Nursing Staff Residency Staff Behavioral Scientist Hosp Admin Director Dept Rep Residents (Chief, Reviewee) Fellows

CCC Composition Residency PD Asst/Assoc Director Dept Chair Fam Med Faculty Instit’l GME Director Non Fam Med Faculty Patient Rep Nursing staff Residency staff

Logistics WHEN: From “As Needed” to every 6 months Majority meet every 6 months HOW LONG: Majority: < 2hrs

Clinical Competency Committee End-of- Rotation Evaluations Peer Evaluations Self Evaluations Case Logs Student Evaluations Patient/ Family Evaluations Operative Performance Rating Scales Nursing and Ancillary Personnel Evaluations Assessment of Milestones Clinic Workplace Evaluations Mock Orals OSCE ITE Sim Lab Unsolicited Comments

Input to CCC Rotation assessments/evals 100% Preceptor evals ITE reports Direct observation Procedure logs Staff assessments Resident advisor assessments Resident peer assessments 70%

Input to CCC Resident self assessment 66% Resident practice data/metrics Patient eval of residents Field notes OSCE Simulation assessments Other 15.5%

Changes in Assessment post Milestones 85% mapped/ing evaluations to milestones 32% changed frequency of observations (esp preceptors) 51% added new evaluations

High Frequency EventsLow Frequency EventsSpecific Areas Shift cardsJournal clubResearch/Scholarly activity Field notesOther presentationsTransitions of care “on the fly” Advisor Meeting Patient Safety checklists Procedure evals Orientation PCMH Behavioral health evalsChronic disease

Advisors: 66% Program Directors, other faculty, CCC chair and/or support staff: 25% No one: 9% Who does the prep?

44% 31% 6% prep time:

After the CCC, how do residents get feedback? advisors program directors >

 Identifying Milestones in curriculum  Spreadsheets of milestones & curricular elements  Rotation goals and objectives  Revised wording/questions of evaluations  Using evaluation software (New Innovations)  Adding new curriculum components Mapping Sub-competencies &/or Milestones

INCOMING RESIDENTS Patient Care80.4%16.1%3.6% Medical Knowledge81.8%12.7%5.5% Systems Based Practice87.3%9.1%3.6% Practice Based Learning & Improvement 85.5%10.9%3.6% Professionalism73.2%12.1%12.5%1.8% Communication78.2%12.7%7.3%1.8%

END OF PGY Patient Care2.9%16.4%71.6%7.5%1.5% Medical Knowledge4.5%14.9%70.2%8.9%1.5% Systems Based Practice3.1%15.4%75.4%4.6%1.5% Practice Based Learning & Improvement 1.5%15.4%76.9%4.6%1.5% Professionalism1.5%13.6%68.2%9.1%6.1%1.5% Communication1.5%16.6%66.7%10.6%6.1%1.5%

END OF PGY Patient Care03.1%14.1%75%6.3%1.6% Medical Knowledge4.7%14.1%71.9%7.8%1.6% Systems Based Practice1.6% 19.7%70.5%4. 9%1.6% Practice Based Learning & Improvement 1.6% 15.9%74.6%4.8%1.6% Professionalism1.6%14.3%69.8%9.5%4.8% Communication1.6%12.7%71.4%11.1%3.2%

END OF PGY Patient Care5.6%12.7%77.5%4.2% Medical Knowledge7%11.3%77.5%4.2% Systems Based Practice1.4%7%12.7%74.7%4.2% Practice Based Learning & Improvement 1.5%5.8%14.5%73.9%4.4% Professionalism5.7%12.9%72.9%7.1%1.4% Communication5.7%11.4%74.3%8.6%

Top 5 Challenging Sub-competencies PC-4: address issues…that remain over time w/o clear dx SBP-1: provides cost-conscious medical care SBP-3: advocates for individual and community health PBLI-1: locates, appraises, assimilates evidence PBLI-3: improves systems in which physician provides care

“best practices” - infrastructure Using an electronic format to assess milestones: Milestone-based rotation evaluations Milestone reports from electronic platform Identification of expected levels for each class

“best practices” – education Faculty development Resident education Self-assessments

“best practices” - prep Review of resident data prior to CCC By coordinator, advisor, faculty member, etc. Meetings prior to CCC to review data Pre-Identification of which member to review which resident

“best practices” the CCC CCC Milestone assessment by PGY Class Using excel worksheets/handouts/and other tools to organize data & projecting data on screen at meetings

Do you have any best practices to share?

Jennifer Middleton, MD, MPH, FAAFP Associate Program Director Riverside Methodist Hospital Family Medicine Residency Sherrilyn M. Sethi, D.MH Assistant Program Director Worcester Family Medicine Residency Pamela Vnenchak, MD Deputy Program Director Lancaster General Family Medicine Residency Sara Beth Karp, MD, FAAFP Family Medicine Faculty Mt. Sinai Beth Israel Family Medicine Residency

PROGRAM 8 members APD: Chair PD Program Administrator FM faculty GME director 6 Semi-Annual Meetings 60 Minute Meetings 1 meeting for each PGY class twice a year approx. 10 min/resident. Ongoing gathering of resident data by Med Ed makes prepping for CCC meetings (fairly) painless CCC Membership Best practices CCC Meeting CCC Membership CCC Process PCMH community health center on the campus of a large tertiary care hospital Pre-Prep: Data to advisors: Rotation evals Pt #s/metrics Procedure log Advisor assigns prelim milestones Presentation: Post-Process Review prelim milestones; discussion Report to PD -> discusses with biannual review

3 Semi-Annual Meetings 90 Minute Meetings Arranged by PGY Class Each Resident reviewed in 7-10 mins. CCC Membership CCC Meeting 9 Members (6 reviewers) APD: Chair PD Program Administrator 3 Health Center Education Dirs. 3 Associate PDs CCC Membership CCC Process PROGRAM Community Based Health Centers BEST PRACTICES Pre-Prep: (3wks) Presentation Post-Process Presentation Schedule Data: - Dashboard - Evalue Reports Milestone Packet Data projected Milestone Packet Discussion Data Collection Milestone Report Milestone Self- Assessment CCC Education Mapping Milestones to Curriculum Milestone Packet EValue Milestone Summary Report

BEST PRACTICES CCC Education Mapping Milestones to Curriculum EValue Milestone Summary Report

PROGRAM 5 Faculty members plus 1 administrative support person No formal chair Minimum of 3 must be present at meeting Res & Advisor meet Fall & Spring for Semi-annual Review that includes Milestones Summary, Res self rating & advisor rating submitted to CCC The final ratings of CCC are shared with resident & advisor. mtgs over 1-2 weeks ~35 hrs total Begin with R1s CCC members review SAR summary, Res & Adv ratings ahead, full res mtg NewInnovations min /res Resident self assessment Key role of advisor CCC standardizes ratings criteria CCC Membership Best Practices CCC Process CCC Membership CCC Meeting Community Based 1 Rural Family Health Center & 1 Urban Family Health Center (residents rotate through both) New Innovations Residency Suite for evals, Milestones

PROGRAM 4 core Members FM Fac : Chair PD Behavioral Health Program Administrator Excel spreadsheet with color coding “met/not met” can be viewed by resident or by class Intensive prep and support of advisors as they synthesize milestones CCC Membership BEST PRACTICE CCC Meeting CCC Membership CCC Process Urban FQHC Pre-Prep: Presentation: Post-Process Staff and CCC prep of files Each advisor synthesizes milestones and other data Advisor presents synthesis Faculty discussion Advisor meets with resident to review CCC “cleans up” Cluster of weekly meetings around Semi- Annual Evals 4 weeks in a row 2 hours Meetings attended by all faculty

BEST PRACTICES

NEXT STEPS finish survey data analysis disseminate findings define and test “best practices” What do you think we should be doing next?

JOIN US! It’s not too late to participate in our survey! Or to join our learning community!

Questions/comments?