Residency Milestones Implementation 2015 Marc Tunzi, MD Associate Program Director Chief Family Medicine Service UCSF Natividad Family Med Residency Salinas, California Walt Mills, M.D., MMM, Associate Program Director RPS Consultant Steve Harrison, MD Program Director

Activity Disclaimer The material presented here is being made available by the American Academy of Family Physicians for educational purposes only. This material is not intended to represent the only, nor necessarily best, methods or procedures appropriate for the medical situations discussed. Rather, it is intended to present an approach, view, statement, or opinion of the faculty, which may be helpful to others who face similar situations. The AAFP disclaims any and all liability for injury or other damages resulting to any individual using this material and for all claims that might arise out of the use of the techniques demonstrated therein by such individuals, whether these claims shall be asserted by a physician or any other person. Every effort has been made to ensure the accuracy of the data presented here. Physicians may care to check specific details such as drug doses and contraindications, etc., in standard sources prior to clinical application. This material might contain recommendations/guidelines developed by other organizations. Please note that although these guidelines might be included, this does not necessarily imply the endorsement by the AAFP. Faculty Disclosure It is the policy of the AAFP that all individuals in a position to control content disclose any relationships with commercial interests upon nomination/invitation of participation. Disclosure documents are reviewed for potential conflicts of interest (COI), and if identified, conflicts are resolved prior to confirmation of participation. Only those participants who had no conflict of interest or who agreed to an identified resolution process prior to their participation were involved in this CME activity. All faculty in a position to control content for this session have indicated they have no relevant financial relationships to disclose.

Goals Simplify resident rotation evaluations to those Milestones with highest value and relevance Design the resident portfolio evaluations that provide timely, comprehensive, actionable data for the CCC Describe an innovative, multidisciplinary faculty development program to improve competence and participation in providing reliable evaluation and feedback using Milestones

18 mo baby *

Competencies as levels of advancement in martial arts Level I-white belt Level II-yellow belt Level III-green belt Level IV-brown belt Level V-black belt *

Practical understanding of competencies Level 0-not ready for internship Level I-advanced medical student or beginning intern: Not ready to do much without help Level II-end of first year: Ready to spread their wings and try their art *

Practical understanding of competencies- continued Level III-end of second-year: Reasonably good, not ready to care for your family Level IV-end of third year: you would recommend to a friend or family member Level V-someone who is as good or better than you *

Anatomy of a Milestone Medical Students

Natividad FMR Case Study Fall 2013 paper copies FM milestones used for first CCC meeting Advisers meet with the advisees and fill out the forms before the CCC meetings -only 1 advisor did that Significant lack of data was noted including missing rotation evaluations *

Case study continued CCC : PD, clinic liaison, year group liaison, behaviorist, and advisor when available Significant time spent rehashing the milestones format until all faculty were in agreement on process First CCC committee spent 8 hours on 8 residents *

Currently CCC committee spends 4 hours for 8 to 10 residents New innovations allows electronic selection of sub competence to be evaluated by rotations Not all sub competencies need to be evaluated in each rotation Who else discovered rotation evaluations with too many subcompentencies were not completed? *

Unexpected results There were several stealth residents (residents who were doing much better than we suspected) We had no system to evaluate PBL1 (analysis of the medical literature and/or statistics) until third year What Milestones have you found most challenging with your curriculum? *

Unexpected results Problem receiving evaluations that requiring huge effort to fix How do you handle delinquent evaluations? *

“Discovered” Difficult-to-Assess Competencies PC/MK are relatively easy to assess with traditional precepting models and standardized testing (ITE; ABFM) Professionalism, Interpersonal communications skills, Practice-based learning and improvement, System-based practice are less tangible,more difficult to assess. Many competencies and subcompetencies take place outside of the preceptor room and often in the patient room (direct observation)

Wanted Summative Data to Complete Reporting Forms Gather Information from different assessment tools to be reviewed by CCC Use multiple independent assessments from multiple evaluators Decrease effect of inter-rater bias Create more robust bank of information

Used STFM Assessment Tools onToolkit/MilestonesResidentAssessment/Assessment MethodsandTools

From the What to the How (What we did/are doing) Utilize synergy between different program structures: Clinical Competency Committee Program Evaluation Committee Faculty Development

Program Evaluation Committee Resident Survey Faculty Survey Evaluation and Feedback scored low Action Plan: Redesign/Improve Evaluation

Intern Orientation and Evaluative Curriculum Sessions scheduled throughout July Faculty assigned sessions based on curriculum oversight and identified areas of expertise Built within the curriculum are multiple resident evaluative tools Reviewed by the program director, the behavioral medicine faculty, and participating faculty Develop competency-based summative evaluations Individualized learning plans (ILP) targeting areas for growth or remediation. Individualized learning plans (ILP) targeting areas for growth or remediation. *

Use/Modify Existing Tools Many of the tools that each of our programs are already using can be used to assess resident progress relative to the Milestones. Share ideas for mapping your current assessment tools to the Milestones

Potential Opportunities for Teaching and Evaluation PGY 1 PGY 2 PGY 3 Other Ideas: Action Plan: Level 1Level 2Level 3Level 4Level 5 How, when and by whom is this content taught? When/by whom: When/by whom: When/by whom: When/by whom: How : How: How evaluated?

Program Evaluation Committee: Action Plan to Improve Rotation Evaluation“Map” Performance Improvement Primer for Residency Education. Mills, W. UCSF Natividad FMR

Natividad FMR New Innovations Redesigned Evaluations for each Rotation Assigned 3-6 Milestones to each Rotation Mapped all Milestones to Portfolio for CCC *

Use Technology to Reduce the Burden of Accreditation Our Residency Management Software may do more than we think! What features of your softwares has helped with implementation of the Milestones? *

All Faculty

The Traditional Precepting Model Worked for Medical Knowledge and Patient Care Competency Evaluation

With direct observation, observers are able to see what actually occurs in the patient room.

The Take Home Message: “Direct Observation” Critical to successful assessment of resident performance relative to the Milestones

Identify New Sources of Teaching and Assessment What sources of feedback and assessment are available outside of those traditionally considered in resident assessment? Core faculty Attending physicians on outside rotations Staff Patients Peers.

Lessons Learned Time to share your’s so far……………. Pair and Share your experience with Milestones, your CCC, Advising, New Resident Evaluation, other ?

COLLABORATION IS KEY Take Home Messages for Faculty Development *

Create a Culture of Excellence Each residency program has a unique teaching and assessment atmosphere. Fundamental questions to ask as you re-design your “Milestone Evaluation System”. Where are you now? learner-centered or teacher-centered adversarial or collaborative hierarchical or democratic tense or relaxed

Create a Culture of Excellence Assessing and documenting resident competence requires a change in culture and a change in processes Small changes without fundamental changes in thinking and structure may keep one compliant/accredited, but fall short of achieving Excellence

ONE LAST THING… Please complete the online evaluation.online evaluation

Questions? Walt Mills, MD Steve Harrison, MD