Preparing for the ACGME's Next Accreditation System (NAS) A Prospective View for Family Medicine Residencies Joseph J. Brocato, PhD, University of Minnesota Barbara Joyce, PhD, Oakland University School of Medicine Andrea Pfeifle, EdD, University of Kentucky Memoona Hasnain, MD, MHPE, PhD, University of Illinois at Chicago Allyson Brotherson, MD, Hennepin County Family Medicine Residency Program Stephen A. Wilson, MD, MPH, UPMC St Margaret Family Medicine Residency

Session Objectives By the end of this session, participants will be able to: Describe the core elements of the FM Milestones State the six steps to follow now to begin to prepare for NAS Identify several core assessment methods necessary to meet requirements for the Milestone Project and self-assess gaps List some of the resources available to assist with Milestone integration Describe the role and function of the Clinical Competency Committee State the elements of a model for evaluator faculty development useful as a starting point for local residency faculty development Construct a NAS action plan (table-based small group activity) for use at their home institution

Disclosure We are showing examples of how residency management software may help Clinical Competency Committees manage data. We are not endorsing any residency management software We encourage you to work with your particular vendor

Developing a Program Prescription ✔ (1) Define and select core faculty ✔ (2) Optimize annual update and board scores ✔ (3) Learn about Milestones ✔ (4) Create Clinical Competency Committees (CCC) ✔ (5) Integrate GME and Quality Safety ✖ (6) Develop an action plan for faculty development

Milestones “Observable developmental steps” (i.e. benchmarks) Organized by competency Specialty-specific Provides common language for discussion of progress Resident achievement of a milestone is assessed by the Clinical Competency Committee (CCC) every 6 months RRC tracks program performance and benchmarks against national specialty data

Milestones (con’t) Will supplement current assessment tools used by program (for now) – CCC will review the resident’s performance using the program’s assessment tools and determine whether the resident has met the milestone(s) – Milestones will have pre-defined behavioral anchors – In 2013 the specialties will begin to roll out their own assessment tools that link to the Milestones Assessments will be sent to ACGME semi-annually Residents will be compared with a cohort from their program until sufficient data is available

Ex: The Family Medicine Milestones

Making Sure the Basics are in Place

** Required

Putting it Another Way… Cognitive ** Clinical Performance Ratings ** Multisource Feedback ** Direct Observation ** Portfolio Procedure/Case Logs** Patient Satisfaction Surveys Simulation/OSCE Other *Required

Self-Assessment Complete self assessment handout to determine what assessment tools are needed in your program

Milestone Resources Assessment Tools STFM – FDMRL – Meded Portal – Holmboe E, Hawkins R. A Practical Guide to the Evaluation of Clinical Competence. Mosby. Philadelphia Residency Management Systems (RMS), such as New Innovations

Milestone Reviews

Milestone Reviews Individual Progress & Benchmark

Clinical Competency Committee

Faculty Development… Moving Forward Working knowledge of core assessment concepts Frame of Reference Training Performance Dimension Scaling Rater Error Training Direct Observation of Competence Training Giving Feedback Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

Basic Principles of Assessment Introduces faculty to basic principles of assessment and sets foundation for more specific areas in the assessment domain Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

Frame of Reference Training Introduce faculty to the FM Milestones Promote discussion on use of Milestones Promote discussion on what is “satisfactory” performance on a milestone Creates a shared frame of reference for faculty expectations of milestone performance which can be communicated to resident Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

Performance Dimension Scaling Faculty development sessions requiring faculty to define and form consensus on what various rating criteria mean on resident assessment tools Faculty development sessions requiring faculty to define and form consensus on rating criteria for milestones Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

Rater Error Training Leniency error Severity error Halo error First impression error Friendship bias Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

Direct Observation of Competence Engage faculty in discussion of what to look for when conducting direct observation of resident Create checklist for faculty to provide feedback to resident

Giving Feedback Training Giving formative and summative feedback Managing the challenging resident

Faculty Facilitators Andrea Pfeifle, EdD, University of Kentucky Memoona Hasnain, MD, MHPE, PhD, University of Illinois at Chicago Allyson Brotherson, MD, Hennepin County Family Medicine Residency Program Stephen Wilson, MD, MPH, UPMC St Margaret Family Medicine Residency Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

Faculty Development Action Plan 1.What skill sets do you feel faculty need to have in the area of assessment? 2.Are there additional faculty development topics needed to prepare your faculty for the FM Milestones?

Contact Joseph Brocato, Ph.D. – Barbara Joyce, Ph.D. –