1. Preparing for the ACGME's Next Accreditation System (NAS) A Prospective View for Family Medicine Residencies Joseph J. Brocato, PhD, University of Minnesota Barbara Joyce, PhD, Oakland University School of Medicine Andrea Pfeifle, EdD, University of Kentucky Memoona Hasnain, MD, MHPE, PhD, University of Illinois at Chicago Allyson Brotherson, MD, Hennepin County Family Medicine Residency Program Stephen A. Wilson, MD, MPH, UPMC St Margaret Family Medicine Residency

2. Session Objectives By the end of this session, participants will be able to: Describe the core elements of the FM Milestones State the six steps to follow now to begin to prepare for NAS Identify several core assessment methods necessary to meet requirements for the Milestone Project and self-assess gaps List some of the resources available to assist with Milestone integration Describe the role and function of the Clinical Competency Committee State the elements of a model for evaluator faculty development useful as a starting point for local residency faculty development Construct a NAS action plan (table-based small group activity) for use at their home institution

3. Disclosure We are showing examples of how residency management software may help Clinical Competency Committees manage data. We are not endorsing any residency management software We encourage you to work with your particular vendor

4. Developing a Program Prescription ✔ (1) Define and select core faculty ✔ (2) Optimize annual update and board scores ✔ (3) Learn about Milestones ✔ (4) Create Clinical Competency Committees (CCC) ✔ (5) Integrate GME and Quality Safety ✖ (6) Develop an action plan for faculty development

5. Milestones “Observable developmental steps” (i.e. benchmarks) Organized by competency Specialty-specific Provides common language for discussion of progress Resident achievement of a milestone is assessed by the Clinical Competency Committee (CCC) every 6 months RRC tracks program performance and benchmarks against national specialty data

6. Milestones (con’t) Will supplement current assessment tools used by program (for now) – CCC will review the resident’s performance using the program’s assessment tools and determine whether the resident has met the milestone(s) – Milestones will have pre-defined behavioral anchors – In 2013 the specialties will begin to roll out their own assessment tools that link to the Milestones Assessments will be sent to ACGME semi-annually Residents will be compared with a cohort from their program until sufficient data is available

7. Ex: The Family Medicine Milestones

8. Making Sure the Basics are in Place

9. ** Required

11. Putting it Another Way… Cognitive ** Clinical Performance Ratings ** Multisource Feedback ** Direct Observation ** Portfolio Procedure/Case Logs** Patient Satisfaction Surveys Simulation/OSCE Other *Required

12. Self-Assessment Complete self assessment handout to determine what assessment tools are needed in your program

13. Milestone Resources Assessment Tools STFM – https://www.stfm.org/rctoolkit https://www.stfm.org/rctoolkit FDMRL – http://www.fmdrl.org http://www.fmdrl.org Meded Portal – https://www.mededportal.org https://www.mededportal.org Holmboe E, Hawkins R. A Practical Guide to the Evaluation of Clinical Competence. Mosby. Philadelphia. 2008 Residency Management Systems (RMS), such as New Innovations

15. Milestone Reviews

17. Milestone Reviews Individual Progress & Benchmark

18. Clinical Competency Committee

19. Faculty Development… Moving Forward Working knowledge of core assessment concepts Frame of Reference Training Performance Dimension Scaling Rater Error Training Direct Observation of Competence Training Giving Feedback Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

20. Basic Principles of Assessment Introduces faculty to basic principles of assessment and sets foundation for more specific areas in the assessment domain Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

21. Frame of Reference Training Introduce faculty to the FM Milestones Promote discussion on use of Milestones Promote discussion on what is “satisfactory” performance on a milestone Creates a shared frame of reference for faculty expectations of milestone performance which can be communicated to resident Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

22. Performance Dimension Scaling Faculty development sessions requiring faculty to define and form consensus on what various rating criteria mean on resident assessment tools Faculty development sessions requiring faculty to define and form consensus on rating criteria for milestones Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

23. Rater Error Training Leniency error Severity error Halo error First impression error Friendship bias Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

24. Direct Observation of Competence Engage faculty in discussion of what to look for when conducting direct observation of resident Create checklist for faculty to provide feedback to resident

25. Giving Feedback Training Giving formative and summative feedback Managing the challenging resident

26. Faculty Facilitators Andrea Pfeifle, EdD, University of Kentucky Memoona Hasnain, MD, MHPE, PhD, University of Illinois at Chicago Allyson Brotherson, MD, Hennepin County Family Medicine Residency Program Stephen Wilson, MD, MPH, UPMC St Margaret Family Medicine Residency Holmboe E, Hawkins R. Evaluation of Clinical Competence. Mosby Elesevier. Philadelphia PA. Pg. 219

27. Faculty Development Action Plan 1.What skill sets do you feel faculty need to have in the area of assessment? 2.Are there additional faculty development topics needed to prepare your faculty for the FM Milestones?

28. Contact Joseph Brocato, Ph.D. – broca003@umn.edu broca003@umn.edu Barbara Joyce, Ph.D. – joyce@oakland.edu joyce@oakland.edu