The ACGME Self-Study and 10-Year Site Visit 2015 Fall Program Director Retreat September 24, 2015 Alan J. Smith, PhD, MEd Assistant Dean and Director for GME ACGME Designated Institutional Official

Disclosure Adapted from original ACGME Webinar presentation by Susan Guralick, MD, DIO, Winthrop University Hospital, and Ingrid Philibert, PhD, MBA, ACGME Senior Vice President for Field Activities, June 18, 2015.

Objectives Discuss elements of the self-study and ongoing program evaluation and improvement on which it is based Explain the sequence and components of the self-study and the 10-year site visit Describe program and institutional practices to assist with self-study preparation and execution Explain the components of the 10-year accreditation site visit and Review Committee review

Self-Study Scope Assess program performance and ongoing improvement efforts Facilitate improvement in areas already in substantial compliance with the accreditation standards Initial period: since last accreditation review Ultimately a 10-year interval Review improvement activities, successes achieved, and areas in need of improvement Based on successive Annual Program Evaluations, ACGME Resident and Faculty Survey data, and other relevant information, stakeholder input

Strengths and Areas for Improvement Important to acknowledge and celebrate What should definitely be continued, especially in an environment of limited resources Areas for Improvement identified by: Citations, areas for improvement, and other information from ACGME Annual Program Evaluation Other program/institutional data sources Expected: Longitudinal tracking of strengths and areas for improvement through successive Annual Program Evaluations

Improvement Viewed in Context: Program Aims, Opportunities and Threats Aims as a way to differentiate programs Set and reassessed as part of the Annual Program Evaluation Context: Opportunities and Threats External attractive factors that, if acted upon, will contribute to the program flourishing or may have a negative effect While the program cannot fully control them, it is beneficial to have plans to mitigate their effect

Defining Program Aims Aims are set and revisited/revised as part of the Annual Program Evaluation Relevant considerations Who are our residents/fellows? What do we prepare them for? Fellowship Academic practice Leadership and other roles Who are the patients/populations we care for?

Benefits of a Focus on Program Aims Suggests a relevant dimension of the program: Types of graduates produced for specific community needs, practice settings and roles Allows for a more “tailored” approach to creating a learning environment Focus on specific aims can produce highly desirable graduates that match patient and healthcare system needs 1 Enhances the focus on functional capabilities of graduating residents Fits with a milestones-based approach to assessment 1Hodges BD.”A Tea-Steeping or i-Doc Model for Medical Education?.” Acad Med85(9) Sept 2010, pp. S34-S44.

Defining “Opportunities” Opportunities are external attractive factors that, if acted upon, will contribute to the program flourishing What are capabilities for further evolving the program? How can the program capitalize on them? Has there been recent change in the program’s context that creates an opportunity? Are these opportunities ongoing, or is there a narrow window for them? How critical is the timing?

Defining “Threats” Threats include external factors that affect the program While the program cannot fully control them, it is beneficial to have plans to mitigate their effect What factors beyond the program’s control may place it at risk? What are the changes in residents’ specialty choice, regulation, or other factors that may affect the future success of the program? Are there challenges or unfavorable trends in immediate context that may affect the program? E.g., faculty burdened with heavy clinical load that prevents effective teaching and mentorship

Coordinated Self-Study for Core and Subspecialty Programs Coordination of curriculum and program resources Needs of core and subspecialty programs considered collectively Subspecialties can access core resources Core oversight of fellowships Access common strengths, areas for improvement Coordinated action plans for areas for improvement Increase efficiency Less time and resources spent, coordinated collection and review of data

Self-Study Process Who should conduct the self study? Not defined by ACGME Members of the Program Evaluation Committee (PEC) are the logical choice PEC membership may be expanded (beyond required two faculty members and one resident) Data gathering Many potential sources, including ACGME Resident and Faculty Surveys, program and institutional data Important: stakeholder (residents, faculty and relevant others) participation, input and perspective Data as evidence to support conclusions

Self-Study Data Gathering Annual Program Evaluation data, ACGME Resident and Faculty Survey data, other program and institutional data Focus on data gathering as a learning experience Evaluate strengths and areas for improvement Explore opportunities and threats Reflect stakeholder (residents, faculty, and relevant others) participation, input, and perspective Data as evidence to support conclusions

Tracking Improvements Design and implement solutions Identify individual or group that will be responsible Identify and secure resources Timeline Follow-up is key: endure all issues addressed Documentation to facilitate ongoing tracking A simple spreadsheet recording improvements achieved and ongoing (See handout 3) Record improvements over multiple years

Resident Participation Resident participation critical: They are the beneficiaries of the educational program They have first hand knowledge of areas that need improvement Double benefit: Residents help improve their own education Resident participation in “educational QI effort” can be used to meet the requirement for resident involvement in quality and safety improvement

Self-Study and the Self-Study Summary The program conducts its self-study After the self-study, the program uploads the Self-Study Summary in ADS Information on areas for improvement identified in the self-study are not included in the Summary Timing consideration: Programs expected to upload the self-study summary in the month the Review Committee indicated for their first site visit in NAS Self-Study Site Visit scheduled 12-18 months after uploading the Self-Study Summary in ADS

The Self-Study Summary Brief document prepared by the program, uploaded in ADS (See handout 1) ACGME Template: 2300 words (~4-5 pages) for core program, less for small subspecialty programs Sections: Key Self-Study dimensions Aims Opportunities and Threats Self-Study process Who was involved, how were data collected and interpreted How is ongoing improvement achieved through sequential Annual Program Evaluations Omitted by design: information on strengths and areas for improvement

The 10-Year Site Visit A full accreditation site visit Review of all applicable requirements 12-18-month period is by design, to allow programs to implement improvements “Summary of Achievements” ACGME template uploaded through ADS ~ 1200 words, describing key improvements accomplished from the self-study Program provides information ONLY on the improvements that were realized from their self-study, no information on areas that have not been resolved Program may provide an update to its Self-Study Summary

The 10-Year Site Visit Site visit by a team Site visit team provides verbal feedback Key strengths and suggestions for improvement Site visit team submits a written report through ADS for the Review Committee (RC)

Review Committee Review of the 10-Year Site Visit Available to the Review Committee: ADS data Program’s self-study summary and any updates Program’s summary of improvements achieved as a result of the self-study (which does not include data on areas still in need of improvement) The site visit report from the 10-year (accreditation) site visit Review of program aims and context from the self-study summary allows the RC to assess self-study effectiveness Improvements the program reports are paired with program aims and context from self-study summary Data on improvements achieved are one measure of effectiveness

Review Committee Review of the 10-Year Site Visit RC provides a Letter of Notification from the 10-year (Full Accreditation) Site Visit Citations and areas for improvement RC feedback on the self-study taken into consideration: Program aims and context Improvements reported and verified during the 10-year visit Effectiveness of the self-study, based on the improvements reported by the program Formative feedback (no accreditation impact) for the initial RC assessment of self-study effectiveness

University of Utah Program Self-Study Dates Anesthesiology + Pain Management October 2021 Dermatology July 2021 Emergency Medicine September 2016 Family Medicine May 2020 Internal Medicine + subspecialty fellowships May 2019 Medicine/Pediatrics Medical Genetics Neurology + subspecialty fellowships November 2021 Neurosurgery January 2018 OB/GYN December 2021 Occupational Medicine October 2019

University of Utah Program Self-Study Dates Ophthalmology May 2020 Orthopedic Surgery + subspecialty fellowships June 2019 Otolaryngology + subspecialty fellowships February 2020 Pathology + subspecialty fellowships April 2019 Pediatrics + subspecialty fellowships October 2022 PM&R Psychiatry + subspecialty fellowships Radiology + subspecialty fellowships November 2016 Radiation Oncology March 2018 Surgery + subspecialty fellowships June 2020 Urology + subspecialty fellowships December 2020 Plastic Surgery May 2021

Handouts Self Study Summary Template Suggested Annual Program Evaluation Template Suggested Annual Program Evaluation Action Plan and Follow-up Template JGME article: Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):502-505.

Sample Timeline for IM Core and Subspecialty Programs* Day 1. ACGME Notification of Self-Study Submission Due Date Week 1. Recruit Self-Study Planning Committee Program Director and Associate PDs Program Coordinator Fellowship Program Directors and Associate PDs Faculty member Resident Fellow GME staff? DIO? * Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):502-505.

Sample Timeline for IM Core and Subspecialty Programs* Weeks 2-5. Weekly meetings of the planning committee Independent work done by committee members between meetings Week 2. View ACGME self-study webinar (acgme.org) Review self-study process Develop self-study timeline for the program Create a self-study planning committee worksheet Week 3. Create/review Self-Study SWOT Template Add/edit items as relevant to local/individual programs * Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):502-505.

Sample Timeline for IM Core and Subspecialty Programs* Week 4. Create or adapt self-study survey draft documents Simple surveys: faculty, residents, fellows, coordinators, nurses & other relevant staff, others Week 5. Define the focus group process Week 6. Present Self-Study Planning work to GMEC/Discussion * Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):502-505.

Sample Timeline for IM Core and Subspecialty Programs* Weeks 6-8. Formation of Program-Specific Self-Study Committees Should include PEC members plus others as appropriate (e.g., CCC members, Program Coordinator, Chief Residents, Residents/Fellows, Medical Educator Week 7. Surveys Distributed Weeks 7-8. Survey Data Collected Weeks 8-9. Program-specific committees meet Program-specific self-study planning Define program aims and perform SWOT analysis Weeks 9-11. Survey groups participate in focus groups to delve into/clarify information gathered in surveys * Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):502-505.

Sample Timeline for IM Core and Subspecialty Programs* Weeks 14-17. Self-Study Program-Specific Committees Meet Weekly for: Data analysis SWOT analysis review with additional data collected Perform full self-self study Create self-study document Weeks 18-19. Write Self-Study document for submission to ACGME Weeks 19-20. Finalize Self-Study documents for submission. * Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):502-505.

THANKS!

Resident/Fellow Schedules in E*Value

Truth or Consequences . . . Providing an accurate resident / fellow training schedule to the institution is a foundational program responsibility Accurate, timely training schedules in E*Value are essential for: Affiliate billing (over $20M yearly) CMS/Medicare Reimbursement (over $30M yearly) Information/Data Integrity with Affiliates & Government

The Problem GME Office must routinely resolve 50-75 missing FTEs at month-end Discrepancies and late reporting of Leave of Absences, vacation days, or changes in training location are rampant GME dedicating an employee 3-4 full days each month to audit and resolve schedule omissions, conflicts and mistakes in E*Value Unprofessional, embarrassing after-the-fact corrections to affiliate bills are commonplace due to scheduling errors

Program Director Support Needed Ensure that your Program Coordinator / Manager knows where your people will be training before the rotation begins Ensure that training schedules are confirmed and input in E*Value by the 15th of each month Promptly report any changes that occur after the 15th to GME Encourage your Program Coordinator / Manager to utilize the monthly schedule verification checklist provided by GME and to double-check schedules before the 15th

We’d really rather not go there, but … The University currently assesses a charge to departments or individuals for payroll corrections or missing scheduled training classes. Please help us avoid implementing this type of ‘penalty’ system. Thank You!