1. What do I need to know about the 10-Year Accreditation Site Visit?
by Lynne Meyer, PhD, MPH
University of Florida
College of Medicine
Graduate Medical Education Office
March 2018

2. Session Objectives
Discuss the Self-Study process and components as preparation for 10-year Accreditation Site Visit
Compare and contrast Clinical Competency Committees (CCC) and Program Evaluation Committees (PEC)
Identify key elements of an annual program evaluation (APE)
Discuss Program Improvement Projects (PIP) using Action Plans (AP)
Discuss what is required for the 10-Year Site Visit

3. SELF-STUDY
Discuss the Self-Study process and components as preparation for 10-year Accreditation Site Visit

4. Self-Study: Why? The Focus is on Drivers of Change for Improvement
In the past, citations from the RRC were used as drivers of change
With Self-Studies, programs use internal drivers for change

5. Self-Study: Who?
PEC or Self-Study Group -- The ACGME wants to see high resident/fellow involvement in the improvement process
Everyone should be on the same page, aware of priorities, and where you are heading as a program

6. Self-Study: Key Questions
Is the learning environment effective?
How do you know?
Have you achieved your desired educational outcomes?
Have you done your SWOT? -- analysis of strengths, weaknesses, and plans for improvement (continuous quality improvement)
Each program (core and subs) must complete its own self- study.

7. Q: What is the timing between the Self-Study and the 10-Year Site Visit?
A: months (realistically, at least 24 months)
For at least one annual program evaluation (APE) prior to the 10-year site visit
For time to make improvements
Note: For the 10-year accreditation site visit, the ACGME will not ask programs to provide any information on areas identified during the Self-Study that have not yet resulted in improvements

8. Self-Study: Tips for Large Programs
Consider sub-groups performing a mini-self-study on a specific topic and bring information back to the larger group

9. Self-Study: Components
Vision & Mission
PEC’s Annual Program Evaluations (APE)
Must include items under “Program Evaluation and Improvement” Program Requirements section
Program Aims/Goals (reviewed annually) & SWOT Analysis
Items/areas needing improvement are linked to action plans and tracked year to year
Need to describe how the 3-5 issues focused on were prioritized
Summary Statement that addresses taking the program to the “next level”
Statement of what was learned during the Self-Study process

10. Self-Study: Vision Vision and plans for next 5 years (250 words)
Vision Example: “To be the leading ________residency program in the Southeast United States among the nation’s leading public university programs.”
Plans (including Mission) answer the following questions:
What does the next level look like?
How do we get there (and when do we expect to get there?)
What help, resources, etc. are needed?

11. (distinctive competence) (what) (for whom) (how)
Self-Study: Mission
Mission – one sentence that describes plan to achieve vision
Example
“Through commitment to education, scholarship and safe, high quality patient care, we will improve the health and well-being of Gainesville, FL and referral area patients and their families by producing physicians who continue to learn and/or lead in whatever setting they choose for their career.”
(distinctive competence) (what) (for whom) (how)

12. One Minute Activity Load your program’s website
Do you have vision and mission statements?
Current?
Revisions needed?
Tips:
If you don’t have these statements, consider asking someone with strategic planning expertise to assist you in the process.
Honor everyone’s perspectives including nursing, medical education, research, residents/fellows at all levels of training, etc.
[keep it loaded, we are going to refer to your site again]

13. 10-Year Site Visit Preparation: Mission/Vision
Program leaders’ expectations for the program – ASK:
Do we have a vision and mission?
Can the PD recite the vision and mission?
Can the APDs/Chair recite it?
Are we confident our trainees can recite it?

14. UF COM Mission Who can state our College of Medicine’s Mission?
Institutional leaders’ expectations for the program
Q: How do your program mission/vision/aims/goals relate to the UF College of Medicine’s Mission and Goals?

15. One Minute Activity Load your program’s website
If you have vision and mission statements on your web site -- How well do they relate to the College of Medicine’s mission statement?

16. Self-Study: Components
Program Description -- Concise description (250 words)
Does your program’s description on your website align with your vision?
When is the last time your web site was updated?
Your web site is the first thing 10-Year site visitors review

17. Self-Study: Components
Program Aims/Goals
What does the program strive to “produce” [for what kinds of settings and roles]? (150 words)
How does our program differentiate itself from other programs in the same specialty/subspecialty
What are you good at, what is your ‘brand’?
What is your ideal program?
Does every stakeholder agree?

18. Questions to ask about your Program’s Goals/Aims
How do your goals align with your vision and mission?
Are your program goals SMART?
What outcomes do you have to indicate success of your goals?
How do your program goals align with the institutional goals?
Do you need to revise your program goals?
“A goal properly set is halfway reached” – Abraham Lincoln

19. Sample Aims/Goals by Common Attributes
Competence/Preparation for Practice in the Chosen Clinical Specialty: Train surgeons of outstanding ability and judgment who will be leaders in their areas of interest, and in the setting of their choice
Resident/Graduate Traits: Develop mindfulness, curiosity, patient advocacy, and a commitment to life-long learning, while being adaptable to future changes in the healthcare system
Attributes of Individuals Recruited: Recruit diverse candidates with an emphasis on giving opportunities to students who have a track record of overcoming barriers, and the potential to excel in terms of physician competencies, advocacy and leadership
Service to the Local or Other Specific Communities: Serve our surrounding community in a culturally competent manner, the program aims to recruit residents who reflect the diversity of the population it serves, OR, 70% of our graduates will remain in Florida
Specific Academic Practice/Research/Leadership Roles: Train the next generation of leaders in academic pulmonary and critical care medicine
Attributes of the Program and Its Learning Environment: Provide a culture that is supportive, respectful, and compassionate towards trainees, patients and colleagues

20. Three Minute Activity
Load your program’s website and discuss them with a person or two sitting by you.
What are ways you can measure your program’s aims/goals?
Do they need revision?
How can you track the data?
Are aims/goals related to your vision and mission? OR
How would you evaluate/measure/revise (if needed) these program aims/goals?
Educate residents to be excellent practitioners of medically directed anesthesiology in an anesthesia care team model
Train individuals for expertise in population health and care of the medically underserved
Produce excellent, independent practitioners who will be leaders in academic medicine
Train residents to enter primary care practice

21. Self-Study: Components
Activities/Projects
Activities that further the aims/goals of the program
Listing of actions or projects aligned with aims/goals (250 words)
Review your program’s action plans -- Which ones relate to your program’s aims/goals?

22. 10-Year Site Visit Preparation: Mission/Vision
ASK:
What are our program Aims/Goals?
Who all knows the program Aims/Goals?
What Program Improvement Projects (PIPs) relate to the program Aims/Goals?
What was the outcome of the PIPs, what PIPs are in process, what PIPs are being planned?
Reminder: For the 10-year accreditation site visit, the ACGME will not ask programs to provide any information on areas identified during the Self-Study that have not yet resulted in improvements

23. Self-Study: Components
“SWOT” analysis
Opportunities (250 words)
Threats (250 words)
Suggest an anonymous survey using Qualtrics or similar to ask housestaff & faculty to list at least top three items for each category (S – W – O –T), collate, then review
Significant changes over past 5 years (250 words) – looking backwards and where you’ve come to this point.

24. Self-Study: Process Self-Study Process Description (300 words)
Who is involved?
How was data was analyzed?
How were conclusions reached?
How were improvement areas prioritized? e.g.--
Categorize SWOT items
Important-Immediate attention
Important-Can do later
Important – Can’t do anything about it
SWOT Fishbone
Will it close an existing performance gap?
Effort/Impact Grid
Choose 3-5 areas for prioritization

25. Self-Study: Core Program Role
If core program – what was role with subspecialty programs? (150 words)

26. Self-Study: Learning
What did you learn (200 words)

27. Self-Study: Looking Backwards and Forwards
A five-year look back at changes in the program, and a five-year look forward
Review of program revisions and achievements (APE documentation)
Defining the five-year strategic plan
The approach to the self-study and who was involved
Review of the conduct of the self- study
Self-Study: Looking Backwards and Forwards

28. Self-Study: Looking Forward
The answer to the question “what will take this program to the next level?”
What does the next level look like?
How do we get there (and when do we expect to get there?)
What help, resources, etc. are needed?
Focus will be on long-term and sustainability

29. Self-Study: Executive Summary
Develop a Concise Self-Study Document
In addition to the detailed self-study document(s), also develop an “executive summary” version to be used for:
Further program improvement
Checking for progress during next APE prior to 10-year site visit
Documentation for program’s 10-year site visit
Point of comparison for after the 10-year site visit

30. CCC & PEC
Compare and contrast Clinical Competency Committees (CCC) and Program Evaluation Committees (PEC)

31. ACGME

32. Q: What is the fundamental difference between the PEC and CCC?
A: The CCC assesses each resident/fellow individually and provides recommendations to the program director and the PEC evaluates the program using multiple data sources including, but not limited to aggregate resident/fellow performance data

33. Program Evaluation Committee
Role: To assess the quality of the program
Aggregate and analyze data
Prioritize, plan, develop, implement, and evaluate program’s educational activities
Review and make recommendations for revision of competency-based curriculum goals and objectives
Address areas of non-compliance with ACGME standards
Review the program at least annually using evaluation of faculty, residents, and other
All PEC members (including residents/fellows) should be familiar with the following ACGME documents:
Program Requirements, FAQs, and other “Documents and Resources” found on the RRC website
Milestones
The Milestones Guidebook (Version 2016)
Milestones Guidebook for Residents and Fellows (June 2017)
The PEC may need to meet more than once a year

34. Clinical Competency Committee
Role: To assess the clinical competency of the fellow
Review all fellow evaluations semi-annually
Prepare and ensure reporting of Milestones to ACGME semi-annually
Advise the PD regarding promotion, remediation or dismissal of trainees
All CCC members should be familiar with the following ACGME documents:
Clinical Competency Committees: A Guidebook for Programs (2nd Edition)
The Milestones Guidebook (Version 2016)
CCCs may need to meet more often to review specific residents/fellows in trouble

35. ACGME

36. Q: How many competencies are there? Q: List them
Developing Competence as a Physician
Patient Care and Procedural Skills
Medical Knowledge
Professionalism
Interpersonal & Communication Skills
Systems-based Practice
Practice-based Learning and Improvement
A: Six

37. Q: What is the difference between a subcompetency and a milestone?

38. Q: What is the difference between a subcompetency and a milestone?

39. Q: What is the difference between a subcompetency and a milestone?

40. APE
Identify key elements of an annual program evaluation (APE)

41. APE: Template in New Innovations

42. The PEC is responsible for the APE
PEC Aggregates and Analyzes Data
Review data from various sources (see example sources below)
Identify common themes

43. PEC Prioritizes Program Improvement Projects
Determine 3-5 key strategic program improvement areas to help you reach your “next level”
Link these improvement areas to program aims/goals
What measurements will demonstrate success?
Determine what can reasonably be accomplished in (and 24 or even 36) months

44. PIPs and APs
Discuss Program Improvement Projects (PIP) using Action Plans (AP)

45. PEC Develops Action Plans (AP) for Program Improvement Projects (PIP)
Assemble teams of residents, faculty and staff to take ownership of action items to reach “next level” goals
Use a PDSA model for program improvements
Measure your change
How do you know you’ve been successful?
Some PIPs may require more than one AP
Some PIPs may require a “Plan B”
Complete the PDSA cycle
A: adopt, adapt, or abandon change decision is very important Do
Study
Act
Plan

46. Sample Action Plan (AP) in New Innovations
All action plans, prioritized or not, need to go into New Innovations “Action Plan” module (this makes it easy for your PEC to generate reports)
All action plans must be linked to a citation or concern
All Citations or Concerns must have an identified Source (RRC, PEC--internally identified, or GMEC--IPRC)
You can have one concern with several action plans, or one action plan may be linked to more than one concern
Be sure to document outcomes for APs

47. Action Plans for…. ACGME Citations & Concerns
Program Aims (connected to your program mission and vision)
If you have NOT achieved your program aims, you need action plans to help you achieve your program’s “next level”
Issues related to PEC’s annual program evaluation including ACGME survey results
Issues identified by the Institutional Program Review Committee (IPRC)

48. 10-Year Site Visit
Discuss what is required for the 10-Year Site Visit

49. 10-Year Site Visit: What do Site Visitors Review prior to the visit?
Your web site
Letters of Notification
ACGME resident and faculty surveys
ADS data
Submitted Self-Study documents

50. Eight Steps to Prepare for the 10-Year Accreditation Site Visit
Review PIP data identified and worked on during Self-Study (SS)
A single year’s Annual Program Evaluation
For tracking improvements longitudinally across multiple Annual Program Evaluations
Discuss SS Improvements with Stakeholders
Reassess Program Aims/Goals and SWOT analysis
Discuss Program Aims, Achieved Improvements, and SWOT assessment with Stakeholders

51. Eight Steps to Prepare for the 10-Year Accreditation Site Visit
Complete and Submit Summary of Achievements
Summary of Achievements Template
Self-Study Summary Update
Update ADS
Summary of Achievements, Self-Study Summary Update submitted via ADS 12 days prior to site visit

52. Eight Steps to Prepare for the 10-Year Accreditation Site Visit
8. Set and Confirm Site Visit Logistics
Site Visit Document List – (Review this document sooner than later)

53. Where are the documents I need?

54. Summary: Process Overview with Timing

55. Summary: 10-Year Site Visit Logistics

56. From Dr. Nasca’s CEO Address at the 2018 ACGME Annual Educational Conference, March 2, 2018

57. Questions?