1. Conflict of Interest Kim Walker – No conflicts of interest to disclose
Ann Dohn – No conflicts of interest to disclose
Nancy Piro – No conflicts of interest to disclose

2. PC002c Coordinators and Clinical Competency Committees: How to Streamline and Support the Work of your Program’s CCC
Kim Walker, PhD
Program Manager/Education Specialist
Ann Dohn, MA
DIO & GME Director
Nancy Piro, PhD

3. Session Outcomes Participants will be able to:
Identify new aspects of the coordinators’ evolving role in program administration.
Understand and use program requirements as a guide for planning, organizing and implementing educational and assessment tools.
Develop and utilize a newly developed comprehensive resident performance profile tool to streamline the work of the CCCs.

4. New Role

5. Evolving Role for Coordinators in Evaluations
Education & Evaluation Coordinator/Manager
Extraordinaire
Administrator Scheduler Supreme
Constructing new milestone evaluations to pilot/deliver
Reviewing evaluation completion data for accuracy
Aggregating data for the CCC from multiple sources and forms
Milestone data to ACGME
Deliver evaluations
Develop evaluation forms for PDs to approve
Schedule semi annual evaluations
Ensure summative evaluations completed and filed

6. Now I’m really confused!
Outcomes
Evaluations
CCCs
EPAs
Clarify and briefly define, especially the different between EPAs and milestones.
Milestones
Goals and Objectives

7. Where do I begin?

8. Know the NAS Building Blocks: Concepts defined
Core Competencies
Milestones
EPAs
Curriculum and Evaluations
Clinical Competency Committee (CCC)

9. NAS – Next Accreditation System
What is NAS – in a nutshell:
“an outcomes-based accreditation process through which the doctors of tomorrow will be measured for their competency in performing the essential tasks necessary for clinical practice in the 21st century.”

10. Major Changes: Accreditation based on…
Pre-NAS
Competencies
Site Visits – Up to 5+ year cycles
Internal Reviews
ADS Updates
PIFs
Resident Surveys
Current (New) NAS
Competencies with Milestones
Self-Studies at ~ 8-10 year intervals
Detailed ADS Updates
CLER Visits ~ 18 – 24 months (Institution)
Resident & Faculty Surveys

11. The New Accreditation System (NAS)… Outcomes
Increased Annual reporting by Programs (online)
Reduced volume of accreditation demands … but increased attention to accuracy and completeness of information submitted online
PIF-less Surveyor visits (unless new application)
Two Field Surveyors per visit
No Faculty CVs (only PD)….but Faculty & Resident Scholarly Activity required.

12. The Six AGME Core Competencies
Patient
Care
Interpersonal & Communication Skills
Medical
Knowledge
Six Core
Competencies
For Quality
Patient Care
Practice-based
Learning &
Improvement
Need to redo- Should be PBL&I
Professionalism
Systems-based Practice

13. What Are Milestones?
High Level - Milestones are simply defined as areas of competency/expectations for our trainees
Linked to six core competencies
Defined as a continuum of progressive growth/learning
Advanced Beginner PGY-1
Competent PGY-2 and 3
Proficient Practitioner PGY-4/5
Expert Practitioner - Ongoing
Each trainee assessed with respect to level for each competency

14. Dreyfus Model (1980): Stages of developing expertise
Source: Eraut, M. Developing Professional Knowledge and Competencies. (1994)

15. Milestone Level Definitions
Level 1: The resident is a graduating medical student/experiencing first day of residency.
Level 2: The resident is advancing and demonstrating additional milestones.
Level 3: The resident continues to advance and demonstrate additional milestones; the resident consistently demonstrates the majority of milestones targeted for residency.

16. Milestone Level Definitions (continued)
Level 4: The resident has advanced so that he or she now substantially demonstrates the milestones targeted for residency. This level is designed as the graduation target – not requirement.
Level 5: The resident has advanced beyond performance targets set for residency and is demonstrating “aspirational” goals which might describe the performance of someone who has been in practice for several years. It is expected that only a few exceptional residents will reach this level.

17. Reporting the Milestones
V.A.1.b).(1).(b) prepare and assure the reporting of Milestones evaluations of each resident semi-annually to ACGME (Core)
Milestones are reported directly through ADS
Reporting windows are:
November 1- December 31
May 1- June 15

18. Entrustable Professional Activities (EPA)
Professional life activities that define a medical specialty:
Ground the competencies in a physician’s everyday work
Activities lead to some outcome that can be observed
Complexity of the activities requires an integration of knowledge, skills and attitudes across competency domains
How EPAs Relate to Milestones
Situates competencies in the clinical, authentic context in which they are demonstrated
Aligns what is assessed with what physicians really do in the realm of patient care
Adds meaning to assessment by focusing on integration of competencies in care delivery

19. Examples of EPAs
Facilitate handovers to another healthcare provider either within or across settings
Contribute to the scholarly work of the subspecialty
Co-manage patients with generalists and other subspecialists
Source:

20. Curriculum: Rotation-specific goals and objectives & links to milestones

21. Milestones Impact on Evaluations: Linking questions to milestones
Step Two: Ensure specific evaluation questions are linked to milestones
Advises the referring health care provider(s) about the appropriateness of a procedure in routine clinical situations

22. Milestones Impact on Evaluation System
Allows for more objective methods of assessment and provide better feedback
Provides a process for early identification of residents that are having difficulties
All old and new evaluations and questions should be aligned with and tracked to milestones

23. Clinical Competency Committee (CCC)
V.A.1. The program director must appoint the Clinical Competency Committee.(Core) V.A.1.a) At a minimum the Clinical Competency Committee must be composed of three members of the program faculty.(Core) V.A.1.a).(1) Others eligible for appointment to the committee include faculty from other programs and non- physician members of the health care team.(Detail)
ACGME Common Program Requirements
Approved: February 7, 2012; Effective: July 1, 2013
Approved focused revision: June 9, 2013; Effective: July 1, 2013

24. Clinical Competency Committee (CCC)
V.A.1.b).(1) The Clinical Competency Committee should: V.A.1.b).(1).(a) review all resident evaluations semi- annually; (Core) V.A.1.b).(1).(b) prepare and assure the reporting of Milestones evaluations of each resident semi-annually to ACGME; and, (Core) V.A.1.b).(1).(c) advise the program director regarding resident progress, including promotion, remediation, and dismissal.(Detail)

25. Clinical Competency Committee (CCC)
V.A.1.b) There must be a written description of the responsibilities of the Clinical Competency Committee.(Core)
The reason for a required written description is so that every participant knows what his or her responsibility is to the CCC, and to ensure a fair process that all the members and the program director agree to follow. The responsibilities may go beyond what is listed in the ACGME Program Requirements. For some programs, the CCC will also be the Curriculum Committee or the Program Evaluation Committee, or may exist with a different name with additional responsibilities.
ACGME Common Program Requirements
Approved: February 7, 2012; Effective: July 1, 2013
Approved focused revision: June 9, 2013; Effective: July 1, 2013

26. Clinical Competency Committee (CCC): Written description

27. Clinical Competency Committee (CCC)
How the CCC does its work can be decided by the Program Director
Subcommittees
Assigning residents to faculty members for pre-review
Pre-review work will vary
Scheduling and frequency of meetings
Advisory to Program Director
Does not vote
Confidential
Assessment of all residents
Data provided includes summary evaluations plus resident self-assessment
Provide narrative to PD

28. What Should a CCC Do First?
Understand their specialty Milestones (Posted on acgme.org)
Decide how to assess the Milestones – Program Evaluation Strategy
If necessary, identify new evaluation tools from program director associations, societies, colleges

29. NAS and Milestones and YOU
The program coordinator will play a crucial role in developing, implementing, collecting data on and reporting of milestone evaluation tools.
You are a big part of the equation in how you are able to support your program’s mission at all levels… many balls to juggles requires many hands, many minds. You are the “coordinator” of it all!!!

30. Managing it all…

31. U - R - IT! Understanding RRC program requirements
Requirements applied to evaluation methods/process
Implementing new evaluation system
Tracking completion and accuracy (outliers) for data aggregation

32. 1. Understand Your Program’s New Requirements
Core
Outcomes
Details

33. 1. Understand Your Program’s New Requirements
Each standard/requirement is categorized:
Outcome - All programs must adhere
Core - All programs must adhere
Detail – Considered mandatory for new programs and those that fail to meet core requirements. Allows high-performing programs to innovate.
The focus in the NAS is on educational outcomes. The common and specialty programs
requirements were categorized, with the expectation that programs that demonstrate good
educational outcomes will not be assessed for compliance with the “detail” requirements.
“Detail” requirements will be considered mandatory for new programs and for programs that
have failed to meet expectations for outcomes (and have an accreditation status of “Probation”
or “Continued Accreditation with Warning”), and are intended to offer these programs added
guidance.
Allowing high-performing programs the freedom to meet the detailed requirements with
alternatives will provide such programs the opportunity to innovate.
From : ACGME – NAS FAQs
Source: Implementing The Next Accreditation System ACGME Webinar
John R. Potts, III, M.D.: 4 November 2013

34. 1. Understand Your Program’s New Requirements
Example Program Requirement:
VI.B. Transitions of Care
(Core)
(Core)
(Outcome)

35. 1. Understand Your Program’s New Requirements
Example Program Requirement: VI.B. Transitions of Care
When core and outcome not in compliance, then:
Details

36. U - R - IT! Understanding RRC program requirements
Requirements applied to evaluation methods/process
Linking milestones/EPAs and objectives to evaluation questions
Utilizing milestone scales
Implementing new evaluation system
Tracking completion and accuracy (outliers) for data aggregation

37. 2. Requirements Specific to CCC Review of Trainee
Aggregating/compiling multiple evaluations of individual trainees (V.A.1. Formative Evaluation)
Tracking trainee participation in conferences, journal clubs, didactics (IV.A.3. Didactic Sessions)
Monitoring duty hour compliance (VI.G. Duty Hours)
Reviewing involvement in quality improvement and patient safety activities (IV.A.5.c. PBLI)
Reviewing scholarly work (IV.B. Scholarly Work)
Monitoring and reporting procedure logs (IV.A.5.a)

38. U - R - IT! Understanding RRC program requirements
Requirements applied to evaluation methods/process
Linking milestones/EPAs and objectives to evaluation questions
Utilizing milestone scales
Implementing new evaluation system
Tracking completion and accuracy (outliers) for data aggregation

39. 3. Implementing Evaluation systems Milestone-based/EPAs
Rotation-specific
Patient handovers
Define evaluator groups (faculty, staff, patients)
Set up and timing of delivery systems

40. 3. Implementing Documentation and reporting systems for:
Conference attendance
Scholarly work (Learning Portfolios)
Quality Improvement and Patient Safety (Learning Portfolios / Safety reporting systems)
Duty Hours (recording, monitoring, reporting)
Case Logging (if applicable)

41. U - R - IT! Understanding RRC program requirements
Requirements applied to evaluation methods/process
Linking milestones/EPAs and objectives to evaluation questions
Utilizing milestone scales
Implementing new evaluation system
Tracking completion and accuracy (outliers) for data aggregation

42. 4. Tracking and Reporting…
Start with the end in mind:
CCC biannual reporting windows to ACGME
November 1- December 31 / May 1 - June 15
Back track and set calendar events for:
Periodic monitoring of evaluation completion
Running aggregate reports and reviewing milestone evaluation data
Reviewing case logs, learning portfolios, duty hours

43. Pulling the Data Together
Quality Improvement Activities
In-service training exams
Clinical Competency Committee
End-of-Rotation Evaluations
Safety Incident Reports
Case Logs
Patient/ Family Evaluations
Clinical Skills Assessment
Nursing and Staff / Techs Evaluations
Progress on Milestones
Sim
Lab

44. Managing it all: How will I pull this off?

45. The Toolbox

46. Creating a Resident Performance Profile Goals to support your CCC
Resident performance data that is:
Comprehensive
Consolidated / Aggregated
Easy for CCC to identify strengths, areas for improvement, opportunities for advancement

47. Creating a Resident Performance Profile: Compiling and centralizing data
Complete with auto-fill colors

48. Creating a Resident Performance Profile: Apply visual formatting for trends
Complete with auto-fill colors

49. Creating a Resident Performance Profile
Step 1 – Defining what to track

50. Creating a Resident Performance Profile
Step 1 – Defining what to track

51. Creating a Resident Performance Profile
Step 1 – Defining what to track

52. Creating a Resident Performance Profile
Step 1 – Defining what to track

53. Creating a Resident Performance Profile
Step 1 – Defining what to track

54. Creating a Resident Performance Profile
Step 1 – Defining what to track

55. Creating a Resident Performance Profile Step 2 - Link data sources to milestones

56. Creating a Resident Performance Profile Step 3: CCC defines performance ranges
Example:
For all aggregate milestone evaluation scores for a PGY 3, the CCC has defined these ranges by PGY level in advance of the meeting:
At or Above Expectation: and higher
Below Expectation:
1.7 – 2.7
Remediation:
Below 1.7
Conditional formatting is a super helpful tool to use for visually presenting color coded data that “at a glance” can provide a general trend over a residents’ progress.
STRENGTH
WATCH
AT RISK

57. Creating a Resident Performance Profile Step 4 – Set conditional formatting
Conditional formatting is a super helpful tool to use for visually presenting color coded data that “at a glance” can provide a general trend over a residents’ progress.

58. Creating a Resident Performance Profile Step 4 – Set conditional formatting
Example: Aggregate milestone evaluation data cells
Conditional formatting is a super helpful tool to use for visually presenting color coded data that “at a glance” can provide a general trend over a residents’ progress.
Highlight cells to apply the conditional formatting

59. Creating a Resident Performance Profile Step 4 – Set conditional formatting
Set Ranges: > , < , between

60. Creating a Resident Performance Profile Step 4 – Set conditional formatting
Select, “Greater Than” “Less Than” or “Between” to Set Value Ranges
Choose the corresponding fill color (e.g., red, yellow, green)
Conditional formatting is a super helpful tool to use for visually presenting color coded data that “at a glance” can provide a general trend over a residents’ progress.

61. Resident Performance Profile: Step 5: Enter in data
Complete with auto-fill colors

62. Creating a Resident Performance Profile Visual trends and detailed data
Complete with auto-fill colors

63. More tools…

64. Leveraging Resident Management System (RMS)Tools, if Available
RMSs – becoming more feature rich
Curriculum
Goals and Objectives and learning outcomes by rotation
Teaching and Assessment methodologies
Evaluation tool development
Sharing between programs and institutions

65. Leveraging Resident Management System (RMS)Tools, if Available
Conference attendance statistics
Core competencies linked to specified conferences
Attaching conference materials for later reference
Procedures and levels; linked procedure evaluations

66. Leveraging Resident Management System (RMS)Tools, if Available
Resident portfolio tools
QI participation and outcomes
Scholarly Activity logs

67. Leveraging Resident Management System (RMS)Tools, if Available
Aggregate reporting and graphic summaries
Peer or departmental average, individual average, minimum and maximum scores, standard deviation or listing of all scores

68. Leveraging Calendaring and Task Management Software
Set “data gathering and reporting” appointments with yourself
Remember to start with the end in mind (e.g., CCC meeting dates)
Break down large tasks into smaller tasks to keep it manageable

69. Leveraging Calendaring and Task Management Software

70. Learning through Experience

71. When a CCC Meeting… Does go well Doesn’t go well Data
complete
organized
accurate
Cooperative, collaborative decision making
Efficient use of time
Sound valid conclusions aligned with data
Data
not complete
not organized
not accurate
PD or faculty member dominates meeting
Prolonged inefficient decision making with inability to gain consensus
Unsubstantiated/unreliable conclusions

72. Successful Resident Ranking PGY1- 4: Ready to graduate

73. PGY 1 Ranked at Graduation Level
Milestone range for a PGY1 should not be a 4.0, 4.5 or 5.0 … 1 ?

74. Improvement evident

75. We Should not be at this Point
Tracker can avoid this situation – PGY 7 being passed along although aggregate scores show areas for concern.

76. Beyond data…Creating a climate of CCC Success
Gentle Words of Wisdom
Tight efficient meetings
Ground Rules
Beware of Negative Group Think
Schedule firm standing meeting dates in advance
Reserve room of appropriate size with required audio-visual tools if needed
…..and have snacks

77. Use Technology to Your Advantage… You can be a ‘Rock Star’

78. Session Recap in a Nutshell…
Know your program requirements and follow them unconditionally
Use simple spreadsheet, calendaring and task organizational tools to manage, track and present resident performance data to your CCC
Resident education is a cyclical process – revisit and revise tools and processes each year

79. Questions

80. Contacts Kim Walker - kwalker5@stanford.edu
Ann Dohn -
Nancy Piro -