1. Wendy M. Helkowski, M.D. Program Director University of Pittsburgh Medical Center (UPMC)

2.  The program director must appoint the Clinical Competency Committee. (Core)  V.A.1.a) At a minimum the Clinical Competency Committee must be composed of three members of the program faculty. (Core)  The Clinical Competency Committee should:  V.A.1.b).(1).(a) review all resident evaluations semi- annually; (Core)  V.A.1.b).(1).(b) prepare and assure the reporting of Milestones evaluations of each resident semi-annually to ACGME; and, (Core)  V.A.1.b).(1).(c) advise the program director regarding resident progress, including promotion, remediation, and dismissal. (Detail)

3.  Large program (34 residents)  Half enter program as PGY-1  Large faculty(approx 30)  1 PD, 1 Associate PD, 1 Assistant PD  Bi-monthly Program Evaluation Committee (PEC) meetings  All teaching faculty are invited to the meeting

4.  9 faculty members (selected by PD) and 3 PDs  Meetings held after Bi-monthly PEC Evaluation of individual resident competency concerns including remediation and dismissal Mid-year and end of year progress assessments for all residents

5.  Semi-annual review of evaluations for all 34 residents  Selection of 19 milestone levels for all 34 residents  Total estimated time per resident = 30 minutes  Total estimated time for all 34 residents = 17 hours semi-annually!  12 faculty members spending 2 full days evaluating milestones twice a year??!!

6.  Educate the CCC faculty about the milestones  Improve efficiency and accuracy of milestone level selection  Educate the entire residency faculty about the milestones  Improve efficiency of evaluation review  Reduce time commitment for non-PD faculty  Reduce time commitment for PD faculty

7.  What are they?  Actual milestones tables emailed to the CCC  Selected milestones samples reviewed at PEC meetings  CCC members asked to do a pilot where they completed the set of 19 milestones for one resident using recent evaluations  How will they be used?  Submitted to ACGME semi-annually  Eventual comparison of program data with national data  What are the CCC responsibilities related to the milestones?  Group evaluation semi-annually to determine an appropriate milestone level for each resident in the program

8.  Pilot of CCC milestones assessment 1 year ago  our current evaluations did not provide sufficient data on specific milestones to accurately score a level  Modification of end of rotation evaluations to include milestones language  Modification of semi-annual self evaluations to include milestones language  Repeat pilot using modified evaluations 6 months ago  Improved ability to select a milestone level  Still challenging for the PGY-1 residents because of variable evaluation formats

10.  Divided the CCC into 3 subgroups with separate meeting times  PGY-1 and PGY-2 (3 faculty and 3 PDs)  PGY-3 (3 faculty and 3 PDs)  PGY-4 (3 faculty and 3 PDs)  Prior to the meeting, each member is assigned 1- 2 residents to review and complete the draft electronic milestone form  All evaluations (global, peer, self, multi-source, patient)  SAE scores  Utilization of institution’s electronic evaluation system which includes a milestones program

12.  Each CCC member presented the information on his/her assigned residents  CCC determined actual level  Level was changed in about 1/3 of the items based on discussion and personal experiences with the residents  9-10 residents reviewed in 2 hours  Total time estimate – about 10 hours for PDs and 3 hours for faculty semi-annually

13.  Don’t expect all CCC faculty to come to the first meeting completely prepared  It is helpful to have faculty on the CCC who have worked with the resident during the evaluation period  However, if there was one negative experience by a CCC member and other evaluations are good, it is challenging to have the one CCC member agree to a higher score

14.  Use first CCC milestones assessment as a learning opportunity for the group  Helpful hints about the electronic evaluation program  Discuss differences in end-of-rotation evaluations and semi-annual evaluations of the milestones OUR end of rotation evaluation language leaves out “across a spectrum of ages and impairments” – so the resident may reach a level 4 on one rotation but the CCC should not assign a 4 until the resident has demonstrated this ability across the spectrum

15.  It is helpful to have the actual milestones available at the meetings with review of the language and discussion when there is a discrepancy on the various rotation evaluations  Helpful with “halo effect” and when limited data is available  Choose faculty who reliably complete documentation for the residency program  Consider scheduling CCC meetings early (Nov/May)

16. Thank you!