1.  Nancy Piro, PhD – No conflicts of interest to disclose  Kim Walker, PhD - No conflicts of interest to disclose

2. PC004 BR10 Voice of the Coordinator: Your experience on what works and doesn’t work with CCC meetings Nancy Piro, PhD Education Specialist/Program Manager Stanford University Medical Center Kim Walker, PhD Instructional Designer School of Medicine, Stanford University

3. Confidential – For Discussion Purposes Only Session Overview  Introduction and background on multi-institution survey  Report on survey results from Program Coordinators at multiple institutions on the CCC process and outcomes −Emergent themes on coordinators’ experiences and their roles in the CCC  Quantitative outcomes  Qualitative outcomes −What coordinators tell us was ineffective in the CCC process −Best practices for coordinators supporting a successful CCC  Action planning for implementation of best practices

4. Confidential – For Discussion Purposes Only Objectives & Session Focus  This session will focus on understanding and discussing the results from a multi-institution survey on CCC practices from the Program Coordinators’ perspective.  At the conclusion, participants will be able to: − Review their own practices in light of the survey outcomes − Identify best practices that best suit their program or institution needs − Develop and take home an action plan enabled by these best practices to better facilitate the work of their CCCs

5. Confidential – For Discussion Purposes Only Background

6. Confidential – For Discussion Purposes Only Milestones

7. Confidential – For Discussion Purposes Only Clinical Competency Committee How the CCC does its work is decided by the Program Director

8. Confidential – For Discussion Purposes Only

9. Survey Idea – from Inspiration to Implementation

10. Confidential – For Discussion Purposes Only Survey – Fun Facts  What: − A 14 Question* Survey Monkey survey - Both quantitative (8) and (6)open ended/word questions(6) was developed to assess CCC experience so far with respect to CCCs  When and To Whom: − Between November 12, 2014 and January 15, 2015, the survey link was distributed to a wide sample of coordinators (both within Stanford and across the country to coordinators in the 15 largest institutions, and to PC networks in OB/GYN, Anesthesia, and Surgery)  Analytic Methods Used – Both Qualitative and Quantitative Analyses were used.  Response: 325 Program Coordinators (and 135 Program Directors) responded to our survey by January 15, 2015 * Questions detailed on the following slide

11. Confidential – For Discussion Purposes Only Survey Questions 1.How many members are on your CCC? 2.Were you a part of your CCC meeting ? 3.Please describe your role: 4.What’s your program? 5.How much time (on the average) per resident was spent on the milestone assessments? 6.How many residents or fellows are in your program? 7.Do you provide the CCC members any evaluation data to review before the meeting?

12. Confidential – For Discussion Purposes Only Survey Questions 8.There was no way we could have had an effective meeting without completing the pre-work. 9.Has your program taken any of the following actions to better align your training with your milestones? 10.Who is on your program’s CCC? Please check as many as apply: 11.What data did you provide to your CCC? Please choose as many as apply. 12.Please describe any aspects of the milestone evaluation process that you thought were particularly beneficial: 13.Are there any changes you’ll make before the next CCC meeting? 14.Lastly, what did we miss? Are there any other areas of concern or suggestions?

13. Confidential – For Discussion Purposes Only How to Apply to Your Institution CURRENT PRACTICE (Starting State) POSSIBLE PRACTICE (End State) CCC Membership and Size Faculty Pre-work Alignment of Milestones Evaluation Tools CCC Process Challenges

14. Confidential – For Discussion Purposes Only Q1. How many members are on your CCC?

15. Confidential – For Discussion Purposes Only Q2. Were you part of your CCC meeting?

16. Confidential – For Discussion Purposes Only Q3. PC’s Role in Meeting: Emergent Themes Assessment Data Collection/Distribution (n=113) Minutes / note taker (n=129)

17. Confidential – For Discussion Purposes Only Q3. PC’s Role in Meeting: Emergent Themes Administrative title, other (n=41)  “As a resource member, and as administrative support for the ccc.”  “Program Administrator. Prepared documents, discussed purpose of meeting, emphasized importance and positive impact this will have for both faculty and trainees.”  “Servant”

18. Confidential – For Discussion Purposes Only Q3. PC’s Role in Meeting: Emergent Themes Active participant/meeting facilitator (n=31)  “manage agenda with chair, provide materials. problem-solving, participate in discussion”  “Data collection & presentation of data, active participant in discussions, note taking”  “Active Participant and documentation” Provide input on trainee evaluation on pertinent milestones (n=25)  “Participate in discussions on the competencies from an admin perspective and personal interactions with allied health.”  “As per the requirements from the ACGME, I am not a full member. I am ex-officio. However, I attend all meetings, take minutes, and I do participate in the discussions, to the extent that I can contribute certain information and perspectives, regarding the fellows, which are helpful and pertinent to the discussion.”

19. Confidential – For Discussion Purposes Only Q3. PC’s Role in Meeting: Emergent Themes Passive Participant (n=7)  “I am a supporting member. I do not have a speaking role, but I do all meeting prep.”  “observer and note taker”  “Listening”

20. Confidential – For Discussion Purposes Only Q4. Distribution of Programs Surveyed

21. Confidential – For Discussion Purposes Only Q5. How much time (on the average) per resident was spent on the milestone assessments?

22. Confidential – For Discussion Purposes Only Q6. How many residents or fellows are in your program? Type# Range149 Median21.75 Mode16

23. Confidential – For Discussion Purposes Only Q7. Do you provide the CCC members any evaluation data to review before the meeting?

24. Confidential – For Discussion Purposes Only Q8. There was no way we could have had an effective CCC meeting without completing pre-work.

25. Confidential – For Discussion Purposes Only Q9. Has your program taken any of the following actions to better align your training with your milestones? (1 of 3) Provided additional learning experiences related to milestones? Aligned (or linked) current evaluations with the milestones?

26. Confidential – For Discussion Purposes Only Q9. Has your program taken any of the following actions to better align your training with your milestones? (2 of 3) Used spreadsheet software to aggregate and place all evaluation material together for the CCC? Modified the curriculum with respect to clinical rotations?

27. Confidential – For Discussion Purposes Only Q9. Has your program taken any of the following actions to better align your training with your milestones? (2 of 3) Added didactic sessions to address milestone subcompetencies or elements? Added new trainee evaluation tools?

28. Confidential – For Discussion Purposes Only Q10. Who is on your program's CCC? Please check all that apply.

29. Confidential – For Discussion Purposes Only Q11. What data did you provide to your CCC? Please choose as many as apply.

30. Confidential – For Discussion Purposes Only  Aggregate milestone evaluation scores  Board scores  Conference Attendance  Continuity clinic chart review  Curriculum Vitae  Didactic attendance  Duty Hour Summary  Duty Hour reporting compliance  Evaluation completion rate  In service exam scores  Learning modules completion  Medical records outstanding OP reports  Medical student evaluations  Mock oral results  Narrative evaluation comments  Peer Evaluations  Presentations  Previous evaluation  Procedure logs  Research projects  Self assessment scores  Spider diagrams with comparison data  Continuity clinic attendance Q11. Data included in the CCC Review: Overall Emergent Themes

31. Confidential – For Discussion Purposes Only Q12. Beneficial aspects of the milestone evaluation process CCC faculty assignment and pre-work 31 “Each member reviews and ranks each resident prior to the meeting.”

32. Confidential – For Discussion Purposes Only Q12. Beneficial aspects of the milestone evaluation process Discussions; collaborative committee assessment 32 “allows for a rich discussion among a variety of faculty who have interacted with an individual fellow…”

33. Confidential – For Discussion Purposes Only Q12. Beneficial aspects of the milestone evaluation process Aggregate and Comparative Data

34. Confidential – For Discussion Purposes Only Q12. Beneficial aspects of the milestone evaluation process Residency Management Systems (RMS)

35. Confidential – For Discussion Purposes Only "what are they really getting at here?" “contrived” “nebulous” “uncalibrated” Modifying or developing new evaluation forms Q13. Changes You Will Make: Emergent Themes

36. Confidential – For Discussion Purposes Only Q13. Changes You Will Make: Emergent Themes Streamline meeting process

37. Confidential – For Discussion Purposes Only Q13. Changes You Will Make: Emergent Themes No changes (!!)  “The sub committee was our 3rd iteration of the CCC process and seems to be working well for us now.”  “No. We did a trial run in the spring and did our official one in the fall. We seem to have a good system in place.“  “I feel prepared- CCC meetings thru 2016 are already scheduled… templates are already created… we have a system in place.”

38. Confidential – For Discussion Purposes Only Q14. Other thoughts: Emergent themes Approaches to CCC faculty review process  CCC Subcommittees by training level  Multiple Meetings by training level  Faculty reviews of residents across specific milestones

39. Confidential – For Discussion Purposes Only Q14. Other thoughts: Emergent themes “Food for thought” for ACGME:  “We believe the program coordinator's should be allowed to be full members of the CCC.”  “THE ACGME should modify the language and recognize that the coordinator plays a critical role in the success or failure of the CCC. Also, I believe rotating the chair every three years would be helpful and have discussed this with our PD at length. Especially when you have a chair of the CCC who is difficult to work with.”  “For the initial meeting to go back and look at all that data and initially place residents took an ENORMOUS about of pre-work on my part. Hopefully since that first data entry is done it will move easier. But I wish the ACGME addressed that and recognized how much effort that would require.”

40. Confidential – For Discussion Purposes Only Q14. Other thoughts: Emergent themes Recognizing time-consuming nature of work … and need for support

41. Confidential – For Discussion Purposes Only Overall Results Summary - from Coordinators and Directors  Overall Improved Evaluation and Documentation − Semi-annual evaluations improved − Documentation of trainee strengths and weaknesses  Time Consuming − Need leadership support − Administrative Resources − Ample planning time  CCC Membership Size − Share the burden! − Value of diverse perspectives  Faculty Roles − Responsibilities  Pairing faculty with trainees  Assigning faculty to become “milestone” experts

42. Confidential – For Discussion Purposes Only Lessons Learned / Best Practices It’s your turn and your voice that counts!!! How might you change your practice in light of these findings?

43. Confidential – For Discussion Purposes Only Questions?

44. Confidential – For Discussion Purposes Only Contact Information:  Nancy Piro, PhD; GME Program Manager/Education Specialist −npiro@stanford.edunpiro@stanford.edu  Kim Walker, PhD; Instructional Designer −kwalker5@stanford.edukwalker5@stanford.edu  http://med.stanford.edu/gme/GME_Community.html