2. Outline

3. Introduction

4.  WHAT DID YOU SAY IN OUR SURVEY?

5. Training Level for Faculty 4.1% Fully Embraced 20.4% Partial Implement 55.1% Accept & started 6.1% No Implement 14.3% Unfamiliar ACGME Competencies

6. Implemented Competencies?

7. How Impacted?

8. What is Level of Resistance to Change?

9. Objectives: Participants will be able to:

10. 10 Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85: 787-790.

11. 11

12. Resident Competency Measurement Task Force Membership Chair Sam Cullison, MD, Task Force Chair - Swedish FMR Seattle Wendy Biggs, MD, AAFP Colleen Conry, MD, University of Colorado Alan David, MD, Medical College of Wisconsin Mike Donoff, MD, University of Alberta Julie Dostal, MD, Competency Assessment Work Group Lead, Lehigh Valley FMR Larry Mauksch, MEd, Faculty Development and Dissemination Work Group Lead, University of Washington FMR Tom O'Neill, PhD, ABFM Allen Shaughnessy, PharmD, Tufts University FMR Stephen Wilson, MD, UPMC St Margaret FMR Stacy Brungardt, CAE, Society of Teachers of Family Medicine 12

13. STFM Residency Competency Assessment Toolkit http://www.stfm.org/rctoolkit/ Presentations

14. RC change and the Milestones

15. Competency Assessment

16. Current RC Requirements (p 33): 16

17. 17

18. Proposed New RC Requirements – (on hold) 18

19. Proposed New RC Requirements – (on hold) 19

20. Formative Assessment

21. Formative Competency Assessment

22. Examples of Formative Assessment

23. Formative Assessment (coaching) Examples

24. Formative Feedback examples cont’d

25. ACGME Competencies

26. Establishing a Culture of Competency Assessment

27. Evaluation PTSD

28. STFM Residency Competency Assessment Toolkit

29. Residency Competency Assessment Toolkit Headings

31. How do I get started? From tool kit

32. What is your culture of assessment

33. STFM Residency Competency Assessment Toolkit  http://www.stfm.org/rctoolkit http://www.stfm.org/rctoolkit

34. Culture Shift

35. Culture Shift - Example  P4 – Preparing the Personal Physician for Practice  Innovation – Implementing competency based curriculum

36.  Transparency  I know who is assessing me and how  Direct Observation  Observable behaviors  Direct Communication and Feedback  “Assessor to resident”  Resident involved in the assessment conversations about them as individuals

37. Pieces to the Puzzle  Faculty  Become “resident-centered educators”  Develop and practice skills in giving formative feedback  Openness to receiving feedback  Residents  Overcome evaluation PTSD  Reduce reluctance to reveal gaps  Give faculty feedback

38. Strategies for Creating a Culture of Competency Assessment

39. Miller’s Assessment Pyramid Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 39

40. Competency Based Education Impact onTraining Adapted from Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med65(9 Suppl): S63-7. 40

42. Break

43. Design Implementing competency assessment

44. Eyes on the Prize….  What business are we in?

45. Eyes on the Prize….  So how do we get there?

46. Programmatic Competency Pathway created by Judy Pauwels, MD and Larry Mauksch, M.Ed 46

47. Step 1: Define overall competency goals: the Milestones

48. Step 2: Write goals & objectives for individual curricular elements

49. Step 3: Identify formative tools to be used during each educational element

50. Step 4: Develop linked evaluations for individual curricular elements

51. Step 5: Link individual evaluations, formative assessments back to the milestones

52. Formative Comments and the Milestones - Example

53. Formative Comments and the Milestones

54. Step 6: Articulate specific measurable objectives for each milestone

55. Step 7: Develop IEPs for each resident

56. Steps in Competency Assessment Design Dijkstra, Van DerVleuten, Schuwirth A new framework for designing programs of assessment, Adv in Health Sci Ed 2010 15, 379-393

57. EXERCISE

58. Formative Assessment: Design, Context, and skills

59. Miller’s Assessment Pyramid Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 59

60. Formative VS Summative Roles FormativeSummative Coach(preceptor)Evaluator Uses prior data to influence teaching Using prior data to determine promotion Regularly engages the learner in goal setting Measures the learner’s performance against external goals Integrates learner preferences into educational design Prescribes experiences to the learner

61. Formative Assessment: Design

62. Formative Assessment Skills

63. Notes on Feedback

64. Helpful Feedback Is…

65. Video Design

66. Common and Better Video: Faculty development primer

67. Video Creation

68. Lunch

69. Video creation and viewing

70. Barriers to Competency Assessment What are your barriers?

72. 72 Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85: 787-790

74. Barriers to Competency Assessment What are your barriers?

75. It’s time to start the Family Feud

76. 100 STFM members surveyed, top 6 answers are on the board. In your residency program what are the top six barriers to implementing competency based assessment? Survey says… Lack of TimeData Overwhelming Lack of LeadershipLack of Language Lack of TrainingThis is not needed 1 2 3 4 5 6

77. Barriers to Training & Implementation 4 Lacks and a Feeling

79. Lack of Training

80. Lack of Time

81. Lack of Language

82. Lack of Leadership

86. Direct Observation METHODSMETHODS TIME TIME Pros and Cons

87. Faculty Time Demand Educational Pros Educational Cons Direct observation in the room High Loss of income or other activity Clear view Teach on the fly Trainee initially self conscious Risk upstaging relationship Video review Moderate to High Loss of income or other activity Trainee self observes Strong educational options Delayed practice Requires: Technical expertise Expense Peer Very Low ++ Observations ++ Reflection ++ Practice Less depth Less versatility Reliability ? You (faculty) are observed Low Role modeling Observer self Faculty growth Passive trainee role OSCE & Standardized Patients High Some loss of income or other activity Control Specific Obs Miller’s Pyramid “Fake” Expensive Learning Curve

88. Don’t let the perfect become the enemy of the good

90. Tools Implementing competency assessment

91. Data Collection

92. Select tools that:

93. 93

94. Programmatic Competency Pathway created by Judy Pauwels, MD and Larry Mauksch, M.Ed 94

95. Data Collection – Sample tools

97. Second sample video for rating

98. Data Consolidation

99. Break

100. Faculty development

101. Strategies for Creating a Culture of Competency Assessment

102. Competency Based Education Impact on Training Adapted from Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65 (9 Suppl): S63-7. 102

103. The Eight Step Process of Successful Change Kotter J and Rathgeber H. Our Iceberg is Melting 2005 St. Martins Press, New York, NY.

107. 107

108. Faculty Development

109. Residency Competency Assessment Toolkit Headings

110. STFM Residency Competency Assessment Toolkit http://www.stfm.org/rctoolkit/  Resources  FAQs

111. Planning your next steps

112. Planning Your Next Step Larry Mauksch, M.Ed University of Washington Department of Family Medicine

113. Implemented Competencies?

114. Wrap-up and Evaluations

115. Link to Two Videos The videos of the physician who does and does not demonstrate relationship development, agenda setting and appropriate EHR use can be found at the link below on the “Improving Communication Assessment Project”. The full videos last about 10 minutes and 30 seconds and are for training to use the Patient Centered Observation Form (PCOF). Total training takes about 40 minutes. To show these videos to a group, one must fill in the online data before advancing to the next video. Please do not press the final submit button on the last page unless you enter accurate ratings because this will ruin our data. It is fine to view the videos and enter arbitrary PCOF ratings to advance to the next video and leave the site after viewing the second video. http://uwfamilymedicine.org/pcof

116. References Davidoff F. Music lessons: what musicians can teach doctors (and other health professionals). Ann Intern Med. Mar 15 2011;154(6):426-429. Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. Aug 2009;15(3):379-393. Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009;55(12):1260-1262, e1100-1262. Gwande A. Personal Best: Top Athletes and singers have coaches. Should you? The New Yorker. 2011. http://www.newyorker.com/reporting/2011/10/03/111003fa_fact_gawande. Accessed February 29, 2012. Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. Apr 2011;86(4):460-467. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. Mar 15 2012;366(11):1051-1056. Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician. Sep 2011;57(9):e323-330. ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32(8):669-675. Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med. Jul 2011;86(7):846-852.

117. References http://blogs.gnome.org/bolsh/2009/07/22/barriers-to-community-growth/ http://thefuturebuzz.com/2011/09/26/seo-and-social-media-myths/ http://www.classroomjr.com/greek-myths-for-kids/ http://speakingofresearch.com/tag/myths/ http://www.thebody.com/content/63841/10-harmful-myths-about-hivaids.html http://www.asia.ru/en/ProductInfo/423282.html http://en.wikipedia.org/wiki/Data_(Star_Trek) http://en.wikipedia.org/wiki/Rainbow http://www.stfm.org/rctoolkit/ http://oilersaddict.com/wp/2011/11/05/fast-out-of-the-blocks-can-it-be-maintained/ http://www.sport360.com/article/no-stopping-them-kenyan-women-dominate-daegu