2. Objectives: Participants will be able to:

3. 3 Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85: 787-790.

4. 4

5. Task Force Membership Resident Competency Measurement Task Force Members Chair Sam Cullison, MD, Task Force Chair - Swedish FMR Seattle Wendy Biggs, MD, AAFP Colleen Conry, MD, University of Colorado Alan David, MD, Medical College of Wisconsin Mike Donoff, MD, University of Alberta Julie Dostal, MD, Competency Assessment Work Group Lead, Lehigh Valley FMR Larry Mauksch, MEd, Faculty Development and Dissemination Work Group Lead, University of Washington FMR Tom O'Neill, PhD, ABFM Allen Shaughnessy, PharmD, Tufts University FMR Stephen Wilson, MD, UPMC St Margaret FMR Stacy Brungardt, CAE, Society of Teachers of Family Medicine 5

6. STFM Residency Competency Assessment Toolkit http://www.stfm.org/rctoolkit/ Presentations

7. Outline

8. Introduction

9. Competency Assessment

10. Why is this important?

11. Current RC Requirements (p 33): 11

12. 12

13. Proposed New RC Requirements – (on hold) 13

14. Proposed New RC Requirements – (on hold) 14

15. Formative Assessment

16. Formative Competency Assessment

17. Examples of Formative Assessment

18. Formative Assessment (coaching) Examples

19. Formative Feedback examples cont’d

20. ACGME Competencies

21. Exercise 2  Second video

22. Establishing a Culture of Competency Assessment

23. STFM Residency Competency Assessment Toolkit

24. Residency Competency Assessment Toolkit Headings

26. How do I get started? From tool kit

27. What is your culture of assessment

28. STFM Residency Competency Assessment Toolkit  http://www.stfm.org/rctoolkit http://www.stfm.org/rctoolkit

29. Culture Shift

30. Strategies for Creating a Culture of Competency Assessment

31. Miller’s Assessment Pyramid Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 31

32. Competency Based Education Impact on Training Adapted from Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S63-7. 32

34. Break

35. Design Implementing competency assessment

36. Programmatic Competency Pathway created by Judy Pauwels, MD and Larry Mauksch, M.Ed 36

37. Steps in Competency Assessment Design Dijkstra, Van Der Vleuten, Schuwirth A new framework for designing programs of assessment, Adv in Health Sci Ed 2010 15, 379-393

38. Sample Goals and Milestones

39. Formative Comments and the Milestones - Example

40. Formative Comments and the Milestones

41. Design

42. Skills and Training Implementing competency assessment

43. Miller’s Assessment Pyramid Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 43

44. Formative VS Summative Roles FormativeSummative Coach (preceptor)Evaluator Uses prior data to influence teaching Using prior data to determine promotion Regularly engages the learner in goal setting Measures the learner’s performance against external goals Integrates learner preferences into educational design Prescribes experiences to the learner

45. Formative Assessment Skills

46. Notes on Feedback

47. Helpful Feedback Is…

48. Barriers to Competency Assessment What are your barriers?

49. Barriers to Training and Implementation

50. Lack of training

51. Lack of time

52. Direct Observation: Methods, Time Demand, Pros and Cons Faculty Time Demand Educational Pros Educational Cons Direct observation in the room High Loss of income or other activity Clear view Can teach on the fly Trainee initially self conscious Risk of upstaging relationship Video reviewHigh Loss of income or other activity Trainee self observes, strong educational options Delayed practice Requires technical expertise and expense Closed circuitModerate Some income loss or other activity Fast practice Faculty development Distraction, time limitation, Reliability? PeerVery Low++Observations ++Reflection ++Practice Less depth, versatility, Reliability? You (faculty) are observed LowRole modeling Observer self Faculty growth Passive trainee role

53. Lack of language

54. Lack of leadership

55. Tools Implementing competency assessment

56. Data Collection

57. Select tools that:

58. 58

59. Programmatic Competency Pathway adapted from a model created by Judy Pauwels, MD

60. Data Collection – Sample tools

62. Data Consolidation

63. Faculty development

64. 64

65. Faculty Development

66. Residency Competency Assessment Toolkit Headings

67. STFM Residency Competency Assessment Toolkit http://www.stfm.org/rctoolkit/  Resources  FAQs

68. Planning Your Next Step Larry Mauksch, M.Ed University of Washington Department of Family Medicine

69. Sharing and Wrap-up

70. Link to Two Videos The videos of the physician who does and does not demonstrate relationship development, agenda setting and appropriate EHR use can be found at the link below on the “Improving Communication Assessment Project”. The full videos last about 10 minutes and 30 seconds and are for training to use the Patient Centered Observation Form (PCOF). Total training takes about 40 minutes. To show these videos to a group, one must fill in the online data before advancing to the next video. Please do not press the final submit button on the last page unless you enter accurate ratings because this will ruin our data. It is fine to view the videos and enter arbitrary PCOF ratings to advance to the next video and leave the site after viewing the second video. http://uwfamilymedicine.org/pcof

71. References Davidoff F. Music lessons: what musicians can teach doctors (and other health professionals). Ann Intern Med. Mar 15 2011;154(6):426-429. Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. Aug 2009;15(3):379-393. Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009;55(12):1260-1262, e1100-1262. Gwande A. Personal Best: Top Athletes and singers have coaches. Should you? The New Yorker. 2011. http://www.newyorker.com/reporting/2011/10/03/111003fa_fact_gawande. Accessed February 29, 2012. Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. Apr 2011;86(4):460-467. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. Mar 15 2012;366(11):1051-1056. Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician. Sep 2011;57(9):e323-330. ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32(8):669-675. Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med. Jul 2011;86(7):846-852.