Objectives: Participants will be able to:

3 Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85:

4

Task Force Membership Resident Competency Measurement Task Force Members Chair Sam Cullison, MD, Task Force Chair - Swedish FMR Seattle Wendy Biggs, MD, AAFP Colleen Conry, MD, University of Colorado Alan David, MD, Medical College of Wisconsin Mike Donoff, MD, University of Alberta Julie Dostal, MD, Competency Assessment Work Group Lead, Lehigh Valley FMR Larry Mauksch, MEd, Faculty Development and Dissemination Work Group Lead, University of Washington FMR Tom O'Neill, PhD, ABFM Allen Shaughnessy, PharmD, Tufts University FMR Stephen Wilson, MD, UPMC St Margaret FMR Stacy Brungardt, CAE, Society of Teachers of Family Medicine 5

STFM Residency Competency Assessment Toolkit Presentations

Outline

Introduction

Competency Assessment

Why is this important?

Current RC Requirements (p 33): 11

12

Proposed New RC Requirements – (on hold) 13

Proposed New RC Requirements – (on hold) 14

Formative Assessment

Formative Competency Assessment

Examples of Formative Assessment

Formative Assessment (coaching) Examples

Formative Feedback examples cont’d

ACGME Competencies

Exercise 2  Second video

Establishing a Culture of Competency Assessment

STFM Residency Competency Assessment Toolkit

Residency Competency Assessment Toolkit Headings

How do I get started? From tool kit

What is your culture of assessment

STFM Residency Competency Assessment Toolkit 

Culture Shift

Strategies for Creating a Culture of Competency Assessment

Miller’s Assessment Pyramid Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance Acad Med 65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby

Competency Based Education Impact on Training Adapted from Miller, G. E. The assessment of clinical skills/competence/performance Acad Med 65(9 Suppl): S

Break

Design Implementing competency assessment

Programmatic Competency Pathway created by Judy Pauwels, MD and Larry Mauksch, M.Ed 36

Steps in Competency Assessment Design Dijkstra, Van Der Vleuten, Schuwirth A new framework for designing programs of assessment, Adv in Health Sci Ed ,

Sample Goals and Milestones

Formative Comments and the Milestones - Example

Formative Comments and the Milestones

Design

Skills and Training Implementing competency assessment

Miller’s Assessment Pyramid Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance Acad Med 65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby

Formative VS Summative Roles FormativeSummative Coach (preceptor)Evaluator Uses prior data to influence teaching Using prior data to determine promotion Regularly engages the learner in goal setting Measures the learner’s performance against external goals Integrates learner preferences into educational design Prescribes experiences to the learner

Formative Assessment Skills

Notes on Feedback

Helpful Feedback Is…

Barriers to Competency Assessment What are your barriers?

Barriers to Training and Implementation

Lack of training

Lack of time

Direct Observation: Methods, Time Demand, Pros and Cons Faculty Time Demand Educational Pros Educational Cons Direct observation in the room High Loss of income or other activity Clear view Can teach on the fly Trainee initially self conscious Risk of upstaging relationship Video reviewHigh Loss of income or other activity Trainee self observes, strong educational options Delayed practice Requires technical expertise and expense Closed circuitModerate Some income loss or other activity Fast practice Faculty development Distraction, time limitation, Reliability? PeerVery Low++Observations ++Reflection ++Practice Less depth, versatility, Reliability? You (faculty) are observed LowRole modeling Observer self Faculty growth Passive trainee role

Lack of language

Lack of leadership

Tools Implementing competency assessment

Data Collection

Select tools that:

58

Programmatic Competency Pathway adapted from a model created by Judy Pauwels, MD

Data Collection – Sample tools

Data Consolidation

Faculty development

64

Faculty Development

Residency Competency Assessment Toolkit Headings

STFM Residency Competency Assessment Toolkit  Resources  FAQs

Planning Your Next Step Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Sharing and Wrap-up

Link to Two Videos The videos of the physician who does and does not demonstrate relationship development, agenda setting and appropriate EHR use can be found at the link below on the “Improving Communication Assessment Project”. The full videos last about 10 minutes and 30 seconds and are for training to use the Patient Centered Observation Form (PCOF). Total training takes about 40 minutes. To show these videos to a group, one must fill in the online data before advancing to the next video. Please do not press the final submit button on the last page unless you enter accurate ratings because this will ruin our data. It is fine to view the videos and enter arbitrary PCOF ratings to advance to the next video and leave the site after viewing the second video.

References Davidoff F. Music lessons: what musicians can teach doctors (and other health professionals). Ann Intern Med. Mar ;154(6): Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. Aug 2009;15(3): Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009;55(12): , e Gwande A. Personal Best: Top Athletes and singers have coaches. Should you? The New Yorker Accessed February 29, Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. Apr 2011;86(4): Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. Mar ;366(11): Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician. Sep 2011;57(9):e ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32(8): Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med. Jul 2011;86(7):