Outline

Introduction

 WHAT DID YOU SAY IN OUR SURVEY?

Training Level for Faculty 4.1% Fully Embraced 20.4% Partial Implement 55.1% Accept & started 6.1% No Implement 14.3% Unfamiliar ACGME Competencies

Implemented Competencies?

How Impacted?

What is Level of Resistance to Change?

Objectives: Participants will be able to:

10 Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85:

11

Resident Competency Measurement Task Force Membership Chair Sam Cullison, MD, Task Force Chair - Swedish FMR Seattle Wendy Biggs, MD, AAFP Colleen Conry, MD, University of Colorado Alan David, MD, Medical College of Wisconsin Mike Donoff, MD, University of Alberta Julie Dostal, MD, Competency Assessment Work Group Lead, Lehigh Valley FMR Larry Mauksch, MEd, Faculty Development and Dissemination Work Group Lead, University of Washington FMR Tom O'Neill, PhD, ABFM Allen Shaughnessy, PharmD, Tufts University FMR Stephen Wilson, MD, UPMC St Margaret FMR Stacy Brungardt, CAE, Society of Teachers of Family Medicine 12

STFM Residency Competency Assessment Toolkit Presentations

RC change and the Milestones

Competency Assessment

Current RC Requirements (p 33): 16

17

Proposed New RC Requirements – (on hold) 18

Proposed New RC Requirements – (on hold) 19

Formative Assessment

Formative Competency Assessment

Examples of Formative Assessment

Formative Assessment (coaching) Examples

Formative Feedback examples cont’d

ACGME Competencies

Establishing a Culture of Competency Assessment

Evaluation PTSD

STFM Residency Competency Assessment Toolkit

Residency Competency Assessment Toolkit Headings

How do I get started? From tool kit

What is your culture of assessment

STFM Residency Competency Assessment Toolkit 

Culture Shift

Culture Shift - Example  P4 – Preparing the Personal Physician for Practice  Innovation – Implementing competency based curriculum

 Transparency  I know who is assessing me and how  Direct Observation  Observable behaviors  Direct Communication and Feedback  “Assessor to resident”  Resident involved in the assessment conversations about them as individuals

Pieces to the Puzzle  Faculty  Become “resident-centered educators”  Develop and practice skills in giving formative feedback  Openness to receiving feedback  Residents  Overcome evaluation PTSD  Reduce reluctance to reveal gaps  Give faculty feedback

Strategies for Creating a Culture of Competency Assessment

Miller’s Assessment Pyramid Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance Acad Med65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby

Competency Based Education Impact onTraining Adapted from Miller, G. E. The assessment of clinical skills/competence/performance Acad Med65(9 Suppl): S

Break

Design Implementing competency assessment

Eyes on the Prize….  What business are we in?

Eyes on the Prize….  So how do we get there?

Programmatic Competency Pathway created by Judy Pauwels, MD and Larry Mauksch, M.Ed 46

Step 1: Define overall competency goals: the Milestones

Step 2: Write goals & objectives for individual curricular elements

Step 3: Identify formative tools to be used during each educational element

Step 4: Develop linked evaluations for individual curricular elements

Step 5: Link individual evaluations, formative assessments back to the milestones

Formative Comments and the Milestones - Example

Formative Comments and the Milestones

Step 6: Articulate specific measurable objectives for each milestone

Step 7: Develop IEPs for each resident

Steps in Competency Assessment Design Dijkstra, Van DerVleuten, Schuwirth A new framework for designing programs of assessment, Adv in Health Sci Ed ,

EXERCISE

Formative Assessment: Design, Context, and skills

Miller’s Assessment Pyramid Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance Acad Med65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby

Formative VS Summative Roles FormativeSummative Coach(preceptor)Evaluator Uses prior data to influence teaching Using prior data to determine promotion Regularly engages the learner in goal setting Measures the learner’s performance against external goals Integrates learner preferences into educational design Prescribes experiences to the learner

Formative Assessment: Design

Formative Assessment Skills

Notes on Feedback

Helpful Feedback Is…

Video Design

Common and Better Video: Faculty development primer

Video Creation

Lunch

Video creation and viewing

Barriers to Competency Assessment What are your barriers?

72 Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85:

Barriers to Competency Assessment What are your barriers?

It’s time to start the Family Feud

100 STFM members surveyed, top 6 answers are on the board. In your residency program what are the top six barriers to implementing competency based assessment? Survey says… Lack of TimeData Overwhelming Lack of LeadershipLack of Language Lack of TrainingThis is not needed

Barriers to Training & Implementation 4 Lacks and a Feeling

Lack of Training

Lack of Time

Lack of Language

Lack of Leadership

Direct Observation METHODSMETHODS TIME TIME Pros and Cons

Faculty Time Demand Educational Pros Educational Cons Direct observation in the room High Loss of income or other activity Clear view Teach on the fly Trainee initially self conscious Risk upstaging relationship Video review Moderate to High Loss of income or other activity Trainee self observes Strong educational options Delayed practice Requires: Technical expertise Expense Peer Very Low ++ Observations ++ Reflection ++ Practice Less depth Less versatility Reliability ? You (faculty) are observed Low Role modeling Observer self Faculty growth Passive trainee role OSCE & Standardized Patients High Some loss of income or other activity Control Specific Obs Miller’s Pyramid “Fake” Expensive Learning Curve

Don’t let the perfect become the enemy of the good

Tools Implementing competency assessment

Data Collection

Select tools that:

93

Programmatic Competency Pathway created by Judy Pauwels, MD and Larry Mauksch, M.Ed 94

Data Collection – Sample tools

Second sample video for rating

Data Consolidation

Break

Faculty development

Strategies for Creating a Culture of Competency Assessment

Competency Based Education Impact on Training Adapted from Miller, G. E. The assessment of clinical skills/competence/performance Acad Med 65 (9 Suppl): S

The Eight Step Process of Successful Change Kotter J and Rathgeber H. Our Iceberg is Melting 2005 St. Martins Press, New York, NY.

107

Faculty Development

Residency Competency Assessment Toolkit Headings

STFM Residency Competency Assessment Toolkit  Resources  FAQs

Planning your next steps

Planning Your Next Step Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Implemented Competencies?

Wrap-up and Evaluations

Link to Two Videos The videos of the physician who does and does not demonstrate relationship development, agenda setting and appropriate EHR use can be found at the link below on the “Improving Communication Assessment Project”. The full videos last about 10 minutes and 30 seconds and are for training to use the Patient Centered Observation Form (PCOF). Total training takes about 40 minutes. To show these videos to a group, one must fill in the online data before advancing to the next video. Please do not press the final submit button on the last page unless you enter accurate ratings because this will ruin our data. It is fine to view the videos and enter arbitrary PCOF ratings to advance to the next video and leave the site after viewing the second video.

References Davidoff F. Music lessons: what musicians can teach doctors (and other health professionals). Ann Intern Med. Mar ;154(6): Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. Aug 2009;15(3): Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009;55(12): , e Gwande A. Personal Best: Top Athletes and singers have coaches. Should you? The New Yorker Accessed February 29, Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. Apr 2011;86(4): Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. Mar ;366(11): Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician. Sep 2011;57(9):e ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32(8): Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med. Jul 2011;86(7):

References