Determining and Tracking Resident Procedural Competency in the New Accreditation System Michael D. Geurin, MD, FAAFP Emily J. Colson, MD Tanya Lila Hamilton,

Slides:



Advertisements
Similar presentations
Nursing Diagnosis: Definition
Advertisements

Feedback in Clinical Skills Session in Pre-clinical Years Dr. Steve Martin Island Medical Program.
Introduction to Competency-Based Residency Education
Clinical Competency Committees (CCC): 3 different perspectives Sharon Dabrow: Pediatrics PD Cuc Mai: Internal Medicine PD Todd Kumm: Radiology PD.
Assigning Milestone Evaluations in Internal Medicine
Clinical Evaluation Tools Copyright 2008 by The Health Alliance of MidAmerica LLC.
PRESENTED BY: Michael T. Flannery, M.D., F.A.C.P. Professor of Medicine GME Internal Review Director.
Learning Objectives, Performance Tasks and Rubrics: Demonstrating Understanding and Defining What Good Is Brenda Lyseng Minnesota State Colleges.
Islamic University of Gaza Faculty of Nursing
Chapter 10 Establishing the Performance Management System
Promoting Excellence in Family Medicine nMRCGP Workplace-based Assessment March 2007.
Laboratory Personnel Dr/Ehsan Moahmen Rizk.
CBES Essentials for Residents, Fellows, and Faculty A 10-minute primer on student performance assessment in required clerkships Stanford School of Medicine.
Comparison: Traditional vs. Outcome Project Evaluative Processes Craig McClure, MD Educational Outcomes Service Group University of Arizona December 2004.
Classroom Assessment A Practical Guide for Educators by Craig A. Mertler Chapter 5 Informal Assessments.
Family Medicine Program By the end of this session, faculty will 1.Understand what is meant by competence and the competence trajectory expected during.
Triple C Competency-based Curriculum: Implications for Family Medicine Residency Programs.
End of Shift Milestone Evaluation Forms
GUIDELINES FOR CURRICULUM PLANNING Jose Y. Cueto Jr., MD, MHPEd Member Board of Medicine.
U N I V E R S I T Ä T S M E D I Z I N B E R L I N Entrustable professional activities for learning in competency-based undergraduate medical education.
Clinical Observer Training Session 1 Course Overview Courtesy: HIP.
Classroom Assessments Checklists, Rating Scales, and Rubrics
Assessment tool OSCE AH Mehrparvar,MD Occupational Medicine department Yazd University of Medical Sciences.
EVALUATION: Making it Work Borrowed from the “Mahec” preceptor program. e-Learning_Tools.asp M odified for the UBC, Family Practice,
McGaw’s Overview of the Next Accreditation System (NAS)
Psychology Workforce Development for Primary Care Cynthia D. Belar, PhD, ABPP Executive Director, APA Education Directorate Collaborative.
Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,
Inter-Rater Reliability Respiratory Ivy Tech Community College-Indianapolis.
Developing an Assessment System B. Joyce, PhD 2006.
Student assessment Assessment tools AH Mehrparvar,MD Occupational Medicine department Yazd University of Medical Sciences.
TUSK Competency Framework Project November 20, 2008.
What is a Planned Curriculum?
(c) 2007 McGraw-Hill Higher Education. All rights reserved. Accountability and Teacher Evaluation Chapter 14.
Section 1. Introduction Orientation to Virginia’s QRIS.
Performance Appraisal Basics MANA 4328 Dr. Jeanne Michalski
Based on the performance appraisal system, the nursing home reported an improvement in the reduction of medication errors. However, adverse clinical.
Workplace based assessment for the nMRCGP. nMRCGP Integrated assessment package comprising:  Applied knowledge test (AKT)  Clinical skills assessment.
Not Just “MK-1” How learning the skills of EBM relates to the pediatric milestones Martha S Wright, MD, MEd Rainbow Babies and Children’s Hospital.
Accreditation Council for Graduate Medical Education Milestones are Coming: A Conversation with the Family Medicine Milestones Committee May 2013.
Chapter 8 Performance Management and Appraisal
A Procedural Competency Evaluation Process: Systematic and Consistent Lance Fuchs, MD FAAFP Vidush Athyal, MD FAAFP Dennis Andrade, MD Kaiser Permanente.
Patricia Kokotailo, Sarah Pitts, Sheryl Ryan, Karen Soren, Maria Trent
Daniel J. Schumacher, MD, MEd Mary Pat Frintner, MSPH Presented at: Association of Pediatric Program Directors Spring Meeting April 1, 2016 New Orleans,
Jhpiego’s Global Presence 1. ENSURING QUALITY OF TRAINING Waqar Saleem 1 st December, 2010 Geneva.
Rural Family Medicine Residency Training and Developing a Rural Area-of-Concentration Carl Rasmussen, MD OHSU Family Medicine Residency Portland, OR
Developing Global Family Medicine Faculty “de Novo” John G Halvorsen, MD, MS Professor Emeritus of Family and Community Medicine University of Illinois.
Preparing for the ACGME's Next Accreditation System (NAS) A Prospective View for Family Medicine Residencies Joseph J. Brocato, PhD, University of Minnesota.
Compass Learning Management System An Online Environment that Facilitates Learning and Assessment in Residency Education Molly Cohen-Osher, MD, Greg Sawin,
Practice CSE-3: A Formative, Low Stakes Experience for Third Year Family Medicine Clerkship Students Jeri R. Reid, MD Donna M. Roberts, MD Mary B. Carter,
Approaching Milestones Documentation: Tricks, Tips, and Examples Describing What We Want in a Family Physician: From Competencies to Milestones Allen F.
Best Practices for Procedure Documentation and Tracking Patricia Bouknight, MD Rebecca Beagle, RN, BSN, CHDA, CPHQ Spartanburg Family Medicine Residency.
Boston University Global Health Collaborative
Implementation of a 3-year Competency Based Procedures Curriculum Dale A. Patterson, MD Associate Director The Toledo Hospital Family Practice Residency.
ACGME CLER Visit USF and TGH September 22-24, 2014 Results.
Next Accreditation System (NAS) Primer Cuc Mai IM Residency Program Director Annual PD Workshop 2015.
Entrustable Professional Activities (EPAs) for the Assessment of Early Medical Students H. Carrie Chen MD MSEd1, Margaret McNamara MD1, Arianne Teherani.
Classroom Assessments Checklists, Rating Scales, and Rubrics
Introduction to Evaluation
A Foundation for Procedure Acquisition and Competence using On-Line Resources, Individualized Education, and Simulation Beth Anne Fox, MD, MPH Jason Moore,
Communicating Milestones Evaluation Data and Comments from the CCC to Residents Using an Innovative “Grade Card” Steven McDonald, MD – Program Director.
Classroom Assessments Checklists, Rating Scales, and Rubrics
This presentation includes the audio recording from the “Review of the Internal Medicine Subspecialty Reporting Milestones” webinar held on September 11,
This presentation includes the audio recording from the “Review of the Internal Medicine Subspecialty Reporting Milestones” webinar held on September 9,
CBEI Essentials for Residents, Fellows, Advanced Practice Providers, and Faculty A 10-minute primer on student performance assessment in required clerkships.
To Entrust or Not to Entrust
Job Analysis CHAPTER FOUR Screen graphics created by:
Evaluation Rating Forms
Clinical Educator and Teacher Candidate Training Chapter for the Candidate Preservice Assessment for Student Teachers (CPAST) Form Developed by the VARI-EPP*
Involving Families Early Childhood Outcomes Center.
Presentation transcript:

Determining and Tracking Resident Procedural Competency in the New Accreditation System Michael D. Geurin, MD, FAAFP Emily J. Colson, MD Tanya Lila Hamilton, MD

Disclosures Dr. Geurin and Dr. Colson have nothing to disclose. Dr. Hamilton is an unpaid residency trainer for Nexplanon. We will pass out a handout with all of the content (and then some) at the end of the seminar—for now, feel free to fully engage with the experience of the presentation Handout and sample documents are being uploaded to STFM’s Resource Library: FMDRL.org

Objectives On completion of this seminar, participants should be able to... Describe and utilize objectively descriptive evaluation scales Construct a competency-based, objectively descriptive evaluation tool of a procedure Use these skills to develop a customized procedural competency system at their residency program

Procedures-Related Requirements Residents must be able to competently perform all medical, diagnostic, and surgical procedures considered essential for the area of practice. (Outcome) Residents must receive training to perform clinical procedures required for their future practices in ambulatory and hospital environments. (Core)

Procedures-Related Requirements The program director and family medicine faculty should develop a list of procedural competencies required for completion by all residents in the program prior to graduation. (Core) –This list must be based on the anticipated practice needs of all family medicine residents. (Core) –In creating this list, the faculty should consider the current practices of program graduates, national data regarding procedural care in family medicine, and the needs of the community to be served. (Core)

Procedures-Related Requirements The program must provide objective assessments of competence in patient care and procedural skills... based on the specialty-specific Milestones; (Core) –This assessment must involve direct observation of resident-patient encounters. (Detail) –Each resident must be assessed in data gathering, clinical reasoning, patient management, and procedures in both the inpatient and outpatient settings. (Detail)

Procedures-Related Requirements The family medicine program faculty shall, as a group, be qualified to teach all required procedures as listed in this document. During the training program, the resident must: –Keep a log of each procedure performed.

Procedures-Related Requirements Procedural Medicine: The program must have defined mechanisms to train residents to competency in the following procedures: –Joint injections. –Biopsy of dermal lesions. –Excision of subcutaneous lesions. –Incision and drainage of abscess. –Cryosurgery of skin. –Curettage of skin lesion. –Laceration repair. –Endometrial biopsy. –Office microscopy. –Splinting. –EKG interpretation. –Office spirometry. –Toenail removal. –Defibrillation. –Removal of cerumen from ear canal. –Endotracheal intubation.

Procedures-Related Requirements Optional Procedures: –Vasectomy –Central line placement –Vaginal delivery –Episiotomy repair –Flexible sigmoidoscopy –Colonoscopy –Lumbar puncture –IUD insertion –Breast cyst aspiration –Epistaxis management (nasal packing/anterior cautery) –Trigger point injections –Allergy testing –Neonatal circumcision

How do medical educators know when a resident is “ready” to perform a procedure independently? Several techniques have been utilized, with inherent drawbacks...

Determining Competency Knowledge-based tests –Verbal (“walk me through the steps of the procedure” ≈ pimping) –Written (e.g., multiple-choice test) Completion of set number of procedures Supervisor observation of technical section of the procedure

Rating Scales Developed in the early 1900s specifically to try to extend measurement beyond that of knowledge to that of skills and attitudes 4 –Thurstone (1920s) advocated for scales to use “equal-appearing intervals” –Likert (1932) added descriptors at each point along the scale (e.g., strongly agree, agree, undecided...) Source: Holmboe ES, Hawkins RE. Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA: Mosby Elsevier; 2008.

Bias and Error Many traditional rating scales are subjective (“pass” is in the eye of the beholder) This increases the likelihood of different forms of bias and error affecting performance evaluations –One-on-one vs. group evaluations

A type of confirmation bias in which a positive feeling in one area leads to neutral or ambiguous performance areas to be viewed positively Halo Effect

The converse of the halo effect, in which dislike in one area creates a predisposition to view other performance areas negatively Horns Effect

When evaluation occurs by groups, tendency of the group (“the sunflowers”) to align with the views of the most powerful or charismatic member of the discussion (“the sun”) Sunflower Management

Being too lenient or too harsh for reasons that have nothing to do with the facts of performance. This is common when evaluators lack sufficient information or the time to prepare adequately. Leniency/Severity Error

The tendency to rate individuals around the midpoint of a rating scale, avoiding the extremes of the scale, despite variations in actual performance. Central Tendency Bias A corollary with the milestones would be to tend to rate all interns at Level 2, and all graduating residents at Level 4

Rating based on recent performance rather than properly weighting performance over whole time period Proximity Bias

The converse of proximity bias: basing the current evaluation on past evaluation(s), rather than fairly evaluating current performance Longevity Bias

The tendency to place an undue emphasis on a resident’s internal characteristics to explain poor performance in a given situation, rather than considering external factors, such as problems with the educational program Fundamental Attribution Error

The converse of fundamental attribution error: overemphasing situational factors when addressing poor performance Actor-Observer Bias

Tweaking: matching the assessment to a predetermined outcomes that make the program/organization look good (for example, when reporting sub-competency performance to the Accreditation Data System)

Behavioral Anchors More recently, behaviorally anchored rating scales (BARS) have been developed, providing more objective descriptors of performance along the scale –Goal: limit bias and improve the psychometrics of the performance evaluations. –While the ACGME milestones are not intended to be used as global rating scales, the progression of behaviorally anchored descriptors is similar to that seen in a BARS Source: Holmboe ES, Hawkins RE. Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA: Mosby Elsevier; 2008.

Sample BARS Source: Holmboe ES, Hawkins RE. Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA: Mosby Elsevier; 2008.

Sample BARS Source: University of Washington Department of Family Medicine

Why Use Behavioral Anchors? May improve inter-rater reliability May support shared frames of reference in the faculty May increase the validity of the evaluation if anchors derived from “gold standard”

Checklists The use of checklists to document whether specific behaviors did or did not occur can help to improve accuracy of the evaluation Not sufficient alone for global or longitudinal evaluations

MFMR Procedures Competency- Based Evaluation System (CBES) Faculty developed list of required and recommended procedures Faculty give competency ratings at each observed procedure (BARS).

Procedure Evaluation

MFMR Procedures CBES The textbook Procedures for Primary Care and the digital resource Procedures Consult are our “gold standard” for procedures The tally of the procedures competently performed unassisted is collected in a system called the Procedures Passport.

Procedures Passport

MFMR Procedures CBES During a focused procedure clinic the supervising faculty completes a three-part form –Tally of procedures –Milestone-based evaluation (PC5) –Collaborates with the resident to develop goals for the next procedure clinic.

MFMR Procedures CBES Basic Skills Qualification (BSQ) as final “test” for each procedure –Must first complete minimum number at “competently performed unassisted” level –Compares performance to standardized technical and cognitive expectations

Procedures Passport

MFMR Procedures CBES The MFMR BSQs were adapted from the Providence St. Peter Family Medicine Residency Program (with them giving credit to the Tufts University FMR at Cambridge Health Alliance)

MFMR Procedures CBES When a resident has passed the BSQ for a procedure, the resident is considered competent for independent performance. The Procedures Passport tracks this throughout the resident’s time in the program.

Thank you! Any Questions or Comments? Handout and sample documents are being uploaded to STFM’s Resource Library: FMDRL.org Please evaluate this session! stfm.org/sessionevaluation