How Behavioral Science Faculty Can Help Competency Assessment Glow and Flow Larry Mauksch, M.Ed Senior lecturer, Department of Family Medicine University.

Slides:



Advertisements
Similar presentations
Evaluation Overview - Basics. Purpose of Testing Diagnostic Formative Summative.
Advertisements

Performance Assessment
Objectives Explain the purpose of the RIME feedback method.
Competencies, Milestones & EPAs: What Does It All Mean?
Workplace assessment Dr. Kieran Walsh, Editor, BMJ Learning. 2.
MILESTONES, EPAS, NAS…AND OTHER ACGME JARGON Committee on Graduate Medical Education September 24, 2012 Sara LP Ross, MD.
The Milestone Project For Neurosurgical Resident Coordinators.
Assigning Milestone Evaluations in Internal Medicine
دکتر فرشید عابدی. Competence competence in medicine : “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning,
Workplace-based Assessment. Overview Types of assessment Assessment for learning Assessment of learning Purpose of WBA Benefits of WBA Miller’s Pyramid.
Clinical Skills Verification rater Training MODULE 2 Training Faculty Evaluators of Clinical Skills: Drivers of Change in Assessment Joan Anzia, M.D. Tony.
Joan E. St. Onge, M.D. UMMSM At Holy Cross Hospital Internal Medicine Residency Faculty Development January 23, 2013 The Evaluation Toolkit.
An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.
An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.
Assessment of Clinical Competence in Health Professionals Education
Accreditation Council for Graduate Medical Education © 2014 Accreditation Council for Graduate Medical Education Competency-based Medical Education (CBME)
Comparison: Traditional vs. Outcome Project Evaluative Processes Craig McClure, MD Educational Outcomes Service Group University of Arizona December 2004.
Debriefing in Medical Simulation Manu Madhok, MD, MPH Emergency Department Children’s Hospital and Clinics of Minnesota.
Triple C Competency-based Curriculum: Implications for Family Medicine Residency Programs.
Core Competency Assessment in Emergency Medicine from Design to Implementation Christian Arbelaez, MD, MPH Assistant Residency Director Harvard Affiliated.
ACGME OUTCOME PROJECT : THE PROGRAM COORDINATOR’S ROLE Jim Kerwin, MD University of Arizona.
Medical education research in the Department of Family Medicine Shelley Ross Faculty Development Date: February 14, 2013.
Paired Observation and Video Editing Blending Continuous Feedback and Self-Directed Learning to Assess Communication Competency Larry Mauksch, M.Ed, University.
Moving beyond the psychometric discourse: A model for programmatic assessment Researching Medical Education Association for the Study of Medical education.
Triple C Competency-based Curriculum A Brief Overview for Residents Copyright © 2013 The College of Family Physicians of Canada.
BRIDGET C. O’BRIEN, PhD UNIVERSITY OF CALIFORNIA, SAN FRANCISCO Educating Physicians: A focus on integration, inquiry, innovation and improvement.
Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,
Writing Narratives Based on ACGME Competencies. Narratives What Are They?  Written Evaluation of Student Performance Formative  Mid-Course Evaluation.
Meaningful Evaluation: Framework, Process, Impact Inis Jane Bardella, M.D., FAAFP Associate Dean for Faculty Development and Global Health Initiatives.
Developing an Assessment System B. Joyce, PhD 2006.
Clinical Competency Committees What Faculty need to know Academic Affairs Committee ACEP,JMTF, CORD 1.
“R.I.M.E.” MODEL – A SYNTHETIC EVALUATION CONCEPT R eporter I nterpreter M anager- E ducator Pangaro LN. A new vocabulary and other innovations for improving.
Assessment Tools.
What is “Competency” in the New Millennium? Shirley Schlessinger, MD, FACP Associate Dean for Graduate Medical Education University of Mississippi Medical.
What is a Planned Curriculum?
Competency Based Medical Education Coming Soon to A University Near You! Dr. Janice Chisholm October 21, 2015.
Assessing Learners The Teaching Center Department of Pediatrics UNC School of Medicine The Teaching Center.
Mount Auburn Practice Improvement Program (MA-PIP)
Educational Outcomes Service Group: Overview of Year One Lynne Tomasa, PhD May 15, 2003.
John Doe’s Educational Achievement Data Competencies CanMeds ACGME Activities and Responsibilities Event based Date of report: October 5, 2011.
Not Just “MK-1” How learning the skills of EBM relates to the pediatric milestones Martha S Wright, MD, MEd Rainbow Babies and Children’s Hospital.
Strategies And Tools to Teach Patient Centered Interactions: Larry Mauksch, M.Ed Senior Lecturer Department of Family Medicine University of Washington.
The use of OSCE to assess Patient Care, Professionalism and Interpersonal Communication Milestones in EM residents Miriam Kulkarni, MD, Harsh Sule, MD,
Strategies and Tools to Teach Patient Centered Interactions: Blending Efficiency and Quality Larry Mauksch, M.Ed Consultant and Trainer Senior Lecturer.
Advising and Mentoring: Guiding Learners to Success in Residency and Beyond Raj Woolever, MD, FAAFP Program Director Deborah Taylor, PhD Associate Program.
INTRODUCTION TO ASSESSMENT METHODS USED IN MEDICAL EDUCATION AND THEIR RATIONALE.
Qualitative Analysis of Student- Patient Interviews in Underserved Clinics William B. Shore, MD Jessica Muller, PhD George Saba, PhD UCSF-Family and Community.
Outline Introduction  WHAT DID YOU SAY IN OUR SURVEY?
Patricia Kokotailo, Sarah Pitts, Sheryl Ryan, Karen Soren, Maria Trent
IDENTIFICATION, DIAGNOSIS & REMEDIATION OF THE STRUGGLING LEARNER Jeannette Guerrasio, MD University of Colorado, SOM Chris Knight, MD University of Washington,
Defining Grades in the Surgery Clerkship Jeremy M. Lipman, MD MetroHealth Medical Center Case Western Reserve University School of Medicine.
Click to edit Master subtitle style Competence by Design (CBD) Foundations of Assessment.
Henry M. Sondheimer, MD Association of American Medical Colleges 7 August 2013 A Common Taxonomy of Competency Domains for the Health Professions and Competencies.
Objectives: Participants will be able to: 3 Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85:
Preparing for the ACGME's Next Accreditation System (NAS) A Prospective View for Family Medicine Residencies Joseph J. Brocato, PhD, University of Minnesota.
Approaching Milestones Documentation: Tricks, Tips, and Examples Describing What We Want in a Family Physician: From Competencies to Milestones Allen F.
Maria Gabriela Castro MD Archana Kudrimoti MBBS MPH David Sacks PhD
Use of Mentored Residency Teams to Enhance Addiction Medicine Education Maureen Strohm, MD, Ken Saffier, MD, Julie Nyquist, PhD, Steve Eickelberg, MD MERF.
Strategies And Tools to Teach Patient Centered Interactions: Larry Mauksch, M.Ed Senior Lecturer Department of Family Medicine University of Washington.
+ Priory Curriculum & Instruction Faculty Inservice, August 2014.
CBME and CanMEDS 2015 Susan Glover Takahashi, MA, PhD Laura Leigh Murgaski, MScCH Lisa St. Amant Jesse Montgomery Kim O’Hearn.
A Multidisciplinary Transitions in Care Workshop for Medical Students
Entrustable Professional Activities
The purpose of medical education is to improve patient care
Angela Buffington, PhD, MA
EPAs as a Tool for Resident Evaluation
EPAs as Curriculum Tools
Assessment 101 Zubair Amin MD MHPE.
Assessment of Clinical Competencies
Presentation transcript:

How Behavioral Science Faculty Can Help Competency Assessment Glow and Flow Larry Mauksch, M.Ed Senior lecturer, Department of Family Medicine University of Washington School of Medicine Incoming Editor, Families, Systems, and Health Presentation and video will be placed on FMDRL Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Disclosures  Grant support from the Picker Foundation and Arthur Vining Davis Foundations  Paid private consultant to academic and community health organizations  Paid by STFM as an “On the Road” faculty  Co-owner of a formative assessment and development software company, MedFAD.com Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Outline Background: NAS, CBE, Milestones, EPAs Understand the differences and connections between formative and summative assessment Direct observation is criticalIt is about you, not the toolPromote tacit knowledge articulationDissolve evaluation PTSDSupport cultural change

Half Yearly Assessment Cycle Data collection and consolidation from multiple sources Given to: Resident Advisor CCC CCC provides summary impression ( with updated milestone rating) to resident and advisor Resident and Advisor meet to review summaries, CCC impressions and create IEP CCC summary and IEP to faculty influencing subsequent formative assessment

Milestones and Entrustable Professional Activities (EPAs) Milestones Read them? Used them?

Patient Care and procedures PC-1 Urgent and Emergent PC-2 Chronic illness PC-3 Health prevention promotion PC-4 Undiffer- entiated patients PC-5 Proced- ures Medical knowledge MK-1 breadth and depth MK-2 Critical thinking Professionalism Prof-1 Fully integrate Prof-2 Action awareness Prof-3 Humanism Cultural Prof-4 Self care growth Systems based practice SBP-1 Cost conscious SBP-2 Safety SBP-3 Advocate SBP-4 Coordin’te team care Practice based learning and improvement PBLI-1 Find / use evidence PBLI-2 Self direct’d learning PBLI-3 Improves system Communication C-1 Relation- ships C-2 Effective C-3 With team, others C-4 Improve w/ tech Family Medicine: 6 Competencies and 22 Milestones

Milestone Levels  Level 1: The resident demonstrates milestones expected of a resident who has had some education in family medicine.  Level 2: The resident is advancing and demonstrating additional milestones.  Level 3: The resident continues to advance and demonstrate additional milestones ; the resident consistently demonstrates the majority of milestones targeted for residency.  Level 4: The resident has advanced so that he or she now substantially demonstrates the milestones targeted for residency. This level is designed as the graduation target.  Level 5: The resident has advanced beyond performance targets set for residency and is demonstrating “aspirational” goals which might describe the performance of someone who has been in practice for several years. It is expected that only a few exceptional residents will reach this level.

Henry, Holmboe, Frankel. Evidence-based competencies for improving communication skills in graduate medical education: A review with suggestions for implementation. Med Teach. May 2013;35(5): Competency— ” ability to… Take accurate and complete patient historiesPatient Care 3, level 1 Communicate with other doctorsComm 3, L 1-5: SBP 4, L 3 Communicate w/ other health care team members Comm 3, L 1-5; SPB 4, L 3 Set agendasCommunication 2, Level 3 Assess and improve patient adherencePatient Care 2, Level 2-5 Deliver diagnostic and prognostic newsCommunication 2, Level 2,4 Elicit patients’ beliefs, perspectives, concernsC 1, L 2; C 2, L 3; Prof 3, L 2 Treatment plansPC 1, L 2-3; PC 2 L 2-3; PC 4, L 4; Prof 3, L 3 Establish rapport and demonstrate empathyC 1 L 2-4; PC 4, L4; Prof 3, L 1 Manage conflict and negotiate with patientsCommunication 1, Level 4 Basic patient counseling skillsCommunication 2, Level 4 Counseling families and caregiversCommunication 2, Level 4

Enstrustable Professional Activities Olle ten Cate, PhD Journal of Graduate Medical Education, March 2013 “Tasks or responsibilities that can be entrusted to a trainee once sufficient, specific competence is reached to allow for unsupervised execution” Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Entrustable Professional Activities and the Milestones Larry Mauksch, M.Ed University of Washington Department of Family Medicine EPA: Newborn care Common mental illness Chronic illness in adults Health behavior change Office Procedures End of life and palliative care IPCProfPt CareSBPPBLIMK

Both are Necessary and Neither are Sufficient for CBE EPAs Milestones Skills Knowledge Attitudes Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Which Statements are true? 1. Milestones are designed for reporting resident development to the ACGME 2. Milestones can help guide your teaching of residents throughout training 3. It is expected that a mature family medicine curriculum will produce most graduating residents at level 5 for all milestones 4. The milestones represents years of educational research about professional development in family medicine Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Which Statements are true? 1. Milestones are designed for reporting resident development to the ACGME 2. Milestones can help guide teaching of residents throughout training 3. It is expected that a mature family medicine curriculum will produce most graduating residents at level 5 4. The milestones represents years of educational research about professional development in family medicine Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Miller’s Assessment Pyramid Faculty observation (with real patients) Does Standardized patients Shows how Matching or critical response questioning Knows how Multiple choice questions Knows Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance Acad Med 65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby

Time to Achieve Reliability Across Tools Medical Education 2005; 39 : 309–317 (references to studies omitted but are in the article) InstrumentDescription1 hr2 hrs4hrs8hrs Multiple ChoiceShort stem ,93 Patient management prob Simulation of patient Key feature caseShort pt case, followed by write in answer Oral examinationOral exam on pt cases Long case examOral exam on previously unobserved real patient OSCESimulate realistic encounters, round robin Mini- CEXShort follow up oral exam on observed real patient Practice video assessment Selected encounters from real practice Incognito SPReal consults scored by undetected SPs

Miller’s Assessment Pyramid Contextual assessment Entrustable professional activities Qualitative measure relying on faculty expertise Does Discrete KSAs Standardized Discrete tools Shows how Knows How Knows Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance Acad Med 65(9 Suppl): S63-7. Van Vleuten and Schuwirth. Assessing professional competence: from methods to program. Medical Education, 2005,

Understand the Differences and Connections between Formative and Summative Assessment Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Formative vs Summative Assessment to Meet ACGME Requirements 1. Formative assessment is not necessary if you have a high quality summative assessment program 2. A high quality summative assessment is a function of high quality formative assessment Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Model of Program Assessment Van Der Vleuten et al. Medical Teacher 2012; 34: 205–214 Principles of Assessment Any single data point is flawed Any standardized assessment can have validity built into the instrument Validity of non-standardized assessment resides in the users and not so much in the instruments The number of data points used should be on a continuum and should be proportional to the stakes of the assessment Assessment drives learning by providing meaningful information to the learner Expert judgment is imperative

It is more about you than the tool van der Vleuten et al The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. Dec 2010;24(6): Trained faculty offer formative assessment influenced by context and focused on need in ways that checklists can not provide Multiple observations with purposeful sampling by multiple expert raters offering authentic assessments may provide a more valid picture of “does” than quantitative tools

Conflict of Interest Issues: Formative and Summative Mindsets Can faculty functioning in a formative mode, fairly block out summative thinking? Should faculty on the CCC provide formative assessment? How much involvement should residents have in creating formative notes?

Don’t allow quantitative reductionism caused by fulfilling ACGME reporting requirements to prevent you from providing meaningful summative statements to learners Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Direct Observation is Critical Ross V, Mauksch L, Huntington J, Beard JM. Interdisciplinary direct observation: impact on precepting, residents, and faculty. Fam Med. May 2012;44(5):

‘Documented observations with feedback is the sine qua non of competency based education’ M. Donoff, MD University of Alberta Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009;55(12): , e

Risks and Benefits of Checklists Which statements are false? 1. Using multiple instruments to assess multiple competencies may overwhelm programs because reliable sampling requires observations 2. They ignore context 3. Verbal anchors are better than numbers 4. They help teach observers as well as trainees 5. They help teach, reinforce and assess critical or broadly applied skills 6. Adding subjective narrative to checklists increases assessment value

Faculty Development Ineffable– we can know more than we tell The Tacit Dimension (1966) Michael Polanyi, MD, PhD Our challenge is to unpack, deconstruct, articulate As we do this our trainees grow and we grow as clinicians and teachers

Teachers as Learners Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med. Jul 2011;86(7): Egnew TR, Mauksch LB, Greer T, Farber SJ. Integrating communication training into a required family medicine clerkship. Acad Med. Aug 2004;79(8): Mauksch L, Farber S, Greer HT. Design, Dissemination, and Evaluation of an Advanced Communication Elective at Seven U.S. Medical Schools. Acad Med. Apr Beckman TJ. Lessons learned from a peer review of bedside teaching. Acad Med. Apr 2004;79(4):

Deconstructing by Creating SCRIPPT Scratch your head What is the problem? What did I see? How would I say or do this? Create an explanation, verbal example (out loud) or an action Interpret your understanding within a KSA category Practice --Have the trainee try or Praise to enhance awareness of skill use Troubleshoot- Consider pitfalls and refinements

Show first video- poor closing- - here

Deconstructing by Creating SCRIPPT Scratch your head What is the problem? What did I see? How would I say or do this? Create an explanation, verbal example (out loud) or an action Interpret your understanding within a KSA category Practice --Have the trainee try or Praise to enhance awareness of skill use Troubleshoot- Consider pitfalls and refinements

Show better video here- combining AVS and Teachback

Deconstructing by Creating SCRIPPT Scratch your head What is the problem? What did I see? How would I say or do this? Create an explanation, verbal example (out loud) or an action Interpret your understanding within a KSA category Practice --Have the trainee try or Praise to enhance awareness of skill use Troubleshoot- Consider pitfalls and refinements

Evaluation PTSD Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Evaluation PTSD Delayed, negative, often summative feedback with no formative context Shame and belittlement. Often experienced or observed in the presence of others Faculty may not admit that some knowledge is ineffable and resort to “pimping” Produces An unconscious barrier to providing helpful and supportive feedback or behavior that is replicated in one’s own teaching

Competency Based Education Implements and evaluates CBE in residency program Does Demonstrates use of direct observation feedback to learner Shows how Can describe and generate competency assessment tools/skills Knows how Can define formative and summative assessment Knows Impact onTraining Adapted from Miller, G. E. The assessment of clinical skills/competence/performance Acad Med65(9 Suppl): S

Behavioral Science Faculty Can Lead Manage your anxiety about the milestones. They help you know where to aim your teaching and deconstruction Remember, its about faculty development, not the toolsRole model deconstructing tacit knowledgeAddress evaluation PTSDLead with tracked, organized, formative assessment Larry Mauksch, M.Ed University of Washington Department of Family Medicine

ACGME website Webinar #3 Implementing Milestones, Clinical Competency Committees and Assessment, second section, by Pamela Derstine, PhD “ Assessment of Resident Milestones” Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009;55(12): , e Ross S, Poth CA, Donoff MG, et al. Involving users in the refinement of the competency-based achievement system: an innovative approach to competency- based assessment. Med Teach. 2012;34(2):e Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician. Sep 2011;57(9):e Dijkstra, Van Der Vleuten, Schuwirth A new framework for designing programs of assessment, Adv in Health Sci Ed , Miller, G. E. The assessment of clinical skills/competence/performance Acad Med 65(9 Suppl): S63-7. Competence Based Education: Selected articles and resources

Larry Mauksch, M.Ed University of Washington Department of Family Medicine Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 2012;34(3): Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. Apr 2011;86(4): Henry SG, Holmboe ES, Frankel RM. Evidence-based competencies for improving communication skills in graduate medical education: A review with suggestions for implementation. Med Teach. May 2013;35(5): Schuwirth LW, Van der Vleuten CP. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6): van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ. Mar 2005;39(3): van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34(3): van der Vleuten CP, Schuwirth LW, Scheele F, Driessen EW, Hodges B. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. Dec 2010;24(6):

EPA articles  March, 2013 Issue of the Journal of Graduate Medical Education  Carraccio CL, Englander R. From flexner to competencies: reflections on a decade and the journey ahead. Acad Med. Aug 2013;88(8):  ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32(8):  Chang A, Bowen JL, Buranosky RA, et al. Transforming primary care training--patient-centered medical home entrustable professional activities for internal medicine residents. J Gen Intern Med. Jun 2012;28(6): Larry Mauksch, M.Ed University of Washington Department of Family Medicine

Thank you Questions and Comments? Larry Mauksch, M.Ed University of Washington Department of Family Medicine