Teaching and Assessing in Competency Based Education Faculty Development Workshop Erick Hung, MD Associate Professor, Department of Psychiatry Lindsay Mazotti, M.D. Associate Professor, Department of Medicine, KLIC Clerkship Director Kaiser Permanente Oakland Patricia S. O’Sullivan, Ed.D. Office of Research and Development in Medical Education

Why are we here? Ensure that we advance learners in all competencies No gaps! Learners have had opportunities to learn and have been assessed in all areas in order to improve.

Objectives  Describe competency based education  Identify promising teaching and assessment practices and strategies for successful implementation  Describe the relationship between teaching and assessing competencies and overall program evaluation.

Schedule 00 – 0:15Welcome, Overview and Introductions 0:15 – 0:45 Competency Update 0:45 - 1:45Teaching Strategies 1:45 - 2:00Break 2:00 – 3:00Assessment Tools 3:00 – 3:30Implementation 3:30 – 3:45Summary 3:45 – 4:00 Performance Assessment & Evaluation

Competency Based Education 1.Explicit and clearly aligned with expected competencies; 2.Criteria-driven, focusing on accountability in reaching benchmarks and, ultimately, competence; 3.Grounded in “real-life” experiences; 4.Focused on fostering the learners’ ability to self-assess; 5.Individualized, providing more opportunities for independent study and achievement of milestones.

Name the Competencies used at UCSF Medical Knowledge Patient Care Practice Based Learning and Improvement Systems Based Practice Professionalism Interpersonal and Communication Skills

Milestones Explicate the general competencies by describing a developmental progression of observable behaviors Provide specific feedback and evaluation to learners Ensure acquisition of necessary knowledge skills and attitudes for advancement

Milestone

EPA defined A core unit of work reflecting a responsibility that should only be entrusted upon someone with adequate competencies Olle ten Cate Medical Teacher 2010;32: CONTEXT is key!

Entrustable Professional Activities (EPAs) Define important clinical activities Link competencies / milestones Include professional judgment of competence by clinicians

2 Viewpoint: Competency-Based Postgraduate Training: Can We Bridge the Gap between Theory and Clinical Practice? ten Cate, Olle; Scheele, Fedde Academic Medicine. 82(6): , June 2007.

Dreyfus Model

Your program should document and demonstrate Learning opportunities in each competency domain Evidence of multiple assessment methods Use of aggregate data to improve the educational program

Medical Knowledge Residents must demonstrate knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and how to apply this knowledge to patient care. Acquisition Analysis Application

Teaching Medical Knowledge Content Specialty specific curriculum that addresses key skill sets RRC Specialty specific requirements Setting Clinical Teaching Lectures/Seminars/Conferences Journal Club Procedural workshops Board review courses or formalized group study experiences Simulations Self directed learning through case based modules

Patient Care Residents must be able to provide patient care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health. Gathering information Synthesis Partnering with patients/families

Teaching Patient Care Content Specialty specific skills that address key skill sets Specialty specific procedural knowledge Knowledge about information technology Setting Clinical Teaching Lectures/Seminars/Conferences Role Modeling Workshops Simulations Self directed learning through case based modules

Practice Based Learning and Improvement Residents must be able to investigate and evaluate their patient care practices, appraise and assimilate scientific evidence, and improve their patient care practices. Life-long learning Evidence based medicine Quality improvement Teaching skills

Teaching Practice Based Learning and Improvement Content Self Reflection Quality or Practice Improvement Setting Self reflect on practice and determine improvement Lectures/Seminars/Conferences Quality Improvement Project

Teaching Practice Based Learning and Improvement (cont.) Content Evidence Based Medicine Teaching Skills Setting Lectures/Seminars/Conferences Journal Club Research Project Clinical Teaching Role Modeling Interactive Workshop

Systems-based Practice Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide optimal health care. Health care delivery system Cost effective practice Patient safety and advocacy/Systems causes of error

Teaching Systems Based Practice Content Health care system Different types of medical practice and delivery systems System resources System issues and the reduction of errors Conducting a root case analysis Setting Clinical teaching Patient Safety projects Systems based approach to M&M Lectures/Seminars/Conferences Interdisciplinary Teams

Teaching Systems Based Practice (cont.) Content Cost Effective Practice Tools and techniques for controlling costs and allocating resources Understanding of financing/insurance structures Mock practice experiences/mock financials Evaluating risk and benefit of costly prescribing Setting Clinical Teaching Practice management curricula or projects

Content Patient Safety and Advocacy Setting Lecture/Seminar/Conference Clinical teaching Individual or Group Projects Conducting a root cause analysis on near miss or sentinel event Systems based approach to M & M Teaching Systems Based Practice (cont.)

Professionalism Residents must demonstrate professionalism, as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population. Professional behavior Ethical principles Cultural competence

Teaching Professionalism Content Respect, altruism, integrity, honesty, compassion, empathy Setting Clinical teaching Case based teaching Mentoring Role modeling Role plays and clinical vignettes

Teaching Professionalism (cont.) Content Ethics Setting Clinical teaching Case based teaching Ethics Committee Mentoring Role modeling

Teaching Professionalism (cont.) Content Cultural Competence Setting Clinical teaching Case based teaching Interactive Workshops Lecture/Conference/Seminar Institutional Initiatives Role modeling Mentoring

Interpersonal and Communication Skills Residents must be able to demonstrate interpersonal and communication skills that result in effective information exchange and teaming with patients, their patients families, and professional associates. Communicating with patients and families Communicating with team members Scholarly Communication

Teaching Interpersonal and Communication Skills Content Communication with Patients and Families Setting Clinical teaching Role modeling Case based teaching Interactive workshops or seminars using role-plays

Teaching Interpersonal and Communication Skills (cont.) Content Communication with colleagues Setting Standardized communication around handoff Clinical teaching Role modeling Interactive workshops or seminars

Teaching Interpersonal and Communication Skills (cont.) Content Scholarly communication Setting Grand Rounds Presenting lectures, seminars, conferences Writing abstracts Presenting a poster Scholarly articles

UCSF Resources Glossary of Competencies and Assessments Evaluation and Assessment for GME Programs UCSF UME Competencies

Teaching Strategies 0:45– 1:15Group Activity – Madness to Methods 1:15 – 1:45Share Best Method to Objective Match __________________________________________________ Each group nominate one or two “best method to objective match”

Playing Method to Madness Each player receives methods cards equal to the number of players in the game, + one extra (5 players = 6 cards per player). Dealer turns over the first objective card leaving it face up in the table. Each player chooses one methods card from his/her hand that will achieve the teaching objective. A player holding a “wild card” can create a new or use a previously discussed method. Beginning clockwise from the dealer, all players place their chosen methods card face up near the objective card in the center of the table. Each player has one minute to persuade the other players that their teaching method will effectively, efficiently and appropriately achieve the objective. Once all players have presented their “method”, all players vote for the best method for achieving the teaching objective, but cannot vote for themselves. Players receive a point for each vote they receive. The dealer tallies the votes by player and records on a score sheet. The winner is the player with the most votes. Simpson D, Fenzel J, Rehm J, Marcdante K. Enriching Educators' Repertoire of Appropriate Instructional Methods. MedEdPORTAL; Available from:

Debrief the Game What did you learn from playing the game?

Debrief the Game What did you learn from playing the game? Large variety of teaching methods Some work better than others for a specific objective Match objective and teaching method

Why assess? Whatever we measure, we tend to improve David Leach, former CEO of ACGME

What makes a good assessment tool? Appropriate for what you are measuring User must know how to use it Monitor tool’s use to make sure it is working

What makes a good assessment of a learner? Sufficient data Sufficient contact with learner Sufficient sources of information Ability to render a judgment based on the information

Miller’s Assessment Triangle Does Shows How Knows How Knows Record Review Patient Survey 360° Ratings Undercover SPs Observation Video logs Checklists OSCE’s SP’s MCQ’s Checklists Procedure/Case Logs Clinical context-based tests Oral exams Essays Factual tests Oral exams MCQ’s Essays

Assessment Tools 2:00 – 2:30Group Activity – Looking Glass 2:30 – 3:00Share Best Practices __________________________________________________ For the competency your group selected:  Identify/share the methods your program is currently using to assess this competency (in addition to any global assessments you conduct after a rotation)  What specific tools do you use, how frequently, and how you prepare the learners and evaluators for the assessment?  Discuss the challenges you have faced and surmounted in implementing these assessments.  Nominate one or two “best practices” from the group.

Learner Evaluation Expectations Formative –An assessment should occur for each learning experience –Assessments of competency/milestones reflective of the 6 areas –Multiple formats & evaluators (faculty, peers, patients, self, others) –Progressive improvement by learner across the continuum –Documented At least semi-annual evaluation for residents At end of course/clerkship/rotation Summative –Screening Committee review –Competency Committee review –In other schools?

Clinical Competency Committees Must have a written description of committee responsibilities –Review all resident evaluations by all evaluators semi-annually –Prepare and assure the reporting of Milestones evaluations of each resident semiannually to ACGME –Make recommendations to program director for resident progress (promotion, remediation, dismissal)

Reflect on Teaching and Assessing to Evaluate the Program

Program Improvement Formal systematic evaluation annually Must monitor and track: –Learner performance –Faculty development –Graduate performance –Program evaluations by learners and faculty Must use results to improve the program –Written plan of action –Reviewed and approved by teaching faculty

Review of Objectives  Describe competency based education  Identify promising teaching and assessment practices and strategies for successful implementation  Describe the relationship between teaching and assessing competencies and overall program evaluation.

Performance Assessment Teach for UCSF Certificate Follow the link provided in the All Participants: –Complete Action Plan –Complete Evaluation