Teaching and Assessing in Competency-Based Education Faculty Development Workshop Liz Gatewood, DNP, RN, FNP-C, CNE Renee Kinman, MD, PhD, MEd Liz and Renee UCSF Current Development Team: Patricia O’Sullivan, EdD; Erick Hung, MD; Lindsay Mazotti, MD Contributors: Lee Learman MD, PhD; Susan Promes, MD

Creative Commons License Attribution-NonCommercial-Share Alike 3.0 Unported You are free: to Share — to copy, distribute and transmit the work to Remix — to adapt the work Under the following conditions: Attribution. You must give the original authors credit (but not in any way that suggests that they endorse you or your use of the work). Noncommercial. You may not use this work for commercial purposes. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. See http://creativecommons.org/licenses/by-nc-sa/3.0/ for full license. Liz

Objectives Describe competency-based education Identify promising teaching & assessment practices & strategies for successful implementation Describe the relationship between teaching & assessing competencies & overall program evaluation

What do each of you want to get out of this workshop?

Schedule 09:00 – 09:15 Welcome, Overview, & Introductions 09:15 – 09:45 Introduction of Competency 09:45 - 10:45 Teaching Strategies 10:45 - 11:00 Break 11:00 – 12:30 Assessment Tools 12:30 – 12:45 Implementation & summary 12:45 – 1:00 Performance Assessment & Evaluation

Competency-Based Education: In with the new? Out with the old Renee

What is Competency-Based Education? An outcomes-based approach to the design, implementation, assessment, & evaluation of a medical program using an organizing framework of competencies In this type of approach, an individual needs to demonstrate mastery of specific knowledge, skills, & attitudes before moving on to the next level

Unlike the traditional Flexner-based medical education model, in a Competency-Based Medical Education model Just “being there” No longer gives you credit

What do we care more about? Instead, in a competency model, one needs to ask the following question: What do we care more about?

(e.g. variable time, but fixed outcomes) That the learning experience occurs in a time-specified manner? (e.g. fixed time, but variable outcomes) OR That the students learn the desired lessons & skills, regardless of the time spent learning? (e.g. variable time, but fixed outcomes)

What is the ultimate goal of medical education? But 1st: If we want to effectively discuss competency-based education, we need to start at the beginning by asking the following question: What is the ultimate goal of medical education? In other words, what do you think are the outcomes that we as health educators wish to achieve, & what are you looking for your trainees to accomplish? ”Outcome” According to Lucey et al*, the goal is to produce a physician workforce that is capable of & committed to providing reliably safe, timely, effective, efficient, equitable, & patient-centered care This means that physicians must not only master a large body of knowledge, but also possess the ability to apply that knowledge in service to others conduct themselves as professionals work effectively in teams communicate compassionately with patients & respectfully with colleagues collaborate to improve systems of care, & engage in critical reflection & lifelong learning * CR Lucey, GE Thibault, O ten Cate. Competency-Based, Time-Variable Education in the Health Professions: Crossroads. Academic Medicine. 2018;93:S1–S5

According to Lucey et al According to Lucey et al*, the goal is to produce a healthcare workforce that is capable of (& committed to) providing reliably safe, timely, effective, efficient, equitable, & patient-centered care This means that health care professionals must not only master a large body of knowledge, but also possess the ability to apply that knowledge in service to others conduct themselves as professionals work effectively in teams communicate compassionately with patients & respectfully with colleagues collaborate to improve systems of care, & engage in critical reflection & lifelong learning * CR Lucey, GE Thibault, O ten Cate. Competency-Based, Time-Variable Education in the Health Professions: Crossroads. Academic Medicine. 2018;93:S1–S5

As medical educators, we thus need to demonstrate to the public that our graduating physicians & other health care professionals are competent to perform these tasks

Thus in competency-based education, one needs to look at the outcomes goal, then develop a curriculum to meet that goal End goal: Production of a health professional capable of & committed to providing reliably safe, timely, effective, efficient, equitable, & patient-centered care ? Curriculum

What competency-based education programs are you already aware of? Disclaimer: For the purpose of this discussion, we’re going to talk about the ACGME (the physician arm), but obviously other health professions have their own protocols to ensure that their graduates are also competent

This shift in medical education began in 1999, when the Accreditation Council for Graduate Medical Education (ACGME) began to design a competency initiative called The Outcome Project

Prior to this time, the ACGME evaluated residency training programs on their “potential” to educate residents (e.g. what resources did the program provide their residents) But the point of The Outcome Project was to place an emphasis on actual resident accomplishments & assessment of outcomes

How was medical education supposedly going to benefit with a competency focus? Medical competence became more broadly defined Focus was on observed competence, not assumed competence solely because of length of experience Focus shifted from fixed time (3 years) & flexible standards to fixed standards but flexible time

The Outcome Project Officially Launched in 2002, with the introduction of the Six Domains of Clinical Competency

Overview of

These domains included the following...

But there were problems…. A decade after the initiation of the Outcome Project, still had confusion & controversy about how to integrate the competencies into training in a meaningful way Liz

Some of these operational problems included… Competence descriptions that were too analytical Assessment instruments that were inadequate Bureaucracy issues with collecting & reporting data

Yet another barrier was lack of understanding of how the knowledge, skills, & attitudes needed to perform these complex tasks develops over time

This is where the “Milestones” came in…

Milestone Emergency Medicine Patient Care Subcompetency: Emergency Stabilization Milestone

The purpose of the “Milestones” was to allow each of the individual specialties to identify the behaviors & attributes that describe the competencies each trainee was expected to have accomplished by the completion of each year of residency training

In other words, milestones are a means to track progress & skill acquisition in residency training by describing a developmental progression of observable behaviors within a set of previously described core competencies (Think of them as similar to the developmental milestones that we measure in pediatrics)

What has been your own experience with this What has been your own experience with this? (either the Milestones, or a similar competency-based program that your own profession uses) What have you found that has been useful? What have you experienced that drives you crazy?

So now, moving onto our last term: Entrustable Professional Activities (EPAs) EPAs together constitute the core of the profession Make “decisions of entrustment” for “entrustable activity”

EPAs are supposed to be simply the routine professional life activities of a physician based on their specialty & subspecialty They are supposed to designate that “a practitioner has demonstrated the necessary skills & attitudes to be trusted to independently perform this activity So an EPA for a pediatric hospitalist could be “to serve as the primary admitting pediatrician for previously well children suffering from common acute problems”

What has been your experience with EPAs thus far? Have any of your programs started to use them? Do you think that they might prove useful? What are problems you can foresee with trying to build these or use these?

Now that you have the overview, lets break down competency-based education into its different components… Renee

A competency-based system of education requires 4 components: Identification of the desired outcomes Defining performance levels for each competency Developing a framework for assessing competencies Continuous evaluation of the program to see if it is indeed producing the desired outcomes Away from Flexner. Competence=knowledge.

Step 1: Identifying the Outcomes Determine a competency framework comprised of broad “domains of competence” that, in aggregate, define the desired outcome To look at an appropriate analogy, let’s look at driving a car Ask participants their required competencies** Examples: ACGME/ABMS framework and the CanMeds Framework There is a reference called the Physician Competency Reference that outlines 58 competencies in 8 domains. At UCSF we utilize the 6 standard domains of NONPF

Define our desired outcome: To drive a car Competency Domains 1. Knowledge Traffic laws Controls of a car Dynamics of the car 2. Skill Physical skill at controlling car Computational skill/process of adaptation 3. Attitude “Professionalism”- Etiquette of driving Self-control, anger management I want to produce someone who can drive a car. What would be the domains of competence needed to drive a car? K, S, A

Step 2: Define Performance Levels (Milestones) Markers of achievement of levels of performance (ability) in a developmental continuum Can roughly corresponding to stages of progress: novice through expert Liz

Dreyfus Model Dreyfus Model of Skill Acquisition (2004)

Define Some Milestones Domain of Competence: Knowledge Sub-Competency: Traffic laws Novice Competent Expert ? Novice Competent Expert Describes common traffic laws in the US Identifies three most common traffic citations in the US Compares and contrasts US traffic laws & UK traffic laws In the ONE domain of Knowledge, specifically re: Traffic laws

Step 3: Develop a Framework for Assessment of Competence How do I know if my driver has reached these milestones? Novice Competent Expert Describes common traffic laws in the US Identifies three most common traffic citations in the US Compares and contrasts US traffic laws & UK traffic laws Renee Assessment options: MC test Short answer essay Simulation/Game Direct Observation Reflection Multisource evaluation

Step 4: Evaluate the Driving School Number of accidents in 1st year driving Traffic citations Do students achieve milestones as fast as other schools? Do students have a gap in one competency or fail to meet higher level milestones? Liz Does this program of learning actually prepare learners for the desired outcome?

To Summarize… Competency-based education means that… Students must complete numerous assessments to demonstrate skills Students are ranked by proficiency level Individuals can complete whenever they have demonstrated competency Students work to achieve at least a minimal level of skill Renee

But ultimately, it boils down to this… If a child can’t learn the way we teach, maybe we should teach the way they learn ~ Ignacio Estrada

Teaching Strategies 1 hour Group Activity – Method to Madness 30 minutes Share Best Method to Objective Match ________________________________________________ Each group should nominate one or two “best method to objective match” Liz Use other than Patient care and medical knowledge. Get glossary Identify quietly someone to be the facilitator in each group to make sure people are working through workbook and that they stay on schedule 3 tables of 6-8

Playing Method to Madness Each player receives methods cards equal to the number of players in the game, + one extra (5 players = 6 cards per player) Dealer turns over the first objective card leaving it face up in the table. Each player chooses one methods card from his/her hand that will achieve the teaching objective. A player holding a “wild card” can create a new or use a previously discussed method Beginning clockwise from the dealer, all players place their chosen methods card face up near the objective card in the center of the table Each player has one minute to persuade the other players that their teaching method will effectively, efficiently and appropriately achieve the objective Once all players have presented their “method”, all players vote for the best method for achieving the teaching objective, but cannot vote for themselves Players receive a point for each vote they receive The dealer tallies the votes by player and records on a score sheet The winner is the player with the most votes Simpson D, Fenzel J, Rehm J, Marcdante K. Enriching Educators' Repertoire of Appropriate Instructional Methods . MedEdPORTAL; 2010. Available from: www.mededportal.org/publication/7968

Debrief the Game What were examples of good matches between objective and method? What did you learn from the game? Large variety of teaching methods Some work better than others for a specific objective Match objective and teaching method Skills assessment

BREAK

Why do we need to assess? Whatever we measure, we tend to improve - David Leach, former CEO of ACGME Liz

What makes a good assessment tool? Appropriate for what you are measuring User must know how to use it Assessors trained Learners prepared Results are monitored to make sure the tool is working

What do you need to appropriately assess a learner? Sufficient data Sufficient contact with learner Sufficient sources of information Renee Ability to render a judgement based on the information provided

Miller’s Assessment Triangle Does Shows How Knows How Knows Record Review Patient Survey 360°Ratings Undercover SPs Observation Video logs Checklists OSCE’s SP’s MCQ’s Checklists Procedure/Case Logs Clinical context-based tests Oral exams Essays What kinds of assessments go along Miller’s pyramid of competence Factual tests Oral exams MCQ’s Essays 51

Assessment Strategies: Medbiquitous Liz https://medbiq.org/curriculum/vocabularies.pdf

We generally talk about two different types of assessment Formative Summative What do you think are the differences between each of these? How would we use each of them?

Formative Assessment An assessment should occur for each learning experience Assessments of competency/milestones should be reflective of the 6 areas of competency Should have multiple formats & evaluators (faculty, peers, patients, self, others) Looking for progressive improvement by learner across the continuum What needs to be documented? At least a semi-annual evaluation for residents At end of each course/clerkship/rotation

Summative Assessment Screening Committee review Competency Committee review May have different requirements in other health professions schools

Clinical Competence Committees Must have a written description of committee responsibilities Review all resident evaluations by all evaluators semi-annually Prepare and assure the reporting of Milestones evaluations of each resident semiannually to ACGME Make recommendations to program director for resident progress (promotion, remediation, dismissal) Renee

This easy-to-use tool can be used to help health professions educators reflect on current practices & diagnose assessment system deficits that may adversely affect decisions about learner progress & achievement in competency-based educational environments (Bierer, Academic Medicine 2018)

CASE STUDY

FNP Observed Structured Clinical Exams

Step 1: What is an objective? Independent Practice Competency Provides patient-centered care recognizing cultural diversity and the patient or designee as a full partner in decision-making

Step 2: Assessment method OSCE (Observed Structured Clinical Observation) Direct observation Clinical performance rating & checklist Standardized patient & faculty evaluate the student

Faculty Evaluation Patient Evaluation

Step 3: Workflow Continually evaluated through 2 year program 3 separate points / milestones Faculty completes Evaluation Formative assessment Summative Faculty provides feedback to learner Standardized patient provides feedback

Step 4: How is information used? Faculty meet to discuss all FNP students performance Identify students with weakness in communication Advisor provides individualized support May impact clinical placement (faculty rotation versus volunteer faculty)

Step 5: Learner orientation Orientation to program Policy Orientation to clinical rotations OSCE overview Practice sessions Website with OSCE FAQ, info, Kanbar info, policies including remediation

Step 6: Faculty orientation Orient to OSCE Rationale Goals Policies Student Feedback Train with experienced faculty Watch Practice concurrently (compare reaction & scores) Complete their own

Step 7: Good assessment? Is the tool feasible in the workplace? Are faculty completing it? Do learners appreciate the tool? Do faculty appreciate the tool? Are learners’ skills improving with the tool? Are faculty providing better feedback with the tool? Does the faculty feel confident that the tool measures the competency?

Program Improvement Formal systematic evaluation annually Must monitor and track: Learner performance Faculty development Graduate performance Program evaluations by learners and faculty Must use results to improve the program Written plan of action Reviewed & approved by teaching faculty

ASSESSMENT ACTIVITY Liz

Assessment Activity Think about a current assessment challenge in your education program. With a partner, share your assessment challenge: Why is it challenging? How is it challenging? Who is challenging for?

Steps in Selecting & Implementing an Assessment Think about a current assessment dilemma. This can be a competency domain, milestone, or EPA What is an objective related to that competency you find difficult to assess? List a method to assess that objective Outline the steps necessary to incorporate your assessment into a workflow? How is information from this assessment used? How do you orient your learners? How do you orient your faculty? How might you provide evidence that this was a good assessment?

NOW TO YOUR EXAMPLES!

The Future of Competency- Based Medical Education? Renee

The Future of CBME: Continuity Operationalizing CBME principles requires continuity within & across phases of education, training, & the practice continuum Opportunities: Link phases of learning Allow for sustained, meaningful relationships (with teachers, patients) Longitudinal integrated clerkships UMEGME continuum Englander & Carraccio, Acad Med 2018

The Future of CBME: Time-Variable Education Time as a resource, rather than a threat Allow learners to pursue advanced opportunities along developmental continuum if reach desired level of competence early Adapt system for learners needing additional time, who may not meet all competencies Focus on individualization of learning Lucey, Thibault, & ten Cate, Acad Med 2018

A Word about Faculty Development

Review of Objectives Describe competency based education Identify promising teaching & assessment practices & strategies for successful implementation Describe the relationship between teaching & assessing competencies & overall program evaluation.

Performance Assessment Teach for UCSF Certificate Please complete evaluation to fulfill requirements of the workshop http://tiny.ucsf.edu/Competencies

UCSF Resources Glossary of Competencies and Assessments http://meded.ucsf.edu/gme/competencies-assessment Evaluation and Assessment for GME Programs http://meded.ucsf.edu/gme/gme-evaluation-assessment-tools UCSF UME Competencies http://meded.ucsf.edu/ume/md-competencies