1. Teaching and Assessing in Competency-Based Education Faculty Development Workshop
Liz Gatewood, DNP, RN, FNP-C, CNE Renee Kinman, MD, PhD, MEd
Liz and Renee
UCSF Current Development Team:
Patricia O’Sullivan, EdD; Erick Hung, MD; Lindsay Mazotti, MD
Contributors: Lee Learman MD, PhD; Susan Promes, MD

2. Creative Commons License
Attribution-NonCommercial-Share Alike 3.0 Unported
You are free:
to Share — to copy, distribute and transmit the work
to Remix — to adapt the work
Under the following conditions:
Attribution. You must give the original authors credit (but not in any way that suggests that they endorse you or your use of the work).
Noncommercial. You may not use this work for commercial purposes.
Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.
See for full license.
Liz

3. Objectives Describe competency-based education
Identify promising teaching & assessment practices & strategies for successful implementation
Describe the relationship between teaching & assessing competencies & overall program evaluation

4. What do each of you want to get out of this workshop?

5. Schedule 09:00 – 09:15 Welcome, Overview, & Introductions
09:15 – 09:45 Introduction of Competency
09: :45 Teaching Strategies
10: :00 Break
11:00 – 12:30 Assessment Tools
12:30 – 12:45 Implementation & summary
12:45 – 1:00 Performance Assessment & Evaluation

6. Competency-Based Education:
In with the new?
Out with the old
Renee

7. What is Competency-Based Education?
An outcomes-based approach to the design, implementation, assessment, & evaluation of a medical program using an organizing framework of competencies
In this type of approach, an individual needs to demonstrate mastery of specific knowledge, skills, & attitudes before moving on to the next level

8. Unlike the traditional Flexner-based medical education model, in a Competency-Based Medical Education model
Just “being there”
No longer gives you credit

9. What do we care more about?
Instead, in a competency model, one needs to ask the following question:
What do we care more about?

10. (e.g. variable time, but fixed outcomes)
That the learning experience occurs in a time-specified manner? (e.g. fixed time, but variable outcomes) OR
That the students learn the desired lessons & skills, regardless of the time spent learning?
(e.g. variable time, but fixed outcomes)

11. What is the ultimate goal of medical education?
But 1st: If we want to effectively discuss competency-based education, we need to start at the beginning by asking the following question:
What is the ultimate goal of medical education?
In other words, what do you think are the outcomes that we as health educators wish to achieve, & what are you looking for your trainees to accomplish?
”Outcome”
According to Lucey et al*, the goal is to produce a physician workforce that is capable of & committed to providing reliably safe, timely, effective, efficient, equitable, & patient-centered care
This means that physicians must not only master a large body of knowledge, but also possess the ability to
apply that knowledge in service to others
conduct themselves as professionals
work effectively in teams
communicate compassionately with patients & respectfully with colleagues
collaborate to improve systems of care, &
engage in critical reflection & lifelong learning
* CR Lucey, GE Thibault, O ten Cate. Competency-Based, Time-Variable Education
in the Health Professions: Crossroads. Academic Medicine. 2018;93:S1–S5

12. According to Lucey et al
According to Lucey et al*, the goal is to produce a healthcare workforce that is capable of (& committed to) providing reliably safe, timely, effective, efficient, equitable, & patient-centered care
This means that health care professionals must not only master a large body of knowledge, but also possess the ability to
apply that knowledge in service to others
conduct themselves as professionals
work effectively in teams
communicate compassionately with patients & respectfully with colleagues
collaborate to improve systems of care, &
engage in critical reflection & lifelong learning
* CR Lucey, GE Thibault, O ten Cate. Competency-Based, Time-Variable Education in the Health Professions: Crossroads. Academic Medicine. 2018;93:S1–S5

13. As medical educators, we thus need to demonstrate to the public that our graduating physicians & other health care professionals are competent to perform these tasks

14. Thus in competency-based education, one needs to look at the outcomes goal, then develop a curriculum to meet that goal
End goal: Production of a health professional capable of & committed to providing reliably safe, timely, effective, efficient, equitable, & patient-centered care ?
Curriculum

15. What competency-based education programs are you already aware of?
Disclaimer:
For the purpose of this discussion, we’re going to talk about the ACGME (the physician arm), but obviously other health professions have their own protocols to ensure that their graduates are also competent

16. This shift in medical education began in 1999, when the Accreditation Council for Graduate Medical Education (ACGME) began to design a competency initiative called
The Outcome Project

17. Prior to this time, the ACGME evaluated residency training programs on their “potential” to educate residents (e.g. what resources did the program provide their residents)
But the point of The Outcome Project was to place an emphasis on actual resident accomplishments & assessment of outcomes

18. How was medical education supposedly going to benefit with a competency focus?
Medical competence became more broadly defined
Focus was on observed competence, not assumed competence solely because of length of experience
Focus shifted from
fixed time (3 years) & flexible standards to
fixed standards but flexible time

19. The Outcome Project Officially Launched in 2002, with the introduction of the Six Domains of Clinical Competency

20. Overview of

21. These domains included the following...

22. But there were problems….
A decade after the initiation of the Outcome Project, still had confusion & controversy about how to integrate the competencies into training in a meaningful way
Liz

23. Some of these operational problems included…
Competence descriptions that were too analytical
Assessment instruments that were inadequate
Bureaucracy issues with collecting & reporting data

24. Yet another barrier was lack of understanding of how the knowledge, skills, & attitudes needed to perform these complex tasks develops over time

25. This is where the “Milestones” came in…

26. Milestone Emergency Medicine Patient Care
Subcompetency: Emergency Stabilization
Milestone

27. The purpose of the “Milestones” was to allow each of the individual specialties to identify the behaviors & attributes that describe the competencies each trainee was expected to have accomplished by the completion of each year of residency training

28. In other words, milestones are a means to track
progress & skill acquisition in residency training
by describing a
developmental progression of observable behaviors
within a set of previously described core competencies
(Think of them as similar to the developmental milestones that we measure in pediatrics)

29. What has been your own experience with this
What has been your own experience with this? (either the Milestones, or a similar competency-based program that your own profession uses)
What have you found that has been useful?
What have you experienced that drives you crazy?

30. So now, moving onto our last term: Entrustable Professional Activities (EPAs)
EPAs together constitute the core of the profession
Make “decisions of entrustment” for “entrustable activity”

31. EPAs are supposed to be simply the routine professional life activities of a physician based on their specialty & subspecialty
They are supposed to designate that “a practitioner has demonstrated the necessary skills & attitudes to be trusted to independently perform this activity
So an EPA for a pediatric hospitalist could be “to serve as the primary admitting pediatrician for previously well children suffering from common acute problems”

32. What has been your experience with EPAs thus far?
Have any of your programs started to use them?
Do you think that they might prove useful?
What are problems you can foresee with trying to build these or use these?

33. Now that you have the overview, lets break down competency-based education into its different components…
Renee

34. A competency-based system of education requires 4 components:
Identification of the desired outcomes
Defining performance levels for each competency
Developing a framework for assessing competencies
Continuous evaluation of the program to see if it is indeed producing the desired outcomes
Away from Flexner. Competence=knowledge.

35. Step 1: Identifying the Outcomes
Determine a competency framework comprised of broad “domains of competence” that, in aggregate, define the desired outcome
To look at an appropriate analogy, let’s look at driving a car
Ask participants their required competencies**
Examples: ACGME/ABMS framework and the CanMeds Framework
There is a reference called the Physician Competency Reference that outlines 58 competencies in 8 domains.
At UCSF we utilize the 6 standard domains of
NONPF

36. Define our desired outcome: To drive a car
Competency Domains
1. Knowledge
Traffic laws
Controls of a car
Dynamics of the car
2. Skill
Physical skill at controlling car
Computational skill/process of adaptation
3. Attitude
“Professionalism”- Etiquette of driving
Self-control, anger management
I want to produce someone who can drive a car.
What would be the domains of competence needed to drive a car? K, S, A

37. Step 2: Define Performance Levels (Milestones)
Markers of achievement of levels of performance (ability) in a developmental continuum
Can roughly corresponding to stages of progress: novice through expert
Liz

38. Dreyfus Model
Dreyfus Model of Skill Acquisition (2004)

39. Define Some Milestones
Domain of Competence: Knowledge Sub-Competency: Traffic laws
Novice
Competent
Expert ?
Novice
Competent
Expert
Describes common traffic laws in the US
Identifies three most common traffic citations in the US
Compares and contrasts US traffic laws & UK traffic laws
In the ONE domain of Knowledge, specifically re: Traffic laws

40. Step 3: Develop a Framework for Assessment of Competence
How do I know if my driver has reached these milestones?
Novice
Competent
Expert
Describes common traffic laws in the US
Identifies three most common traffic citations in the US
Compares and contrasts US traffic laws & UK traffic laws
Renee
Assessment options:
MC test
Short answer essay
Simulation/Game
Direct Observation
Reflection
Multisource evaluation

41. Step 4: Evaluate the Driving School
Number of accidents in 1st year driving
Traffic citations
Do students achieve milestones as fast as other schools?
Do students have a gap in one competency or fail to meet higher level milestones?
Liz
Does this program of learning actually prepare learners for the desired outcome?

42. To Summarize… Competency-based education means that…
Students must complete numerous assessments to demonstrate skills
Students are ranked by proficiency level
Individuals can complete whenever they have demonstrated competency
Students work to achieve at least a minimal level of skill
Renee

43. But ultimately, it boils down to this…
If a child can’t learn the way we teach, maybe we should teach the way they learn
~ Ignacio Estrada

44. Teaching Strategies 1 hour Group Activity – Method to Madness
30 minutes Share Best Method to Objective Match
________________________________________________
Each group should nominate one or two “best method to objective match”
Liz
Use other than Patient care and medical knowledge.
Get glossary
Identify quietly someone to be the facilitator in each group to make sure people are working through workbook and that they stay on schedule
3 tables of 6-8

45. Playing Method to Madness
Each player receives methods cards equal to the number of players in the game, + one extra (5 players = 6 cards per player)
Dealer turns over the first objective card leaving it face up in the table. Each player chooses one methods card from his/her hand that will achieve the teaching objective. A player holding a “wild card” can create a new or use a previously discussed method
Beginning clockwise from the dealer, all players place their chosen methods card face up near the objective card in the center of the table
Each player has one minute to persuade the other players that their teaching method will effectively, efficiently and appropriately achieve the objective
Once all players have presented their “method”, all players vote for the best method for achieving the teaching objective, but cannot vote for themselves
Players receive a point for each vote they receive
The dealer tallies the votes by player and records on a score sheet
The winner is the player with the most votes
Simpson D, Fenzel J, Rehm J, Marcdante K. Enriching Educators' Repertoire of Appropriate Instructional Methods . MedEdPORTAL; Available from:

46. Debrief the Game
What were examples of good matches between objective and method?
What did you learn from the game?
Large variety of teaching methods
Some work better than others for a specific objective
Match objective and teaching method
Skills assessment

47. BREAK

48. Why do we need to assess? Whatever we measure, we tend to improve
- David Leach,
former CEO of ACGME
Liz

49. What makes a good assessment tool?
Appropriate for what you are measuring
User must know how to use it
Assessors trained
Learners prepared
Results are monitored to make sure the tool is working

50. What do you need to appropriately assess a learner?
Sufficient data
Sufficient contact with learner
Sufficient sources of information
Renee
Ability to render a judgement based on the information provided

51. Miller’s Assessment Triangle
Does
Shows How
Knows How
Knows
Record Review
Patient Survey
360°Ratings
Undercover SPs
Observation
Video logs
Checklists
OSCE’s
SP’s
MCQ’s
Checklists
Procedure/Case Logs
Clinical context-based tests
Oral exams
Essays
What kinds of assessments go along Miller’s pyramid of competence
Factual tests
Oral exams
MCQ’s
Essays 51

53. Assessment Strategies: Medbiquitous
Liz

54. We generally talk about two different types of assessment
Formative
Summative
What do you think are the differences between each of these?
How would we use each of them?

55. Formative Assessment
An assessment should occur for each learning experience
Assessments of competency/milestones should be reflective of the 6 areas of competency
Should have multiple formats & evaluators (faculty, peers, patients, self, others)
Looking for progressive improvement by learner across the continuum
What needs to be documented?
At least a semi-annual evaluation for residents
At end of each course/clerkship/rotation

56. Summative Assessment Screening Committee review
Competency Committee review
May have different requirements in other health professions schools

57. Clinical Competence Committees
Must have a written description of committee responsibilities
Review all resident evaluations by all evaluators semi-annually
Prepare and assure the reporting of Milestones evaluations of each resident semiannually to ACGME
Make recommendations to program director for resident progress (promotion, remediation, dismissal)
Renee

59. This easy-to-use tool can be used to help health professions educators reflect on current practices & diagnose assessment system deficits that may adversely affect decisions about learner progress & achievement in competency-based educational environments (Bierer, Academic Medicine 2018)

60. CASE STUDY

61. FNP Observed Structured Clinical Exams

62. Step 1: What is an objective?
Independent Practice Competency
Provides patient-centered care recognizing cultural diversity and the patient or designee as a full partner in decision-making

63. Step 2: Assessment method
OSCE (Observed Structured Clinical Observation)
Direct observation
Clinical performance rating & checklist
Standardized patient & faculty evaluate the student

64. Faculty Evaluation
Patient Evaluation

65. Step 3: Workflow Continually evaluated through 2 year program
3 separate points / milestones
Faculty completes Evaluation
Formative assessment
Summative
Faculty provides feedback to learner
Standardized patient provides feedback

66. Step 4: How is information used?
Faculty meet to discuss all FNP students performance
Identify students with weakness in communication
Advisor provides individualized support
May impact clinical placement (faculty rotation versus volunteer faculty)

67. Step 5: Learner orientation
Orientation to program
Policy
Orientation to clinical rotations
OSCE overview
Practice sessions
Website with OSCE FAQ, info, Kanbar info, policies including remediation

68. Step 6: Faculty orientation
Orient to OSCE
Rationale
Goals
Policies
Student Feedback
Train with experienced faculty
Watch
Practice concurrently (compare reaction & scores)
Complete their own

69. Step 7: Good assessment?
Is the tool feasible in the workplace? Are faculty completing it?
Do learners appreciate the tool?
Do faculty appreciate the tool?
Are learners’ skills improving with the tool?
Are faculty providing better feedback with the tool?
Does the faculty feel confident that the tool measures the competency?

70. Program Improvement Formal systematic evaluation annually
Must monitor and track:
Learner performance
Faculty development
Graduate performance
Program evaluations by learners and faculty
Must use results to improve the program
Written plan of action
Reviewed & approved by teaching faculty

71. ASSESSMENT ACTIVITY
Liz

72. Assessment Activity
Think about a current assessment challenge in your education program.
With a partner, share your assessment challenge:
Why is it challenging? How is it challenging?
Who is challenging for?

73. Steps in Selecting & Implementing an Assessment
Think about a current assessment dilemma. This can be a competency domain, milestone, or EPA
What is an objective related to that competency you find difficult to assess?
List a method to assess that objective
Outline the steps necessary to incorporate your assessment into a workflow?
How is information from this assessment used?
How do you orient your learners?
How do you orient your faculty?
How might you provide evidence that this was a good assessment?

74. NOW TO YOUR EXAMPLES!

75. The Future of Competency- Based Medical Education?
Renee

76. The Future of CBME: Continuity
Operationalizing CBME principles requires continuity within & across phases of education, training, & the practice continuum
Opportunities:
Link phases of learning
Allow for sustained, meaningful relationships (with teachers, patients)
Longitudinal integrated clerkships
UMEGME continuum
Englander & Carraccio, Acad Med 2018

77. The Future of CBME: Time-Variable Education
Time as a resource, rather than a threat
Allow learners to pursue advanced opportunities along developmental continuum if reach desired level of competence early
Adapt system for learners needing additional time, who may not meet all competencies
Focus on individualization of learning
Lucey, Thibault, & ten Cate, Acad Med 2018

78. A Word about Faculty Development

79. Review of Objectives Describe competency based education
Identify promising teaching & assessment practices & strategies for successful implementation
Describe the relationship between teaching & assessing competencies & overall program evaluation.

80. Performance Assessment Teach for UCSF Certificate
Please complete evaluation to fulfill requirements of the workshop

81. UCSF Resources Glossary of Competencies and Assessments
Evaluation and Assessment for GME Programs
UCSF UME Competencies