1. Teaching and Assessing in Competency Based Education Faculty Development Workshop
UCSF Current Development Team:
Patricia O’Sullivan, EdD; Erick Hung, MD; Lindsay Mazotti, MD
Contributors: Lee Learman MD, PhD; Susan Promes, MD

2. Creative Commons License
Attribution-NonCommercial-Share Alike 3.0 Unported
You are free:
to Share — to copy, distribute and transmit the work
to Remix — to adapt the work
Under the following conditions:
Attribution. You must give the original authors credit (but not in any way that suggests that they endorse you or your use of the work).
Noncommercial. You may not use this work for commercial purposes.
Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.
See for full license.

3. Objectives Describe competency based education
Identify promising teaching and assessment practices and strategies for successful implementation
Describe the relationship between teaching and assessing competencies and overall program evaluation.

4. Schedule 00 – 0:15 Welcome, Overview and Introductions
0:15 – 0:45 Introduction of Competency
0: :45 Teaching Strategies
1:45 - 2:00 Break
2:00 – 3:30 Assessment Tools
3:30 – 3:45 Implementation and summary
3:45 – 4:00 Performance Assessment & Evaluation
Lee will do
All facilitate 1 or 2 small groups.
Each do a set up for three
Teaching Pat
Assessment Christy
Implementation Lee

5. Competency Based Education
An outcomes based approach to the design, implementation, assessment and evaluation of a medical program using an organizing framework of competencies
The unit of progression is mastery of specific knowledge, skills and attitudes

6. Shifting the Paradigm
A competency-based system of education requires four components:
identifying the outcomes
defining performance levels for each competency
developing a framework for assessing competencies
continuous evaluation of the program to see if it is indeed producing the desired outcomes
Away from Flexner. Competence=knowledge.

7. Step 1: Identifying the Outcomes
Determine a competency framework comprised of broad “domains of competence” that, in aggregate, define the desired outcome.
Examples: ACGME/ABMS framework and the CanMeds Framework
There is a reference called the Physician Competency Reference that outlines 58 competencies in 8 domains.
At UCSF we utilize the 6 standard domains of

8. Desired Outcome: Driving A Car
Competency Domains
1. Knowledge
traffic laws
controls of a car
dynamics of the car
2. Skill
Physical skill at controlling car
Computational skill/process of adaptation
3.Attitude
“Professionalism”- Etiquette of driving
Self-control, anger management
I want to produce someone who can drive a car.
What would be the domains of competence needed to drive a car? K, S, A

9. Step 2: Define Performance Levels (Milestones)
Markers of achievement of levels of performance (ability) in a developmental continuum.
Can roughly corresponding to stages of progress: novice through expert.

10. Desired Outcome: Driving A Car
Domain of Competence: Knowledge Competency: Traffic laws
Milestones:
Novice:
Proficient:
Expert:
In the ONE domain of Knowledge, specifically re: Traffic laws

11. Dreyfus Model
Dreyfus Model of Skill Acquisition (2004)

12. Desired Outcome: Driving A Car
Domain of Competence: Knowledge
Sub competency: Traffic laws
Milestones:
Novice: Describe common traffic laws in the US
Proficient: Identify three main traffic citations in US
Expert: Contrast US traffic laws and laws in UK
In the ONE domain of Knowledge, specifically re: Traffic laws

13. Step 3: Develop a Framework for Assessment
How do I know if my student can drive a car?
Imagine an observable, measurable activity that a driver must be trusted to perform that crosses domains of competence.
How do I know if my student can drive?
Supervise them first until they are ready to perform without supervision.

14. Step 3: Develop a Framework for Assessment
How do I know if student can drive a car?
Responds to unexpected changes on the road (Yields appropriately to a biker in the crosswalk)
Manages car on limited access roads (Merges onto a freeway)
Parallel parks
How do I know if my student can drive?
I can test them on the rules of the road OR I can put them through the day to day practice of driving and see how they perform.
Supervise them first until they are ready to perform without supervision.
EPAs

15. Competency based education means that
Students must complete numerous assessments to demonstrate skills.
Students are ranked by proficiency level.
Individuals can complete whenever they have demonstrated competency.
Students work to achieve at least a minimal level of skill.

16. Which of the following is a competency for US residents?
Interprofessionalism
Quality Improvement
Professionalism
Procedural skills

17. UCSF’s (and ACGME’s) Competency Framework
Medical Knowledge
Patient Care
Practice Based Learning and Improvement
Systems Based Practice
Professionalism
Interpersonal and Communication Skills
ACGME

18. Other Competency Frameworks
CANMEDS
Physician Competency Reference Set (also includes interprofessional collaboration and personal & professional development)

19. Which statement best describes the relationship among competencies, milestones and EPAs?
EPAs>competencies>milestones
Competencies>milestones>EPAs
Milestones>Competencies>EPAs

20. Milestone

21. EPA defined
A core unit of work reflecting a responsibility that should only be entrusted upon someone with adequate competencies
Olle ten Cate
Medical Teacher 2010;32:
CONTEXT is key!
More specific:
Part of essential professional work in a given context
Independently executable, within a time frame
Leads to recognized output of professional labor
Observable and measurable in process and outcome, leading to a conclusion (“well done” or “not well done”)
Must require sufficient, specific knowledge, skill and attitude, generally acquired through training
Should reflect competencies, important to be acquired
Usually confined to qualified personnel only

22. Entrustable Professional Activities (EPAs)
EPAs together constitute the core of the profession
Make “decisions of entrustment” for “entrustable activity”

23. Viewpoint: Competency-Based Postgraduate Training: Can We Bridge the Gap between Theory and Clinical Practice?
ten Cate, Olle; Scheele, Fedde Academic Medicine. 82(6): , June 2007. 2

24. Entrustable Professional Activities (EPAs)
EPAs together constitute the core of the profession
Make “decisions of entrustment” for “entrustable activity”

25. Programs should document and demonstrate
Learning opportunities in each competency domain
Evidence of multiple assessment methods
Use of aggregate data to improve the educational program

26. Teaching Strategies 0:45– 1:15 Group Activity – Madness to Methods
1:15 – 1:45 Share Best Method to Objective Match
__________________________________________________
Each group nominate one or two “best method to objective match”
Use other than Patient care and medical knowledge.
Get glossary
Identify quietly someone to be the facilitator in each group to make sure people are working through workbook and that they stay on schedule
3 tables of 6-8

27. Playing Method to Madness
Each player receives methods cards equal to the number of players in the game, + one extra (5 players = 6 cards per player).
Dealer turns over the first objective card leaving it face up in the table. Each player chooses one methods card from his/her hand that will achieve the teaching objective. A player holding a “wild card” can create a new or use a previously discussed method.
Beginning clockwise from the dealer, all players place their chosen methods card face up near the objective card in the center of the table.
Each player has one minute to persuade the other players that their teaching method will effectively, efficiently and appropriately achieve the objective.
Once all players have presented their “method”, all players vote for the best method for achieving the teaching objective, but cannot vote for themselves.
Players receive a point for each vote they receive.
The dealer tallies the votes by player and records on a score sheet.
The winner is the player with the most votes.
Simpson D, Fenzel J, Rehm J, Marcdante K. Enriching Educators' Repertoire of Appropriate Instructional Methods . MedEdPORTAL; Available from:

28. Debrief the Game
What were examples of good matches between objective and method?
What did you learn from the game?
Large variety of teaching methods
Some work better than others for a specific objective
Match objective and teaching method

29. BREAK

30. Why assess? Whatever we measure, we tend to improve David Leach,
former CEO of ACGME

31. What makes a good assessment tool?
Appropriate for what you are measuring
User must know how to use it
Assessors trained
Learners prepared
Monitored results to make sure the tool is working

32. What makes a good assessment of a learner?
Sufficient data
Sufficient contact with learner
Sufficient sources of information
Ability to render a judgment based on the information

33. Miller’s Assessment Triangle
Does
Shows How
Knows How
Knows
Record Review
Patient Survey
360° Ratings
Undercover SPs
Observation
Video logs
Checklists
OSCE’s
SP’s
MCQ’s
Checklists
Procedure/Case Logs
Clinical context-based tests
Oral exams
Essays
What kinds of assessments go along Miller’s pyramid of competence
Factual tests
Oral exams
MCQ’s
Essays 52

35. Assessment Tools
2:00– 2:30 Group Activity – Matching Assessment and Objectives
2:30 – 3:00 Share Best Practices
__________________________________________________
Play the game again, but this time with assessment cards.
Deal each person 4 assessment cards
Select one objective-assessment method pair:
Why is it a good match? How will assessors be trained?
How will data be collected?
How frequently will data be collected?
ACGME grid is useful but only with caveat about changes of 1, 2, 3 as recommended

36. Learner Evaluation Expectations
Formative
An assessment should occur for each learning experience
Assessments of competency/milestones reflective of the 6 areas
Multiple formats & evaluators (faculty, peers, patients, self, others)
Progressive improvement by learner across the continuum
Documented
At least semi-annual evaluation for residents
At end of course/clerkship/rotation
Summative
Screening Committee review
Competency Committee review
In other health professions schools?

37. Clinical Competency Committees
Must have a written description of committee responsibilities
Review all resident evaluations by all evaluators semi-annually
Prepare and assure the reporting of Milestones evaluations of each resident semiannually to ACGME
Make recommendations to program director for resident progress (promotion, remediation, dismissal)

39. Reflect on Teaching and Assessing to Evaluate the Program
We will hand out sample workflow documents from Lee
Blank work flow sheet

40. Program Improvement Formal systematic evaluation annually
Must monitor and track:
Learner performance
Faculty development
Graduate performance
Program evaluations by learners and faculty
Must use results to improve the program
Written plan of action
Reviewed and approved by teaching faculty

41. Review of Objectives Describe competency based education
Identify promising teaching and assessment practices and strategies for successful implementation
Describe the relationship between teaching and assessing competencies and overall program evaluation.

42. Performance Assessment Teach for UCSF Certificate
Please complete skills assessment and evaluation to fulfill requirements of the workshop

43. UCSF Resources Glossary of Competencies and Assessments
Evaluation and Assessment for GME Programs
UCSF UME Competencies