1. General competencies from an international perspective
Eric Holmboe

2. Employed by the Accreditation Council for Graduate Medical Education
disclosures
Employed by the Accreditation Council for Graduate Medical Education
Royalties for a textbook on assessment from Mosby-Elsevier

3. Rise of the competency and outcomes movement in medical education

4. Early signals
Increasing pockets of evidence and concern arise around the quality and safety of healthcare in the 1960s and 1970s
A.L. Cochrane: Effectiveness and efficiency
J. Wennberg: Unjustifiable regional variations in care delivery
R. Brook: medical errors

5. Quality of medical training
“Evidently it is not deemed necessary to assay students’ and residents’ clinical performance once they have entered the clinical years. Nor do clinical instructors more than occasionally show how they themselves elicit and check the reliability of the clinical data…

6. Quality of medical training
To a degree that is often at variance with their own professed scientific standards, attending staff all too often accept and use as the basis for discussion, if not recommendations, findings reported by students and residents without ever evaluating the reporter’s mastery of the clinical methods utilized or the reliability of the data obtained.”

7. Quality of medical training
From George Engel
1976 editorial on JAMA study highlighting deficiencies of student and resident’s basic clinical skills

8. Early Principles: CBmE
World Health Organization (1978):
“The intended output of a competency-based programme is a health professional who can practise medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.”
McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based Curriculum Development in Medical Education. World Health Organization, Switzerland, 1978.

9. CBME: Start with System Needs
Frenk J, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010 9

10. the ultimate Outcomes for clinical care & education
A competent (at a minimum) practitioner aligned with:
CMS Triple Aim

11. Good health care and social services defined

12. Treatment for osteoporosis: sweden
National Performance Assessment Quality and Efficiency of Care of Musculoskeletal Diseases in Sweden

13. transition to Unsupervised Practice: Gaps
Office-based Practice Competencies
Inter-Professional team skills
Clinical IT Meaningful Use skills
Population management skills
Reflective practice and CQI skills
Care Coordination
Continuity of Care
Leadership and management skills
Systems thinking
Procedural Skills
As you think about areas of remediation in trainees, it is also important to discuss how successful programs are at achieving outcomes that prepare trainees for practice. So what do end users think? The data is sobering.
Crosson Health Affairs 2011

14. Linking Clinical and Educational outcomes
Triple Aim
Competencies
National Health Service – UK.

15. can cbme be part of the solution?

16. Growing evidence and concern around quality and safety problems
CBME Drivers
Growing evidence and concern around quality and safety problems
Lack of attention to “21st century” competencies
“Uneven” product
Too many trainees graduating with deficiencies
Recognition of gaps in training
Desire to improve educational and clinical outcomes
Inflexible training models
“Pluri-potential stem cell” philosophy
Costs of training, including debt

17. Fundamental Characteristics of CBME
Graduate outcomes in the form of achievement of predefined desired competencies are the goal.
Competencies are derived from the needs of patients, organized into a coherent guiding framework.
Time is a resource for learning, not the basis of progression of competence.
Teaching and learning experiences are sequenced to facilitate an explicitly defined progression of ability in stages.

18. Fundamental Characteristics of CBME
Learning is tailored to the learner's individual progression in some manner.
Numerous direct observations and focused feedback contribute to effective learner development of expertise.
Assessment is planned, systematic, systemic, and integrative.

19. General Competencies: US
Patient care and procedural skills
Medical knowledge
Professionalism
Interpersonal skills and communication
Practice-based learning and improvement (PBLI)
Systems-based practice (SBP)

20. General Competency: PBLI
Residents must demonstrate the ability to investigate and evaluate their care of patients, to appraise and assimilate scientific evidence, and to continuously improve patient care based on constant self-evaluation and life-long learning.
Examples:
Evidence-based medicine
Quality improvement in clinical practice

21. General Competency: SBP
Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care, as well as the ability to call effectively on other resources in the system to provide optimal health care.
Examples:
Interprofessional teamwork
Care coordination such as discharge and consultation
Patient safety

22. Competencies help to define the educational outcomes (abilities)
Competency frameworks are just that – organizational frameworks to guide curriculum and assessment
Competencies help to define the educational outcomes (abilities)

23. Moving from the general to the specific

24. Milestones: general to specialty-specific
General Competencies
Patient Care
Medical Knowledge
Professionalism
Interpersonal Skills & Communication
PBL & I
Systems-based Practice
SPECIALTY TRANSLATION
Specialty Specific Milestones

25. Milestones - What Are They?
By definition a milestone is simply a significant point in development.
Milestones should enable the residents, fellows and the training program to better know an individual’s trajectory of competency development.

26. Developmental Progression or Set of Milestones
Competency
Developmental Progression or Set of Milestones
Sub-competency
Specific Milestone
PC1. History (Appropriate for age and impairment)
Level 1
Level 2
Level 3
Level 4
Level 5
Acquires a general medical history
Acquires a basic physiatric history including medical, functional, and psychosocial elements
Acquires a comprehensive physiatric history integrating medical, functional, and psychosocial elements
Seeks and obtains data from secondary sources when needed
Efficiently acquires and presents a relevant history in a prioritized and hypothesis driven fashion across a wide spectrum of ages and impairments
Elicits subtleties and information that may not be readily volunteered by the patient
Gathers and synthesizes information in a highly efficient manner
Rapidly focuses on presenting problem, and elicits key information in a prioritized fashion
Models the gathering of subtle and difficult information from the patient
This slide describes the anatomy the Milestone system. The General Competency is based in one of the six core competencies, and PC stands for Patient Care in this instance. Each Competency is further broken down into Subcompetencies, which comprise specific skills necessary to achieve the Competency as a whole. A Milestone is actually a description of how a resident is doing, and this slide contains 9 of them. The entire set of milestones for one Subcompetency is referred to as a Set of Milestones.
The levels do not refer to specific PGY levels of training; rather, they simply represent progression with a Level 3 Milestone being more advanced than a Level 2 Milestone. Residents will be expected to progress at different rates, and some will spend more time at one level than another level. Level 5 is aspirational – this is the expected performance level of someone who has been in practice a few years.

27. Milestone: Level Descriptions

28. Learning Curves Residency From Pusic, et. al. Acad Med. 2014
Another way to think about the Milestones is a learning curve (related to Dreyfus and Ericsson). This is a nice graphic from an article from Martin Pusic (NYU) of what a hypothetical learning curve likely looks like in residency – most residents will experience the curve highlighted in red.
From Pusic, et. al. Acad Med. 2014

29. Professional Development: Dreyfus Model
MILESTONES
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Expert/
Master
Proficient
Competent
Development is a
non-linear phenomenon
Advanced Beginner
Reality is skills develop over time and training. We expect different levels of competence for different skills. The Milestones were heavily influenced by developmental models of expertise, such as the work of the Dreyfus brothers and Anders Ericsson. This slide is based on the Dreyfus brothers. We also make the point that curriculum and assessment must be integrated: assessment drives learning but the “right learning” (curriculum) should also guide the appropriate choice of assessment.
Novice
Time, Practice, Experience
Dreyfus SE and Dreyfus HL. 1980
Carraccio CL et al. Acad Med 2008;83:761-7

30. Rebalancing assessment

31. Assessing for the Desired Outcome
Performance in Practice//Multi-source feedback/ Direct Observation
Work-based assessment is mostly accomplished through the observations and questions of faculty, team members, peers and other co-workers
Does (action)
Standardized Patients/Simulation
Shows How
(performance)
Diagnostic Reasoning using clinical vignettes or CSR
Knows How
(competence)
Knows
(knowledge)
Multiple choice Questions

32. Cambridge Model: “flipping” the Pyramid
Performance
System related influences
Individual related influences
Competence
If the goal is performance then need to workplace assessment. When you get to top this is where you need to do all of the meaninfgul assessment. Competence can do with MCQ but doesn’t translate to performance- we have robusts MCQ, but we don’t have WBA assessments
And just ebcause know does mean you do
Competency based assessments were defined as what doctors do in testing situations- performance based assessment defined as measure of what doctors do in practice. Cambridge flips millers- competence is first three levels of miller- greater focus on performance
Interaction of person and system
Work-based assessment has to be the primary focus of our assessment systems
Rethans, Norcini, et al, 2002

33. Common work-based Assessment Methods
Performance reviews using quality measures
Patient surveys
Chart stimulated recall
Direct observation
Video reviews
Portfolios
Multisource feedback
Procedural logs
Faculty evaluations
So here is your 20 minute reivew of common assessment methods
Records of clinical encounters: logbooks
I think the question isn’t really what is availabe to us, but rather

34. The Professional assessment “System”
Unit of Analysis:
Program
Residents
Feedback
Assessments within Program:
Direct observations
Audit and performance data
Multi-source FB
Simulation
ITExam
Accreditation J U D G M E N T
Qual/Quant “Data” Synthesis:
Committee P U B L I C
Feedback
Certification and Credentialing
Feedback
Faculty, PDs and others
Unit of Analysis:
Individual
Milestones and EPAs
as Guiding Framework and Blueprint
This is a Human Process

35. U.S. competency journey - milestones
Dates
Event/Change
1994
ACGME begins work on developing competencies
1999
Six general competency framework approved
2001
Launch of Outcomes Project
2007
First Milestone summit – Internal Medicine
2009
First Milestones published
Milestone sets created for all specialties
2013
First 7 specialties start using and reporting Milestones
2014
All specialties fully in system

36. Thank You