General competencies from an international perspective Eric Holmboe

Employed by the Accreditation Council for Graduate Medical Education disclosures Employed by the Accreditation Council for Graduate Medical Education Royalties for a textbook on assessment from Mosby-Elsevier

Rise of the competency and outcomes movement in medical education

Early signals Increasing pockets of evidence and concern arise around the quality and safety of healthcare in the 1960s and 1970s A.L. Cochrane: Effectiveness and efficiency J. Wennberg: Unjustifiable regional variations in care delivery R. Brook: medical errors

Quality of medical training “Evidently it is not deemed necessary to assay students’ and residents’ clinical performance once they have entered the clinical years. Nor do clinical instructors more than occasionally show how they themselves elicit and check the reliability of the clinical data…  

Quality of medical training To a degree that is often at variance with their own professed scientific standards, attending staff all too often accept and use as the basis for discussion, if not recommendations, findings reported by students and residents without ever evaluating the reporter’s mastery of the clinical methods utilized or the reliability of the data obtained.”  

Quality of medical training From George Engel 1976 editorial on JAMA study highlighting deficiencies of student and resident’s basic clinical skills  

Early Principles: CBmE World Health Organization (1978): “The intended output of a competency-based programme is a health professional who can practise medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.” McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based Curriculum Development in Medical Education. World Health Organization, Switzerland, 1978.

CBME: Start with System Needs Frenk J, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010 9

the ultimate Outcomes for clinical care & education A competent (at a minimum) practitioner aligned with: CMS Triple Aim

Good health care and social services defined

Treatment for osteoporosis: sweden National Performance Assessment 2014. Quality and Efficiency of Care of Musculoskeletal Diseases in Sweden

transition to Unsupervised Practice: Gaps Office-based Practice Competencies Inter-Professional team skills Clinical IT Meaningful Use skills Population management skills Reflective practice and CQI skills Care Coordination Continuity of Care Leadership and management skills Systems thinking Procedural Skills As you think about areas of remediation in trainees, it is also important to discuss how successful programs are at achieving outcomes that prepare trainees for practice. So what do end users think? The data is sobering. Crosson Health Affairs 2011

Linking Clinical and Educational outcomes Triple Aim Competencies National Health Service – UK. http://www.wipp.nhs.uk/tools_gpn/unit6_education.php

can cbme be part of the solution?

Growing evidence and concern around quality and safety problems CBME Drivers Growing evidence and concern around quality and safety problems Lack of attention to “21st century” competencies “Uneven” product Too many trainees graduating with deficiencies Recognition of gaps in training Desire to improve educational and clinical outcomes Inflexible training models “Pluri-potential stem cell” philosophy Costs of training, including debt

Fundamental Characteristics of CBME Graduate outcomes in the form of achievement of predefined desired competencies are the goal. Competencies are derived from the needs of patients, organized into a coherent guiding framework. Time is a resource for learning, not the basis of progression of competence. Teaching and learning experiences are sequenced to facilitate an explicitly defined progression of ability in stages.

Fundamental Characteristics of CBME Learning is tailored to the learner's individual progression in some manner. Numerous direct observations and focused feedback contribute to effective learner development of expertise. Assessment is planned, systematic, systemic, and integrative.

General Competencies: US Patient care and procedural skills Medical knowledge Professionalism Interpersonal skills and communication Practice-based learning and improvement (PBLI) Systems-based practice (SBP)

General Competency: PBLI Residents must demonstrate the ability to investigate and evaluate their care of patients, to appraise and assimilate scientific evidence, and to continuously improve patient care based on constant self-evaluation and life-long learning. Examples: Evidence-based medicine Quality improvement in clinical practice

General Competency: SBP Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care, as well as the ability to call effectively on other resources in the system to provide optimal health care. Examples: Interprofessional teamwork Care coordination such as discharge and consultation Patient safety

Competencies help to define the educational outcomes (abilities) Competency frameworks are just that – organizational frameworks to guide curriculum and assessment Competencies help to define the educational outcomes (abilities)

Moving from the general to the specific

Milestones: general to specialty-specific General Competencies Patient Care Medical Knowledge Professionalism Interpersonal Skills & Communication PBL & I Systems-based Practice SPECIALTY TRANSLATION Specialty Specific Milestones

Milestones - What Are They? By definition a milestone is simply a significant point in development. Milestones should enable the residents, fellows and the training program to better know an individual’s trajectory of competency development.

Developmental Progression or Set of Milestones Competency Developmental Progression or Set of Milestones Sub-competency Specific Milestone PC1. History (Appropriate for age and impairment) Level 1 Level 2 Level 3 Level 4 Level 5 Acquires a general medical history Acquires a basic physiatric history including medical, functional, and psychosocial elements Acquires a comprehensive physiatric history integrating medical, functional, and psychosocial elements   Seeks and obtains data from secondary sources when needed Efficiently acquires and presents a relevant history in a prioritized and hypothesis driven fashion across a wide spectrum of ages and impairments Elicits subtleties and information that may not be readily volunteered by the patient Gathers and synthesizes information in a highly efficient manner Rapidly focuses on presenting problem, and elicits key information in a prioritized fashion Models the gathering of subtle and difficult information from the patient This slide describes the anatomy the Milestone system. The General Competency is based in one of the six core competencies, and PC stands for Patient Care in this instance. Each Competency is further broken down into Subcompetencies, which comprise specific skills necessary to achieve the Competency as a whole. A Milestone is actually a description of how a resident is doing, and this slide contains 9 of them. The entire set of milestones for one Subcompetency is referred to as a Set of Milestones. The levels do not refer to specific PGY levels of training; rather, they simply represent progression with a Level 3 Milestone being more advanced than a Level 2 Milestone. Residents will be expected to progress at different rates, and some will spend more time at one level than another level. Level 5 is aspirational – this is the expected performance level of someone who has been in practice a few years.

Milestone: Level Descriptions

Learning Curves Residency From Pusic, et. al. Acad Med. 2014 Another way to think about the Milestones is a learning curve (related to Dreyfus and Ericsson). This is a nice graphic from an article from Martin Pusic (NYU) of what a hypothetical learning curve likely looks like in residency – most residents will experience the curve highlighted in red. From Pusic, et. al. Acad Med. 2014

Professional Development: Dreyfus Model MILESTONES Curriculum Assessment Curriculum Assessment Curriculum Assessment Curriculum Assessment Curriculum Assessment Curriculum Assessment Expert/ Master Proficient Competent Development is a non-linear phenomenon Advanced Beginner Reality is skills develop over time and training. We expect different levels of competence for different skills. The Milestones were heavily influenced by developmental models of expertise, such as the work of the Dreyfus brothers and Anders Ericsson. This slide is based on the Dreyfus brothers. We also make the point that curriculum and assessment must be integrated: assessment drives learning but the “right learning” (curriculum) should also guide the appropriate choice of assessment. Novice Time, Practice, Experience Dreyfus SE and Dreyfus HL. 1980 Carraccio CL et al. Acad Med 2008;83:761-7

Rebalancing assessment

Assessing for the Desired Outcome Performance in Practice//Multi-source feedback/ Direct Observation Work-based assessment is mostly accomplished through the observations and questions of faculty, team members, peers and other co-workers Does (action) Standardized Patients/Simulation Shows How (performance) Diagnostic Reasoning using clinical vignettes or CSR Knows How (competence) Knows (knowledge) Multiple choice Questions

Cambridge Model: “flipping” the Pyramid Performance System related influences Individual related influences Competence If the goal is performance then need to workplace assessment. When you get to top this is where you need to do all of the meaninfgul assessment. Competence can do with MCQ but doesn’t translate to performance- we have robusts MCQ, but we don’t have WBA assessments And just ebcause know does mean you do Competency based assessments were defined as what doctors do in testing situations- performance based assessment defined as measure of what doctors do in practice. Cambridge flips millers- competence is first three levels of miller- greater focus on performance Interaction of person and system Work-based assessment has to be the primary focus of our assessment systems Rethans, Norcini, et al, 2002

Common work-based Assessment Methods Performance reviews using quality measures Patient surveys Chart stimulated recall Direct observation Video reviews Portfolios Multisource feedback Procedural logs Faculty evaluations So here is your 20 minute reivew of common assessment methods Records of clinical encounters: logbooks I think the question isn’t really what is availabe to us, but rather

The Professional assessment “System” Unit of Analysis: Program Residents Feedback Assessments within Program: Direct observations Audit and performance data Multi-source FB Simulation ITExam Accreditation J U D G M E N T Qual/Quant “Data” Synthesis: Committee P U B L I C Feedback Certification and Credentialing Feedback Faculty, PDs and others Unit of Analysis: Individual Milestones and EPAs as Guiding Framework and Blueprint This is a Human Process

U.S. competency journey - milestones Dates Event/Change 1994 ACGME begins work on developing competencies 1999 Six general competency framework approved 2001 Launch of Outcomes Project 2007 First Milestone summit – Internal Medicine 2009 First Milestones published 2010-13 Milestone sets created for all specialties 2013 First 7 specialties start using and reporting Milestones 2014 All specialties fully in system

Thank You