Faculty: Craig Campbell Date: April 18, 2017 The Newly Trained Physician: The Vision for sustaining and enhancing competence throughout practice Coalition for Physician Enhancement 2017 Faculty: Craig Campbell Date: April 18, 2017

CONFLICT OF INTEREST DECLARATION Craig Campbell I am a full-time Director with the Royal College of Physicians and Surgeons of Canada I have no relationships with any commercial interests to disclose

At the end of this session participants will be able to: Learning Objectives At the end of this session participants will be able to: Describe the rationale for transitioning to competency-based residency training and CPD. Describe at least 2 strategies that will sustain competence and enhance performance throughout practice. Identify one new idea for those working in the field of remediation post-licensure. Ken: This is the type of dialogue we anticipated you may wish to have with Council. This may also be a good plce to share with Council; the reasons for “the pause”. One of our fundamental responsibilities as physicians is to continuously strive to improve the way we deliver care and improve the quality of care we deliver CBD offers us an opportunity to improve the to improve the health and healthcare of Canadians by transforming specialty education from a time-based to a (hybrid) competency-based education system. This shift which will span the learning continuum from residency and into retirement will help ensure that physicians are consistently prepared to continuously provide quality patient care in a rapidly changing healthcare environment. Describe rationale for the pause to implementation and plans for working groups.

Transition from Residency into Practice We need a model for the 21st century We need to respond to calls for change in graduate competencies We need to ensure graduates have all the essential competencies We need to move to a true continuum & progression of competence We need to move away from time-based credentials

Transitions into Practice. Some key questions Are our training programs preparing residents to enter practice with the competencies required to: Be effective lifelong learners? Meet patient and population health needs? What defines the curriculum for physicians once they are in practice? What role should assessment and feedback play in guiding learning and continuous improvement? How can we connect physicians to each other? Gisele As an obstetrician, this particular quote speaks to me… it expresses well the feeling that “the buck now stops with me”.

Andrea’s Journey: Current Residency Model Certification Exam Preparation/Electives Senior Residency Phase Junior Residency Phase Basic Clinical year Exams Multiple ITERs Assessments Andrea is a senior resident begin preparations for her final certification exam

Criticisms of 20th Century Contemporary Training…

Diagnosis of Modern Medical Education: The “Tea Bag Model” Steep residents in a clinical context for long enough – competence will result Q. Is there a better way to ensure competence than just time spent? In my view, a fundamental threat to the future of medical education is this criticism, called the “tea bag model”: it argues that medicine tends to credential physicians based on the time spent in training, not based on their achievement of necessary abilities. Our medical education systems of the 20th century are no longer “good enough”. We need to do better.

Feedback from clinical teachers… Time pressured Supervision / Ad hoc teaching Feedback can be painful or risky Long generic assessment forms are ~useless Concerns about learner progression & expectations Concerns about graduates

Little Time for Observation Holmboe

Variation in faculty expectations I’m surprised you can’t do that at this stage in your training

For Trainees…

For Trainees…

Rater Leniency -Turnbull, 2002

“Rater Range Restriction” Global, generic rating scales “Pleasant” “Read More”

Difficulties with Reports

Failure to Fail

Diagnosis of Modern Medical Education: “Ad-Hoc Education” Limited control over patient experiences, faculty teaching We struggle to ensure our graduates are competent in all needed domains… Put another way, “How can we ensure that our graduates are competent in all needed domains…?”

Contemporary Challenges to our System Residency Education Uses ‘time spent’ as a metric, not ability. Multiple challenges to teach and evaluate many content areas – eHealth, patient safety, biomedical ethics, professionalism, quality improvement…. Infrequent observations of trainees in the work-place with limited opportunities for feedback to improve. Assessments of competence and performance – dichotomous judgments Excessive reliance on high stakes exams Trainees feel unprepared for practice.

The rationale for competency-based medical education

Competency-based Medical Education (CBME) Definition: “an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using competencies as the organizing framework”. CBME therefore is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies © 2009 Royal College and The International CBME Collaborators Medical Teacher 2010

WHO, 1978

CBME: A Global Movement Netherlands USA Singapore Australia Canada Middle East

Competence = Time? X

Competence = Achievement

The Era of Competency-based Medical Education Principles Focus on abilities: Competences serve as the foundation for curriculum design Outcomes-focused: Graduates are required to demonstrate competence in all essential domains, using multiple assessments with feedback Learner-centered: clear understanding of curricular goals and a transparent path to track their progress towards achievement Responsive to societal health needs: graduates enter practice with the ability to meet the needs of patients. Responsive to societal health needs

Competency-based Medical Education “What are the abilities each resident must demonstrate?” at each stage of training? across each CanMEDS role? basis to enter practice! Once in practice… “How do physicians progress in competence to attain expertise?” CBME implies that outcomes—the defined abilities needed of graduates—are the central organizing principle. Teaching methods, even some traditional knowledge areas, are only included if they contribute to the ultimate outcome of the program.

Royal College’s Competence by Design Program

CanMEDS

Stages of physician development Key concepts New CanMEDS Milestones for each enabling competency for each Role across each Stage Stages of physician development New Assessment Organized on EPAs New Accreditation

Competence Continuum Key messages: After several years of work, the Royal College is officially launching the CanMEDS 2015 Framework on Saturday morning. Remember though, the CanMEDS project is part of the broader CBD initiative …. We’re building toward an improved system that facilitates all of the improvements that Dr. Harris outlined earlier Goal of the CanMEDS 2015 project specifically was to update the content for contemporary medicine and create milestones to facilitate CBME Over the course of 2.5 years we engaged MANY partners and consulted widely We released the Framework (and Milestones) in serial fashion over the course of 2014 to gather feedback and to adjust the content as appropriate Our methodology was robust ….. we can all stand behind CanMEDS confidently knowing that it is valid CanMEDS and the milestones is foundational to the rest of the plans associated with CBD. CBD is on track! You can be confident that the Royal College’s model for CanMEDS (consultation, engagement, partnership) is the model we’re taking for CBD

Milestone/ EPA Definitions Milestones: The abilities expected of a health professional at a stage of development. EPAs The key tasks of a discipline that a physician must be able to perform.

Examples of EPAs In the real world Teenager on an errand In medicine Run a code

The O-Score for Entrustment I had to do (learner watched) I walked them through (hands on supervision; learner contributes) I prompted (learner performed but was guided) I had to be there just in case (limited need for supervision) I did not need to be there (entrusted to perform independently)

Milestones within an EPA Typically, each EPA integrates multiple milestones.

Milestones and EPAs within Four Stages of Residency So, within each stage there were be several EPAs which are composed of the component milestones

Assessment of Milestones A “palette” of options Direct Observation Simulation MSF Chart audits among others…

New Progression of Competence

ePortfolio: Learner Dashboard

ePortfolio: Observer Dashboard

Initial Cohorts Initial Launch: July 2017 Anesthesia Otolaryngology and Head and Neck Surgery Another 8 – 12 disciplines are in the loop to launch in July 2018 Key Supports…. Faculty Development strategies CanMEDS interactive Evaluation model – short, medium, long term

What are the implications of CBME for practice?

Will this be enough?

Contemporary Challenges to CPD CPD focuses on knowledge dissemination – Medical Expert Impact between participation in group learning and performance or behavior change is small or limited Outcome measures focus on participation in CPD and self-reported outcomes for change. Persisting gaps in quality of health care / safety of health care systems. Physicians have limited access to data or feedback on their performance / health outcomes. Limited focus on interprofessional collaborative practice to enhance patient outcomes. among others….

New View of Lifelong Learning Across the Continuum Conceptual framework for performance assessment. Khan and Ramachandran, Medical Teacher 2012; 34: 920-928

CB – CPD must be designed to allow Vision for CB-CPD CB – CPD must be designed to allow physicians to answer questions such as… Has my competence or my performance improved? Is the care I provide patients reflect best evidence? Are my patients better off? Is where I work safe for patients? Educational Outcomes (http://www.merriam-webster.com/dictionary/outcome) Patient and Health System Outcomes

View lifelong learning as a foundational competence /skill set required of all physicians in practice

ENABLING COMPETENCIES CanMEDS 2015 – Scholar Role ENABLING COMPETENCIES Physicians are able to: 1.1 Develop, implement, monitor, and revise a personal learning plan to enhance professional practice 1.2 Identify opportunities for learning and improvement by regularly reflecting on and assessing their performance using various internal and external data sources 1.3 Engage in collaborative learning to continuously improve personal practice and contribute to collective improvements in practice

Royal College Lifelong Learning Curriculum: the skills of effective lifelong learners The ability to: Know your practice: create a learning plan based on one’s scope of practice in response to identified patient, population health needs Create a personal knowledge management system Skills of information literacy Store and retrieve evidence, guidelines, tools, practice aids Effectively scan one’s environment for new developments New research that is applicable to your practice Approaches being discarded as no longer effective or safe Raise and answer questions stimulated by practice Inquiry based learning; reflection; self-monitoring Assess one’s performance and health outcomes

Physician Practice Improvement Cycle 1. Understand your practice 2. Assess your practice 3. Create your learning plan 4. Implement your learning plan 5. Evaluate the outcomes

Curriculum in CPD: defined by the patient, population health needs of a physician’s practice not discipline….

Scope of Practice Some practical advantages Guides reflection on and responds to practice specific patient and population health needs

Scope of Practice: Some potential advantages Scope of practice: Anchors learning in the workplace Practice Context Learning Context Performance Patient Safety Quality of Care Formal & informal learning and Assessment Clinics ER Wards ICU Ajout animation

Health care Institutions Competency-based CPD Health care Institutions Physician Data with Feedback Healthcare Teams Patient & Population Needs CPD Providers Evidence & Resources Competencies Strategies & Tools Regulators

Scope of Practice: Some potential concerns Could a focus on ‘scope of practice’: Undermine sustaining foundational skills or competences expected of all physicians? Result in an excessively individualistic approach to practice - leading to further ‘sub-specialization silos’? Fail to address the ‘needs’ of patients or populations? Compete with employer expectations?

One View of ‘Curriculum’ Competences specific to evolving nature of practice Competences expected of all physicians

Provide physicians with regular access to data on their performance and health outcomes with FEEDBACK

My Hypothesis Assessment is an educational imperative for lifelong learning in practice Provides data to: Individuals, groups or teams regarding their practice or performance (to identify unperceived needs) Potential to guide: Learning and change when feedback is provided Development of action plans to improve

Multiple Assessment Options Practice (clinical) audits Patient registries Patient-reported experience/outcome measures Multi-source feedback Simulation Low to high fidelity; on-line; in situ Direct observation of performance Peer review of practice Critical incident de-briefing Academic Detailing …among others

Role of Feedback in Assessment “The role of feedback in improving the effectiveness of workplace based assessments: a systematic review Saedon H BMC Med Edu 2012:12:25 15 studies: 7 MSF; 3 mini-CEX; 2 procedural based assessments; 1 work based in general and 2 studies looked at a combination of 3 to 6 assessments Conclusion: good evidence that if well implemented, feedback from workplace based assessments, particularly MSF, leads to a perceived positive effect on practice

Some key elements of feedback Importance of feedback Impact of workplace based assessment on doctor’s education and performance: a systematic review Miller, BMJ 2012 “Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses”

Feedback to improve self-reflection presentation 6/3/2018 Feedback to improve self-reflection We may not accurately perceive where we are in relation to where we want to be….. -improved self-reflection (sources of feedback need to be credible) -Defined as an effective skill set to identify what is working and developing strategies to change what is not working Self (with or without dialogue with another objective individual) Self with the team (group) Group process -use of portfolio to support learning -striving for ‘excellence’ not just ‘bare’ competence Boud – Reflection: Turning experience into learning Conscious and deliberate process for continuing improvement/development…novice to master

Task performance versus Self-concept presentation 6/3/2018 Task performance versus Self-concept Sargeant – 2008 MSF to physicians Recognition that the feedback is personal info about oneself – difficult to teat objectively, Hard to considerthe info as a task performance rather than a reflection on identity / reputation / self esteem Critical/constructive can in that light create self doubt, anger and frustration……not motiviating to learn and use for improvement…defend…look for reasons Especially when use comparators to others – encourages the focus on self On the performance (task) rather than on self

Informed Self-Assessment Definition “A set of processes through which individuals use external and internal data to generate an appraisal of their own ability”. Mann K, Sargeant J. Acad Med 2011

Dimensions of informed self-assessment External & Internal Conditions Sources of Information Interpreting Information Responses to Information Tensions

Evidence – informed facilitated feedback model presentation 6/3/2018 Guided Reflection Evidence – informed facilitated feedback model Evidence - based facilitated feedback model:  Sargeant J, Armson H, Driessen E, Holmboe E, Mann K, Lockyer J, Silver I. 2013 4 stages : (R2C2) 1. Rapport and relationship building 2. Exploring reactions 3. Exploring content 4. Coaching for change Impact of workplace based assessment on doctor’s education and performance: a systematic review - Miller, BMJ 2012 “Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses”   Abstract references: Sargeant J, Armson H, Driessen E, Holmboe E, Mann K, Lockyer J, Silver I. Facilitating Performance Feedback to Promote Physician Practice Improvement: Implications for Undergraduate (UGME) and Postgraduate Medical Education (PGME). (workshop)CCME Annual Meeting 2013. Quebec, PQ. April 21, 2013. Sargeant J. Facilitating Physician Performance Feedback and Practice Change: Implementations for Quality Improvement. SACME Annual Spring Meeting 2013 Madison, Wisconsin, USA. April 11, 2013. (invited plenary) Sargeant J, Armson H, Driessen E, Holmboe E, Mann K, Lockyer J, Silver I. Evidence-Based Facilitated Feedback: a Model to Enhance Feedback Acceptance and Use. (workshop). AMEE 2013 Annual Meeting. Prague, Czech Republic. August 24-28, 2013. Sargeant J, Armson H, Driessen E, Holmboe E, Mann K, Lockyer J, Silver I. 2013

Informed Self-assessment Overarching theme: No one assessment activity or approach was generally effective in informing self-assessment Effectiveness was moderated by external factors, internal perceptions Conclusion: The value of a specific activity was not in the activity itself, BUT how it was used!

Learning new things: requires a plan to acquire and demonstrate competence

The Practicing Physician Specific Context/Competency Expert/ Master Zone of Performance Proficient New Context/Skill Zone of Performance Competent Advanced Beginner Not a linear process Novice Time, Practice, Experience Dreyfus SE and Dreyfus HL. A 1980 Carraccio CL et al. Acad Med 2008;83:761-7

CB-CPD: Description of Outcomes Level 1: Participation in learning activities Level 2: Learned new knowledge and skills Level 3: Knows how to apply knowledge and skills Level 4: Demonstrates competence Level 5: Improves performance in practice Level 6: Improves health outcomes for patients / populations Levels 1- 3 are important but insufficient for CB-CPD to be ‘outcomes-based’

Learning in practice must focus on both individual and collective competence

An intentional focused on Competency-based CPD Focus on the competencies physicians require to function effectively as members of teams “work effectively with physicians and other colleagues in the health care professions” (Collaborator) An intentional focused on Importance of team work in achieving important health outcomes Patient experience of care provided by teams

Competency-based CPD Focus on the competencies physicians require to contribute to continuous improvement of our health systems An intentional integration of CPD with CQI in the workplace to focus on Improving the process of health care Patient safety Health care costs

Are these the health outcomes competency-based CPD can address?

Some final thoughts on implications

Competency-based Remediation? Remediation strategies should: Use competences as a framework to assess physician’s competence, performance and / or health outcomes Use of milestones / EPAs for practice Competencies provide an explanation for why performance is below standard Expand opportunities to assess Skills of lifelong learning How physicians identify needs relevant to their practice context / scope of practice Connectivity of physicians to a learning environment

Competency-based Remediation? Remediation strategies should: Train and support personnel with the skills to provide effective feedback that turns assessment into a plan for learning and improvement Create a more continuous integrated model of assessment ‘for learning’ throughout practice Not about dichotomizing assessment into pass/fail but providing opportunities to improve performance Implications for coaching models within remediation strategies or the physician’s health system

Competency-based Remediation? Remediation strategies should: Collaborate with the physician’s health system to facilitate support structures / coaching in the workplace Greater emphasis on situated and longitudinal observations in the workplace and learning institutions Acquisition of new skills and abilities Demonstration of abilities over time Translation of assessment experiences into a personal inquiry based learning strategy Frame a plan for improvement.

Competency-based Remediation? Remediation strategies should: Provide opportunities for learners to ‘understand’ why their behavior might vary from ‘typical practice’. Why does variation exist? Strategies to make assessment authentic to the context of practice Contribute to a culture of continuous improvement. Shift towards a shared responsibility between learner and system Harnesses feedback in ways that emphasize the primacy of learning

Competency-based CPD Need to achieve an appropriate balance between Autonomy: The physician’s ability to choose Accountability: Social contract with public

Competency-based CPD The right focus: Designed to address patient/ population and personal needs. The right approach: CQI of practice The right outcomes: Enhancing competence, performance; quality, safety of care The right evidence: Data drives improvement Leverages the commitment and motivation of the profession

Need for Culture Change A System of Continuous Learning and Practice Improvement…. Focus is on enhancing competence, performance, patient outcomes Context is one’s scope of practice Content is based on best evidence Learning processes intentionally integrates: Formal learning Self-planned learning Assessment