1. Faculty: Craig Campbell Date: April 18, 2017
The Newly Trained Physician: The Vision for sustaining and enhancing competence throughout practice
Coalition for Physician Enhancement 2017
Faculty: Craig Campbell
Date: April 18, 2017

2. CONFLICT OF INTEREST DECLARATION
Craig Campbell
I am a full-time Director with the Royal College of Physicians and Surgeons of Canada
I have no relationships with any commercial interests to disclose

3. At the end of this session participants will be able to:
Learning Objectives
At the end of this session participants will be able to:
Describe the rationale for transitioning to competency-based residency training and CPD.
Describe at least 2 strategies that will sustain competence and enhance performance throughout practice.
Identify one new idea for those working in the field of remediation post-licensure.
Ken: This is the type of dialogue we anticipated you may wish to have with Council. This may also be a good plce to share with Council; the reasons for “the pause”.
One of our fundamental responsibilities as physicians is to continuously strive to improve the way we deliver care and improve the quality of care we deliver
CBD offers us an opportunity to improve the to improve the health and healthcare of Canadians by transforming specialty education from a time-based to a (hybrid) competency-based education system.
This shift which will span the learning continuum from residency and into retirement will help ensure that physicians are consistently prepared to continuously provide quality patient care in a rapidly changing healthcare environment.
Describe rationale for the pause to implementation and plans for working groups.

4. Transition from Residency into Practice
We need a model for the 21st century
We need to respond to calls for change in graduate competencies
We need to ensure graduates have all the essential competencies
We need to move to a true continuum & progression of competence
We need to move away from time-based credentials

5. Transitions into Practice.
Some key questions
Are our training programs preparing residents to enter practice with the competencies required to:
Be effective lifelong learners?
Meet patient and population health needs?
What defines the curriculum for physicians once they are in practice?
What role should assessment and feedback play in guiding learning and continuous improvement?
How can we connect physicians to each other?
Gisele
As an obstetrician, this particular quote speaks to me… it expresses well the feeling that “the buck now stops with me”.

6. Andrea’s Journey: Current Residency Model
Certification
Exam Preparation/Electives
Senior Residency Phase
Junior Residency Phase
Basic Clinical year
Exams
Multiple
ITERs
Assessments
Andrea is a senior resident begin preparations for her final certification exam

7. Criticisms of 20th Century Contemporary Training…

8. Diagnosis of Modern Medical Education: The “Tea Bag Model”
Steep residents in a clinical context for long enough – competence will result
Q. Is there a better way to ensure competence than just time spent?
In my view, a fundamental threat to the future of medical education is this criticism, called the “tea bag model”: it argues that medicine tends to credential physicians based on the time spent in training, not based on their achievement of necessary abilities.
Our medical education systems of the 20th century are no longer “good enough”. We need to do better.

9. Feedback from clinical teachers…
Time pressured
Supervision / Ad hoc teaching
Feedback can be painful or risky
Long generic assessment forms are ~useless
Concerns about learner progression & expectations
Concerns about graduates

10. Little Time for Observation
Holmboe

11. Variation in faculty expectations
I’m surprised you can’t do that at this stage in your training

12. For Trainees…

13. For Trainees…

14. Rater Leniency
-Turnbull, 2002

15. “Rater Range Restriction”
Global, generic
rating scales
“Pleasant”
“Read More”

16. Difficulties with Reports

17. Failure to Fail

18. Diagnosis of Modern Medical Education:
“Ad-Hoc Education” Limited control over patient experiences, faculty teaching We struggle to ensure our graduates are competent in all needed domains…
Put another way, “How can we ensure that our graduates are competent in all needed domains…?”

19. Contemporary Challenges to our System
Residency Education
Uses ‘time spent’ as a metric, not ability.
Multiple challenges to teach and evaluate many content areas – eHealth, patient safety, biomedical ethics, professionalism, quality improvement….
Infrequent observations of trainees in the work-place with limited opportunities for feedback to improve.
Assessments of competence and performance – dichotomous judgments
Excessive reliance on high stakes exams
Trainees feel unprepared for practice.

21. The rationale for competency-based medical education

22. Competency-based Medical Education (CBME)
Definition:
“an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using competencies as the organizing framework”.
CBME therefore is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies
© 2009 Royal College and The International CBME Collaborators
Medical Teacher 2010

23. WHO, 1978

24. CBME: A Global Movement
Netherlands
USA
Singapore
Australia
Canada
Middle East

25. Competence = Time? X

26. Competence = Achievement

27. The Era of Competency-based Medical Education
Principles
Focus on abilities: Competences serve as the foundation for curriculum design
Outcomes-focused: Graduates are required to demonstrate competence in all essential domains, using multiple assessments with feedback
Learner-centered: clear understanding of curricular goals and a transparent path to track their progress towards achievement
Responsive to societal health needs: graduates enter practice with the ability to meet the needs of patients.
Responsive to societal health needs

28. Competency-based Medical Education
“What are the abilities each resident must demonstrate?”
at each stage of training?
across each CanMEDS role?
basis to enter practice!
Once in practice…
“How do physicians progress
in competence to attain
expertise?”
CBME implies that outcomes—the defined abilities needed of graduates—are the central organizing principle. Teaching methods, even some traditional knowledge areas, are only included if they contribute to the ultimate outcome of the program.

29. Royal College’s Competence by Design Program

30. CanMEDS

31. Stages of physician development
Key concepts
New CanMEDS
Milestones for each
enabling competency
for each Role across
each Stage
Stages of physician development
New Assessment
Organized on EPAs
New Accreditation

32. Competence Continuum Key messages:
After several years of work, the Royal College is officially launching the CanMEDS 2015 Framework on Saturday morning.
Remember though, the CanMEDS project is part of the broader CBD initiative …. We’re building toward an improved system that facilitates all of the improvements that Dr. Harris outlined earlier
Goal of the CanMEDS 2015 project specifically was to update the content for contemporary medicine and create milestones to facilitate CBME
Over the course of 2.5 years we engaged MANY partners and consulted widely
We released the Framework (and Milestones) in serial fashion over the course of 2014 to gather feedback and to adjust the content as appropriate
Our methodology was robust ….. we can all stand behind CanMEDS confidently knowing that it is valid
CanMEDS and the milestones is foundational to the rest of the plans associated with CBD. CBD is on track!
You can be confident that the Royal College’s model for CanMEDS (consultation, engagement, partnership) is the model we’re taking for CBD

33. Milestone/ EPA Definitions
Milestones:
The abilities expected of a
health professional at a
stage of development.
EPAs
The key tasks of a discipline
that a physician must be
able to perform.

34. Examples of EPAs
In the real world
Teenager on an errand
In medicine
Run a code

35. The O-Score for Entrustment
I had to do (learner watched)
I walked them through (hands on supervision; learner contributes)
I prompted (learner performed but was guided)
I had to be there just in case (limited need for supervision)
I did not need to be there (entrusted to perform independently)

36. Milestones within an EPA
Typically, each EPA integrates multiple milestones.

37. Milestones and EPAs within Four Stages of Residency
So, within each stage there were be several EPAs which are composed of the component milestones

38. Assessment of Milestones
A “palette” of options Direct Observation Simulation MSF Chart audits among others…

39. New Progression of Competence

40. ePortfolio: Learner Dashboard

41. ePortfolio: Observer Dashboard

42. Initial Cohorts
Initial Launch: July 2017
Anesthesia
Otolaryngology and Head and Neck Surgery
Another 8 – 12 disciplines are in the loop to launch in July 2018
Key Supports….
Faculty Development strategies
CanMEDS interactive
Evaluation model – short, medium, long term

43. What are the implications of CBME for practice?

44. Will this be enough?

45. Contemporary Challenges to CPD
CPD focuses on knowledge dissemination – Medical Expert
Impact between participation in group learning and performance or behavior change is small or limited
Outcome measures focus on participation in CPD and self-reported outcomes for change.
Persisting gaps in quality of health care / safety of health care systems.
Physicians have limited access to data or feedback on their performance / health outcomes.
Limited focus on interprofessional collaborative practice to enhance patient outcomes. among others….

46. New View of Lifelong Learning Across the Continuum
Conceptual framework for performance assessment. Khan and Ramachandran, Medical Teacher 2012; 34:

47. CB – CPD must be designed to allow
Vision for CB-CPD
CB – CPD must be designed to allow
physicians to answer questions such as…
Has my competence or my
performance improved?
Is the care I provide patients reflect best evidence?
Are my patients better off?
Is where I work safe for patients?
Educational
Outcomes (
Patient and
Health System
Outcomes

48. View lifelong learning as a foundational competence /skill set required of all physicians in practice

49. ENABLING COMPETENCIES
CanMEDS 2015 – Scholar Role
ENABLING COMPETENCIES
Physicians are able to:
1.1 Develop, implement, monitor, and revise a personal learning plan to enhance professional practice
1.2 Identify opportunities for learning and improvement by regularly reflecting on and assessing their performance using various internal and external data sources
1.3 Engage in collaborative learning to continuously improve personal practice and contribute to collective improvements in practice

50. Royal College Lifelong Learning Curriculum: the skills of effective lifelong learners
The ability to:
Know your practice:
create a learning plan based on one’s scope of practice in response to identified patient, population health needs
Create a personal knowledge management system
Skills of information literacy
Store and retrieve evidence, guidelines, tools, practice aids
Effectively scan one’s environment for new developments
New research that is applicable to your practice
Approaches being discarded as no longer effective or safe
Raise and answer questions stimulated by practice
Inquiry based learning; reflection; self-monitoring
Assess one’s performance and health outcomes

51. Physician Practice Improvement Cycle
1. Understand your practice
2. Assess your practice
3. Create your learning plan
4. Implement your learning plan
5. Evaluate the outcomes

52. Curriculum in CPD: defined by the patient, population health needs of a physician’s practice not discipline….

53. Scope of Practice Some practical advantages
Guides reflection on and responds to practice specific patient and population health needs

54. Scope of Practice: Some potential advantages
Scope of practice: Anchors learning in the workplace
Practice Context
Learning Context
Performance
Patient Safety
Quality of Care
Formal &
informal learning
and
Assessment
Clinics ER
Wards
ICU
Ajout animation

55. Health care Institutions
Competency-based CPD
Health care Institutions
Physician
Data with Feedback
Healthcare
Teams
Patient &
Population
Needs
CPD
Providers
Evidence &
Resources
Competencies
Strategies
& Tools
Regulators

56. Scope of Practice: Some potential concerns
Could a focus on ‘scope of practice’:
Undermine sustaining foundational skills or competences expected of all physicians?
Result in an excessively individualistic approach to practice - leading to further ‘sub-specialization silos’?
Fail to address the ‘needs’ of patients or populations?
Compete with employer expectations?

57. One View of ‘Curriculum’
Competences
specific to
evolving
nature of
practice
Competences expected of all physicians

58. Provide physicians with regular access to data on their performance and health outcomes with FEEDBACK

59. My Hypothesis
Assessment is an educational imperative for lifelong learning in practice
Provides data to:
Individuals, groups or teams regarding their practice or performance (to identify unperceived needs)
Potential to guide:
Learning and change when feedback is provided
Development of action plans to improve

60. Multiple Assessment Options
Practice (clinical) audits
Patient registries
Patient-reported experience/outcome measures
Multi-source feedback
Simulation
Low to high fidelity; on-line; in situ
Direct observation of performance
Peer review of practice
Critical incident de-briefing
Academic Detailing …among others

61. Role of Feedback in Assessment
“The role of feedback in improving the effectiveness
of workplace based assessments: a systematic review Saedon H
BMC Med Edu 2012:12:25
15 studies: 7 MSF; 3 mini-CEX; 2 procedural based assessments; 1 work based in general and 2 studies looked at a combination of 3 to 6 assessments
Conclusion: good evidence that if well implemented, feedback from workplace based assessments, particularly MSF, leads to a perceived positive effect on practice

62. Some key elements of feedback
Importance of feedback
Impact of workplace based assessment on doctor’s education and performance: a systematic review
Miller, BMJ 2012
“Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses”

63. Feedback to improve self-reflection
presentation
6/3/2018
Feedback to improve self-reflection
We may not accurately perceive where we are in relation to where we want to be…..
-improved self-reflection (sources of feedback need to be credible)
-Defined as an effective skill set to identify what is working and developing strategies to change what is not working
Self (with or without dialogue with another objective individual)
Self with the team (group)
Group process
-use of portfolio to support learning
-striving for ‘excellence’ not just ‘bare’ competence
Boud – Reflection: Turning experience into learning
Conscious and deliberate process for continuing improvement/development…novice to master

64. Task performance versus Self-concept
presentation
6/3/2018
Task performance versus Self-concept
Sargeant – MSF to physicians
Recognition that the feedback is personal info about oneself – difficult to teat objectively,
Hard to considerthe info as a task performance rather than a reflection on identity / reputation / self esteem
Critical/constructive can in that light create self doubt, anger and frustration……not motiviating to learn and use for improvement…defend…look for reasons
Especially when use comparators to others – encourages the focus on self
On the performance (task) rather than on self

65. Informed Self-Assessment
Definition
“A set of processes through which individuals use external and internal data to generate an appraisal of their own ability”.
Mann K, Sargeant J.
Acad Med 2011

66. Dimensions of informed self-assessment
External & Internal Conditions
Sources of Information
Interpreting Information
Responses to Information
Tensions

67. Evidence – informed facilitated feedback model
presentation
6/3/2018
Guided Reflection
Evidence – informed facilitated feedback model
Evidence - based facilitated feedback model:
 Sargeant J, Armson H, Driessen E, Holmboe E, Mann K, Lockyer J, Silver I
4 stages : (R2C2) 1. Rapport and relationship building 2. Exploring reactions 3. Exploring content Coaching for change
Impact of workplace based assessment on doctor’s education and performance: a systematic review - Miller, BMJ 2012
“Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses”
Abstract references:
Sargeant J, Armson H, Driessen E, Holmboe E, Mann K, Lockyer J, Silver I. Facilitating Performance Feedback to Promote Physician Practice Improvement: Implications for Undergraduate (UGME) and Postgraduate Medical Education (PGME). (workshop)CCME Annual Meeting Quebec, PQ. April 21, 2013.
Sargeant J. Facilitating Physician Performance Feedback and Practice Change: Implementations for Quality Improvement. SACME Annual Spring Meeting 2013 Madison, Wisconsin, USA. April 11, (invited plenary)
Sargeant J, Armson H, Driessen E, Holmboe E, Mann K, Lockyer J, Silver I. Evidence-Based Facilitated Feedback: a Model to Enhance Feedback Acceptance and Use. (workshop). AMEE 2013 Annual Meeting. Prague, Czech Republic. August 24-28, 2013.
Sargeant J, Armson H, Driessen E, Holmboe E, Mann K, Lockyer J, Silver I

68. Informed Self-assessment
Overarching theme:
No one assessment activity or approach was generally effective in informing self-assessment
Effectiveness was moderated by external factors, internal perceptions
Conclusion:
The value of a specific activity was not in the activity itself, BUT how it was used!

69. Learning new things: requires a plan to acquire and demonstrate competence

70. The Practicing Physician
Specific Context/Competency
Expert/
Master
Zone of Performance
Proficient
New Context/Skill
Zone of Performance
Competent
Advanced Beginner
Not a linear process
Novice
Time, Practice, Experience
Dreyfus SE and Dreyfus HL. A 1980
Carraccio CL et al. Acad Med 2008;83:761-7

71. CB-CPD: Description of Outcomes
Level 1: Participation in learning activities
Level 2: Learned new knowledge and skills
Level 3: Knows how to apply knowledge and skills
Level 4: Demonstrates competence
Level 5: Improves performance in practice
Level 6: Improves health outcomes for patients / populations
Levels 1- 3 are important but insufficient
for CB-CPD to be ‘outcomes-based’

72. Learning in practice must focus on both individual and collective competence

73. An intentional focused on
Competency-based CPD
Focus on the competencies physicians require to function effectively as members of teams
“work effectively with physicians and other colleagues in the health care professions” (Collaborator)
An intentional focused on
Importance of team work in achieving important health outcomes
Patient experience of care provided by teams

74. Competency-based CPD
Focus on the competencies physicians require to contribute to continuous improvement of our health systems
An intentional integration of CPD with CQI in the workplace to focus on
Improving the process of health care
Patient safety
Health care costs

75. Are these the health outcomes competency-based CPD can address?

76. Some final thoughts on implications

77. Competency-based Remediation?
Remediation strategies should:
Use competences as a framework to assess physician’s competence, performance and / or health outcomes
Use of milestones / EPAs for practice
Competencies provide an explanation for why performance is below standard
Expand opportunities to assess
Skills of lifelong learning
How physicians identify needs relevant to their practice context / scope of practice
Connectivity of physicians to a learning environment

78. Competency-based Remediation?
Remediation strategies should:
Train and support personnel with the skills to provide effective feedback that turns assessment into a plan for learning and improvement
Create a more continuous integrated model of assessment ‘for learning’ throughout practice
Not about dichotomizing assessment into pass/fail but providing opportunities to improve performance
Implications for coaching models within remediation strategies or the physician’s health system

79. Competency-based Remediation?
Remediation strategies should:
Collaborate with the physician’s health system to facilitate support structures / coaching in the workplace
Greater emphasis on situated and longitudinal observations in the workplace and learning institutions
Acquisition of new skills and abilities
Demonstration of abilities over time
Translation of assessment experiences into a personal inquiry based learning strategy
Frame a plan for improvement.

80. Competency-based Remediation?
Remediation strategies should:
Provide opportunities for learners to ‘understand’ why their behavior might vary from ‘typical practice’.
Why does variation exist?
Strategies to make assessment authentic to the context of practice
Contribute to a culture of continuous improvement.
Shift towards a shared responsibility between learner and system
Harnesses feedback in ways that emphasize the primacy of learning

81. Competency-based CPD
Need to achieve an appropriate balance between Autonomy: The physician’s ability to choose Accountability: Social contract with public

82. Competency-based CPD
The right focus: Designed to address patient/ population and personal needs. The right approach: CQI of practice The right outcomes: Enhancing competence, performance; quality, safety of care The right evidence: Data drives improvement Leverages the commitment and motivation of the profession

83. Need for Culture Change
A System of Continuous Learning
and Practice Improvement….
Focus is on enhancing competence, performance, patient outcomes
Context is one’s scope of practice
Content is based on best evidence
Learning processes intentionally integrates:
Formal learning
Self-planned learning
Assessment