1. Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012

2. Disclosures  Employed by the American Board of Internal Medicine  I receive royalties from Mosby-Elsevier for a textbook on assessment  I am a member of the board of NBME and Medbiquitous  I serve on committees at the AAMC, ABMS, ACGME and NBME

3. The Quality of Care Problem

4. Teaching Vs. Non-Teaching Hospital Quality Performance Ind.COTH Teaching Non-COTH Teaching Non-Teaching 30-day Mortality AMI15.1%15.9%16.3% Pneumonia10.8%11.1%11.7% 30-day Readmission AMI20.3%19.7%19.6% HCAHPS Nurse communicated well 70.5%70.9%74.9% MD communicated well 76.2%77.0%81% Help when wanted55.0%57.0%64.1% Shahian DM, Nordberg P, Meyer GS, et al. Contemporary performance of U.S. teaching and nonteaching hospitals. Acad Med. 2012; 87: online.

5. Care of the Vulnerable Elderly Study Performance on Geriatric Process of Care Resident Clinics Mean % Practicing Physicians Mean % Univariate F Structure coefficients Documentation of: Gait evaluation28.4%74.2%77.53**.90 Balance evaluation21.6%66.4%65.51**.82 Medical surrogate28.0%54.4%24.00**.65 End-of-life preferences29.5%49.3%12.85**.55 Vision testing done40.0%64.7%19.09**.55 Hearing assessment23.3%40.3%8.06*.41 Screens for: Falls risk18.6%60.8%49.60**.67 Cognitive impairment18.3%52.0%29.02**.60 Depression33.7%62.6%24.09**.57 Lynn LA, et al. Acad Med. 2009.

6. Hospital Comparisons on Quality and Resource Use (Higher scores represent better performance) Non-teaching (N= 997)Teaching (N=186) Quality Composite Score Resource use Composite Score Exemplary Teaching Hospitals Source: L. Binder, CEO of Leapfrog Group, email communication, March 2010

7. “Every system is perfectly designed to achieve the results it gets.” Paul Batalden

8. Medical Education: Restraining Forces on Change

9. The Current “Miracle” of Medical Education Dwell Time

10. Medical Education Architecture 1 1 Holmboe E, Ginsburg S, Bernabeo E. The rotational approach to medical education: time to confront our assumptions. Med Educ. 2011; 45(1):69-80.

11. Thomas Kuhn and “Normal Science”  “Normal science, the activity in which most scientists inevitably spend almost all of their time, is predicated on the assumption that the scientific community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost” Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.

12. Could the Same be True of UME and GME?  “Normal education, the activity in which most educators inevitably spend almost all of their time, is predicated on the assumption that the educational community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost” Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.

13. The “Big Assumption as Truth”  We operate on many assumptions that over time become “truth” without our testing or questioning the veracity of those assumptions –Test assumptions as assumptions  Immunity to change –Preservation of status quo through fear –More comfortable to stay with familiar even when status quo isn’t effective Kegan and Lahey. The Way We Talk Can Change the Way We Work; Immunity to Change.

14. Competency-based Medical Education: A Way Forward?

15. Effective Systems: Where Education Must Occur Nelson EC, et al. Quality by Design. 2007

16. Early Principles: CBME  World Health Organization (1978): –“The intended output of a competency- based programme is a health professional who can practise medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.” McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based Curriculum Development in Medical Education. World Health Organization, Switzerland, 1978.

17. Traditional versus CBME: Start with System Needs 17 Frenk J. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010

18. Competency-Based Medical Education …is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies 1 1 Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical education: theory to practice. Med Teach. 2010; 32: 638–645

19. Outcomes-based Education: General Principles  Patient outcomes ≈ Educational outcomes  Experience ≠ Expertise –Exposure and dwell time are not sufficient proxies for competence You can do something a 100 times wrong and develop experience, but it’s still wrong!  Must engage in effective experiences Critical role for work-based assessments

20. Need for System Approach: Assessment Perspective

21. Structured Portfolio  ITE (formative only)  Monthly Evaluations  MiniCEX  Medical record audit/QI project  Clinical question log  Multisource feedback  Trainee contributions (personal portfolio) o Research project Trainee  Review portfolio  Reflect on contents  Contribute to portfolio Program Leaders  Review portfolio periodically and systematically  Develop early warning system  Encourage reflection and self-assessment Clinical Competency Committee  Periodic review – professional growth opportunities for all  Early warning systems Program Summative Assessment Process Licensing and Certification  USMLE  American Boards of Medical Specialties Assessment During Training: Components Advisor

22. Structured Portfolio Medical record audit and QI project MSF: Directed per protocol Twice/year Practice-based learning and improvement Systems-based prac Mini-CEX: 10/year Interpersonal skills and Communication ITE: 1/year Patient care Faculty Evaluations EBM/ Question Log Medical knowledge Professionalism Multi-faceted Evaluation ■ Trainee-directed ■ Direct observation

23. Time Assessment Activities Training Activities Supporting Activities vvvvvv Intermediate Eval Final Evaluation = learning task = learning artifact = single assessment data-point = single certification data point for mastery tasks = learner reflection and planning = social interaction around reflection (supervision) = learning task being an assessment task also Model For Programmatic Assessment (With permission from CPM van der Vleuten) Committee

24. Structured Portfolio  ITE (formative only)  Monthly Evaluations  MiniCEX  Medical record audit/QI project  Clinical question log  Multisource feedback  Trainee contributions (personal portfolio) o Research project Trainee  Review portfolio  Reflect on contents  Contribute to portfolio Program Leaders  Review portfolio periodically and systematically  Develop early warning system  Encourage reflection and self-assessment Clinical Competency Committee  Periodic review – professional growth opportunities for all  Early warning systems Program Summative Assessment Process Licensing and Certification  USMLE  American Boards of Medical Specialties Assessment During Training: Components Advisor

25. “Wisdom of the Crowd” Williams, Teach. Learn. Med. (2005) –No evidence that individuals in groups dominate discussions. No evidence of ganging up/piling on Thomas (2011) – Group assessment improved inter-rater reliability and reduced range restriction in multiple domains in an internal medicine residency

26. Narratives and Judgments Pangaro (1999) – matching students to a “synthetic” descriptive framework (RIME) reliable and valid across multiple clerkships Regehr (2007) – Matching students to a standardized set of holistic, realistic vignettes improved discrimination of student performance Regehr (2012) – Faculty created narrative “profiles” (16 in all) found to produce consistent rankings of excellent, competent and problematic performance.

27. Strategy to establish trustworthinessCriteria Potential Assessment Strategy (sample) CredibilityProlonged engagementTraining of examiners TriangulationTailored volume of expert judgment based on certainty of information Peer examinationBenchmarking examiners Member checkingIncorporate learner view Structural coherenceScrutiny of committee inconsistencies TransferabilityTime samplingJudgment based on broad sample of data points Thick descriptionJustify decisions DependabilityStepwise replicationUse multiple assessors who have credibility ConfirmabilityAuditGive learners the possibility to appeal to the assessment decision *With permission CPM van der Vleuten

28. The Road Forward: Kelly Caverzagie