® Development of Technology Standards for the Performance Framework: Progress Report by the Competencies Working Group Rosalyn P. Scott, M.D., M.S.H.A. Wright State University Department of Veterans Affairs Tim Willett, M.D. Sim-ONE Valerie Smothers MedBiquitous

® Competencies Working Group Co-Chairs Rosalyn Scott, Department of Veteran's Affairs Tim Willett, Sim-ONE Members Susan Albright, Tufts University Mary Pat Aust, American Association of Critical-Care Nurses Dana Bostrom, Association of American Medical Colleges Terri Cameron, Association of American Medical Colleges Mary Jo Clark, American Association of Colleges of Nursing Stephen Clyman, National Board of Medical Examiners Matthew Cownie, University of the West of England Robert Englander, Association of American Medical Colleges Ted Hanss, University of Michigan Sean Hilton, St. George's University of London Joshua Jacobs, National University of Singapore PJ Kania, Association of American Medical Colleges David Kiger, Tufts University JB McGee, University of Pittsburgh David Price, American Board of Family Medicine Isarin Sathitruangsak, Tufts University Lesley Southgate, St. George's University of London Nabil Zary, Karolinska Institute Invited Experts Connie Bowe, Partners HealthCare International Kelly Caverzagie, University of Nebraska Medical Center Allan Cumming, University of Edinburgh Rachel Ellaway, Northern Ontario School of Medicine Simon Grant, JISC-CETIS Linda Gwinn, Georgetown University Robyn Herring, University of Mississippi Kevin Krane, Tulane University Steven Lieberman, University of Texas Medical Branch Linda Lewin, University of Maryland Melanie Loving, University of Texas Medical Branch Karen Macauley, University of San Diego Hahn School of Nursing and Health Science Chandler Mayfield, University of California, San Francisco Daniel Rehak Juliane Schneider, Harvard University

® ANSI/MEDBIQ CF , Competency Framework Learner and educators can search for learning resources addressing a particular competency. Educators can determine where specific competencies are addressed in a curriculum. Boards and hospitals can track and manage competency data for the professional. Administrators can map one competency framework to another.

® Competency Objects and Frameworks The use of outcome and competency frameworks is a growing part of health professions education and regulation in many countries. To implement competency-based education and assessment, competency frameworks must be connected to curricula, learning resources, and assessment data.

® Unfinished Business…. Assessing Competence Performance Levels How do detailed competencies fit together to do the work we do on a daily basis? What does competence look like in the real world? We all know it when we see it!!

® Performance Framework Requirements and XML data specifications for the exchange of performance levels and performance frameworks Initial focus on undergraduate and graduate medical education North American and International paradigms for medical education should inform the structure of the specification Other healthcare professionals critical to development

® Competency Framework.xml Professionalism.xml A competency object... Relates to other competency objects... xml Which can relate to resources, other data, assessment... Learning Object Educational Activity Educational Achievement Data Current Specifications Performance Framework

® Next Accreditation System: ACGME Competencies and Outcomes Evaluate programs based on reporting of outcomes through educational milestones Patient care Medical Knowledge Professionalism Systems-based practice Practice-based learning and improvement Interpersonal and communication skills Foster innovation and improvement by programs

® What ACGME Plans…… Continuous oversight not episodic sampling Milestone reports (semi-annual), Resident & Faculty questionnaires (annually), Case Logs (annually), Program & Institutional Infrastructure (annually) Program Site Visits at ~10 year intervals Emphasis on program effectiveness as evidenced by resident advancement through the milestones

® Assessing Competence Milestones: Granular Developmental framework from less to more advanced Drill down to smaller elements of behavior that provide the substance of formative feedback, remediation, and a learning roadmap Entrustable Professional Activities: Holistic Integrate competencies within a clinical context Assess clusters of behaviors required to care for patients

® ACGME Milestones: A Developmental Roadmap Milestones are knowledge, skills, attitudes, and other attributes for each of the ACGME competencies organized in a developmental framework. They are descriptors and targets for resident performance as a resident moves from entry into residency through graduation.

® Orthopedic Surgery Milestone

® Framing ACGME Competencies in Entrustable Professional Activities Competencies are no longer abstractions Reflect complex integration of competencies needed in actual day-to-day practice Address authentic practice Entrustable designates that “a practitioner has demonstrated the necessary knowledge, skills and attitudes to be trusted to independently perform this activity.”

® Level of Mastery of an EPA Five stages of decreasing supervision: (1)no task execution (2)task execution under direct supervision on site (3)task execution with supervision quickly available on call (4)unsupervised practice (5)supervision may be provided to juniors

® How it all works together…….

® How are use cases addressed? Map performance frameworks to competencies YES (CompetencyReference) Publish a performance framework for use in curriculum planning and assessment YES (Requires IDs for Components and groups of PerformanceLevels so that assessments may be tied to performance levels) Describe an individual’s current performance for purposes of formative or summative assessment YES (a score and a reference to a group of PerformanceLevels should do the trick)

® How are use cases addressed? View data regarding the performance levels in a program for both external and internal evaluation YES (would require a spec that allows for the exchange of aggregate data) Publish program data for public viewing YES (would require a spec that allows for the exchange of aggregate data Define level of performance required to progress to the next phase or block within a curriculum YES (will require updates to curriculum inventory)

® How do you know a trainee is qualified and what supervision is required? Trainee Professional Profile Is competent in Competency Framework At what level Performance Framework Is entrusted to do Competency Framework