Applying the Performance Framework Data Model June 10, 2013

Performance Framework Metadata: lom, EffectiveDate, RetiredDate, Replaces, IsReplacedBy, SupportingInformation PerformanceScale: id, LeastCompetentScore, MostCompetentScore Component id Title Abbreviation Competency (reference to a competency object) Author Reviewer Background Resources References 1+ ComponentReference: id of a nested Component 0+ PerformanceLevelSet 0 or 1 or PerformanceScale (reference to a LevelScale id) 1 PerformanceLevel 2+ DisplayOrder ScoreValue Label Background Note Indicator: id, Description, Background, Note 1+

Internal Medicine Example Title: The Internal Medicine Milestone Project Identifier: – Catalogue: URI – Entry: Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… Contributions: – Role: Author – Entity: William Iobst, M.D. – Role: Author – Entity: Eva Aagaard, M.D. Effective date:

Internal Medicine Continued Supporting Information – This document presents milestones designed for programs to use … Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

Internal Medicine Component ID: Competency: Title: Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s).

Performance level set Performance Scale Reference: 1 to 5 Position: 1Score Value: 1Label: Critical Deficiencies Indicator (id = i1_1): Does not collect accurate historical data Indicator (id = i1_2): Does not use physical exam to confirm history Indicator (id = i1_3): Relies exclusively on documentation of others to generate own database or differential diagnosis Indicator (id = i1_4): Fails to recognize patient’s central clinical problems Indicator (id = i1_5): Fails to recognize potentially life threatening problems This would allow indicator to be tracked as on/off.

Position: 2Score Value: 2Label: Indicator (id = i2_1): Inconsistently able to acquire accurate historical information in an organized fashion Indicator (id = i2_2): Does not perform an appropriately thorough physical exam or misses key physical exam findings Indicator (id = i2_3): Does not seek or is overly reliant on secondary data Indicator (id = i2_4): Inconsistently recognizes patients’ central clinical problem or develops limited differential diagnoses Position: 3Score Value: 3Label: Indicator (id = i3_1): Consistently acquires accurate and relevant histories from patients Indicator (id = i3_2): Seeks and obtains data from secondary sources when needed Indicator (id = i3_3): Consistently performs accurate and appropriately thorough physical exams Indicator (id = i3_4): Uses collected data to define a patient’s central clinical problem(s)

Position: 4Score Value: 4Label: Ready for unsupervised practice Indicator (id = i4_1): Acquires accurate histories from patients in an efficient, prioritized, and hypothesis-driven fashion Indicator (id = i4_2): Performs accurate physical exams that are targeted to the patient’s complaints Indicator (id = i4_3): Synthesizes data to generate a prioritized differential diagnosis and problem list Indicator (id = i4_4): Effectively uses history and physical examination skills to minimize the need for further diagnostic testing Position: 5Score Value:5Label: Aspirational Indicator (id = i5_1): Obtains relevant historical subtleties, including sensitive information that informs the differential diagnosis; Indicator (id = i5_2): Identifies subtle or unusual physical exam findings Indicator (id = i5_3): Efficiently utilizes all sources of secondary data to inform differential diagnosis Indicator (id = i5_4): Role models and teaches the effective use of history and physical examination skills to minimize the need for further diagnostic testing

Internal Medicine Component Competency: ID: Title: Patient Care Background: The resident is demonstrating satisfactory development of the knowledge, skill…

Performance levels Performance Scale Reference: 1 to 3 Position: 1Score Value: 3Label: Yes Indicator (id = pc1): The resident is demonstrating satisfactory development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care. Position: 2Score Value: 1Label: No Indicator (id = pc2): The resident is demonstrating unsatisfactory development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is not demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care.

Performance levels Position: 3Score Value: 2Label: Marginal Indicator (id = pc3): The resident is demonstrating marginal development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is marginally demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care.

Pediatrics Example Title: The Pediatrics Milestone Project Identifier: – Catalog: URI – Entry: nas.org/assets/PediatricsMilestones Contributions: – Role: Author – Entity: Carol Carraccio, M.D. – Role: Reviewer – Entity: Richard Antonelli, MD, MS Effective date:

Pediatrics Continued Supporting Information – rces/320_PedsMilestonesProject.pdf Performance Scale – 1 to 4 (1 least competent, 4 most competent) – 1 to 5 (1 least competent, 5 most competent) – 1 to 6 (1 least competent, 6 most competent)

Pediatrics Component ID: Title: Gather essential and accurate information about the patient Competency: Author: Daniel Schumacher, MD Background: Early Development of Information-Gathering Skills In the early stages of clinical reasoning, learners must rely upon their knowledge of basic pathophysiology and …

Pediatrics Component Continued References 1.Schmidt HG, Norman GR, Boshuizen HPA. A cognitive perspective on medical expertise: theory and implications. Academic Medicine. 1990;65: Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: translating the Dreyfus Developmental Model to the learning of clinical skills. Academic Medicine. 2008;83: Eva K. What every teacher needs to know about clinical reasoning. Medical Education. 2004;39: Schmidt HG, Boshuizen HPA. On acquiring expertise in medicine. Educational Psychology Review. 1993;5: Schmidt HG, Rikers RMJP. How expertise develops in medicine: knowledge encapsulation and illness script formation. Medical Education. 2007;41: Charlin B, Boshuizen HPA, Custers EJ, Feltovich PJ. Scripts and clinical reasoning. Medical Education. 2007;41: Patel VL, Groen GJ, Patel YC. Cognitive aspects of clinical performance during patient workup: the role of medical expertise.” Advances in Health Sciences Education. 1997;2: Elstein AS, Kagan N, Shulman LS, et al. Methods and theory in the study of medical inquiry. Journal of Medical Education. 1972;47:85-92.

Performance level set Performance Scale: 1 to 5 Position: 1Score Value: 1Label: Indicator (id = pc_i1): Either gathers too little information or exhaustively gathers information following a template regardless of the patient’s chief complaint, with each piece of information gathered seeming as important as the next. Recalls clinical information in the order elicited,7 with the ability to gather, filter, prioritize, and connect pieces of information being limited by and dependent upon analytic reasoning through basic pathophysiology alone. Position: 2Score Value: 2Label: Indicator (id = pc_i2): Clinical experience allows linkage of signs and symptoms of a current patient to those encountered in previous patients. Still relies primarily on analytic reasoning through basic pathophysiology to gather information, but the ability to link current findings to prior clinical encounters allows information to be filtered, prioritized, and synthesized into pertinent positives and negatives as well as broad diagnostic categories. Free text reference No formatting

Position: 3Score Value: 3Label: Indicator (id = pc_i3): Advanced development of pattern recognition leads to the creation of illness scripts, which allow information to be gathered while it is simultaneously filtered, prioritized, and synthesized into specific diagnostic considerations. Data gathering is driven by real-time development of a differential diagnosis early in the information-gathering process.8 Position: 4Score Value: 4Label: Indicator (id = pc_i4): Well-developed illness scripts allow essential and accurate information to be gathered and precise diagnoses to be reached with ease and efficiency when presented with most pediatric problems, but still relies on analytic reasoning through basic pathophysiology to gather information when presented with complex or uncommon problems. Position: 5Score Value: 5Label: Indicator (id = pc_i5): Robust illness scripts and instance scripts (where the specific features of individual patients are remembered and used in future clinical reasoning) lead to unconscious gathering of essential and accurate information in a targeted and efficient manner when presented with all but the most complex or rare clinical problems. These illness and instance scripts are robust enough to enable discrimination among diagnoses with subtle distinguishing features.

Pediatrics Component ID: Title: Prescribe and perform all medical procedures Competency: Author: Patricia Hicks, MD

Background All of the competencies are involved in prescribing and performing medical procedures. In an integrated …

Background Continued The component KSA of each procedure are numerous and complex. They include: Anatomy and Physiology Indications and Benefits Contra-indications and Risks Informed Consent Pain Management, Patient Psychological Preparation Specimen Handling Interpretation of Results or Outcomes Procedural Technique (multiple elements unique to procedure; common elements to all [e.g., sterile technique, situational awareness, course correction]) Post-procedure Management This approach to assessment makes some assumptions: Performance level is specific to each procedure based on the relevant components and level of responsibility of the physician. Given the variability of required components, measures of competence are based on all of the relevant components for that procedure. Performance level for a given procedure, therefore, requires reaching the desired performance level for each of the individual components.

References 1. Wigton R, Nicolas J, Blank L. Procedural skills of the general internist: a survey of 2500 physicians. Annals of Internal Medicine. 1989;111: Wigton R, Blank L, Nicolas J, Tape T. Procedural skills training in internal medicine residencies. Annals of Internal Medicine. 1989;111: Wigton R. Training internists in procedural skills. Annals of Internal Medicine. 1992;116: Hicks C, Gonzales R, Morton M, et al. Procedural experience and comfort level in internal medicine trainees. General Internal Medicine. 2000;15: Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler Publishers; Kirkpatrick L, Kirkpatrick JD. The four levels: an overview. In: Kirkpatrick DL, ed. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler Publishers; 2006: Davis D. Accuracy of physician self-assessment compared with observed measures of competence. Journal of the American Medical Association. 2006:296: Carbine D, Finer N, Knodel E, Rich W. Video recording as a means of evaluating neonatal resuscitation performance. Pediatrics. 2000;106: Adams K, Scott R, Perkin R, Langga L. Comparison of intubation skills between interfacility transport team members. Pediatric Emergency Care. 2000;16: Falck A, Escobedo M, Baillargeon J, et al. Proficiency of pediatric residents in performing neonatal endotracheal intubation. Pediatrics. 2003;112: Colliver J, Vu N, Barrows H. Screening test length for sequential testing with a standardized-patient examination: a receiver operating characteristic (ROC) analysis. Academic Medicine. 1992;67: Jones D, McGuinness G. The future for pediatric residency education: the prescription for more flexibility. Journal of Pediatrics. 2009;154: American Board of Internal Medicine. Internal Medicine Policies. Accessed December 12, Kovacs G, Bullock G, Ackroyd-Stolarz S, et al. A randomized controlled trial on the effect of educational interventions in promoting airway management skill maintenance. Annals of Emergency Medicine. 2000;36: Wickstrom GC, Kelley DK, Keyserling TC, et al. Confidence of academic generalist internists and family physicians to teach ambulatory procedures. Journal of General Internal Medicine. 2000;15: Beauchamp TL, Childress JF. Principles of Biomedical Ethics. 5th ed. New York, NY: Oxford University Press; Sarker S, Chang A, Albrani T, Vincent C. Constructing hierarchical task analysis in surgery. Surgical Endoscopy. 2008;22: Shepherd A. HTA as a framework for task analysis. Ergonomics. 1998;41:

Nested Component ID: Competency: _physiology.xml Title: Anatomy and Physiology

Performance levels Performance Scale Reference: 1 to 4 Position: 1Score Value: 1Label: Beginning of Spectrum Indicator (id = pcap_i1): 2 SD below mean on knowledge test Position: 2Score Value: 2Label: Indicator (id = pcap_i2): 1 SD below mean on knowledge test Position: 3Score Value: 3Label: Indicator (id = pcap_i3): 1 SD above mean on knowledge test Position: 4Score Value: 4Label: Indicator (id = pcap_i4): 2 SD above mean on knowledge test

Pediatrics Component from ID: Title: Identify strengths, deficiencies, and limits in one’s knowledge and expertise Competency:

Performance Levels Performance Scale Reference: 1 to 4 Position: 1Score Value: 1Label: Level 1 Indicator (id = pbli_i1): The learner acknowledges external assessments, but understanding of his performance is superficial and limited to the overall grade or bottom line; has little understanding of how the performance measure relates in a meaningful way to his specific level of Knowledge, Skills and Attitudes (KSA) Background: Description: Example: During a semiannual review, a learner is unable to describe in any specific terms how he has performed when asked to do so by his mentor. In response, the mentor reviews and interprets the learner’s evaluations and then asks the learner to reflect on the discussion. The learner repeats the language used and recites the overall score/grade without interpretation of further meaning or inference regarding the reported performance assessment.

Nursing (DNP Eval) Example Title: University of San Diego Hahn School of Nursing and Health Science DNPC 630 Residency DNP NP Student Evaluation Identifier: – Catalog: URI – Entry: Performance Scale – 1 to 3 (1 least competent, 3 most competent)

DNP Component Competency: ID: 5678 Title: Prepared to practice independently managing previously diagnosed and undiagnosed patients.

Performance Levels Performance Scale Reference: 1 to 3 Position: 1Score Value: 3Label: Indicator (id = pi_i1): Met Position: 2Score Value: 2Label: Indicator (id = pi_i2): In progress Position: 3Score Value: 1Label: Indicator (id = pi_i3): Not Met

National University of Singapore Example Title: Standards of Achievement Identifier: – Catalog: URI – Entry: Performance Scale – 1 to 5 (1 least competent, 5 most competent)

NUS Component ID: 2013_1 Title: Standards of Achievement

Performance Levels Performance Scale Reference: 1 to 5 Position: 1Score Value: 1Label: Indicator (id = i1): Unable to achieve outcome. Position: 2Score Value: 2Label: Indicator (id = i2): Requires a lot of guidance to achieve outcome. Position: 3Score Value: 3Label: Indicator (id = i3): Requires moderate amount of guidance to achieve outcome.

Performance Levels Position: 4Score Value: 4Label: Indicator (id = i4): Able to achieve outcome with little or no guidance (entrustment) Position: 5Score Value: 5Label: Indicator (id = i5): Has the ability to guide/teach others.

Diagnostic Radiology Example (see Title: The Diagnostic Radiology Milestone Project Identifier: – Catalogue: URI – Entry: nas.org/assets/DiagnosticRadiologyMilestones Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… Contributions: – Role: Author – Entity: Kay Vydareny, MD – Role: Author – Entity: E. Stephen Amis Jr., MD Effective date:

Diagnostic Radiology Continued Supporting Information – This document presents milestones designed for programs to use… Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

Diagnostic Radiology Component ID: Competency: Title: Professional Values and Ethics Background: Possible Methods of Assessment/Examples: – End-of-Rotation Global Assessment – 360 Evaluation/Multi-rater/Peer – Simulation/OSCE – Direct observation and feedback – Conference attendance logs – Timeliness in completing institutional and program requirements Is it appropriate to put this in background?

Diagnostic Radiology Component continued Resources – Resource Citation: Teaching and Assessing Professionalism: A Program Director’s Guide by the ABP and APPD. See Chapter 8: Measuring Professionalism, Critical incidents, Peer assessments, Multi-source assessments, Professionalism Mini-Evaluation Exercise (P-MEX) – Resouce: Citation: The Professionalism Mini-Evaluation Exercise: A Preliminary Investigation. Richard Cruess, Jodi Herold McIlroy, Sylvia Cruess, Shiphra Ginsburg, and Yvonne Steinert Acad Med Oct;81(10 Suppl):S74-8 These are suggested educational tools.

Diagnostic Radiology Component continued Resources (continued) – Resource Citation: ABRF Online Modules on Ethics and Professionalism Dcterms: references:

Performance level set Performance Scale Reference: 1 to 5 Position: 1Score Value: 1Label: Level 1 Indicator (id = p1_1): recognizes the importance and priority of patient care and advocates for patient interests Indicator (id = p1_2): fulfills work-related responsibilities Indicator (id = p1_3): is truthful Indicator (id = p1_4): recognizes personal limitations and seeks help when appropriate Indicator (id = i1_5): recognizes personal impairment and seeks help when needed Etc…

Position: 2Score Value: 2Label: Level 2 Indicator (id = p2_1): Is an effective health care team member Indicator (id = p2_2): Demonstrates professional behaviors listed in the second column Position: 3Score Value: 3Label: Level 3 Indicator (id = p3_1): Is an effective health care team leader, promoting primacy of patient welfare, patient autonomy, and social justice Indicator (id = p3_2): Demonstrates professional behaviors listed in the second column Position: 4Score Value: 4Label: Level 4 Indicator (id = p4_1): Serves as a role model for professional behavior Indicator (id = p4_2): Demonstrates professional behaviors listed in the second column

Position: 5Score Value: 5Label: Level 5 Indicator (id = p5_1): Participates in local and national organizations to advance professionalism in radiology Indicator (id = p5_2): Mentors others regarding professionalism and ethics

Psychiatry Example (see ) Title: The Psychiatry Milestone Project Identifier: – Catalogue: URI – Entry: Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… Contributions: – Role: Author – Entity: Sheldon Benjamin, M.D. – Role: Author – Entity: Adrienne L. Bentman, M.D Effective date:

Psychiatry Continued Supporting Information – This document presents milestones designed for programs to use… Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

Psychiatry Component ID: Competency: Title: Psychotherapy Refers to 1) the practice and delivery of psychotherapies, including but not limited to psychodynamic (1), cognitive-behavioral (2), and supportive therapies (3); 2) exposure to couples, family, and group therapies; and 3) integrating psychotherapy with psychopharmacology Note (position = 1): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, and to understand the concepts of resistance/defenses, transference/countertransference We don’t have a Note field in at the Component level currently, but based on this example, maybe we should add it. Or accommodate in some way.

Psychiatry Component (continued) Note (position = 2): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, including behavior change, skills acquisition, and addressing cognitive distortions Note (position = 3): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, and to strengthen the patient’s adaptive defenses, resilience, and social supports

Performance level set Performance Scale Reference: 1 to 5 Position: 1Score Value: 1Label: Level 1 Indicator (id = pc4_1_1): 1.1 Accurately identifies patient emotions, particularly sadness, anger, and fear (4) Note: This thread, consisting of the first item at each level, regarding the development of empathy across residency is adapted from the AADPRT Psychotherapy Workgroup’s document “Benchmarks for Psychotherapy Training.” Indicator (id = pc4_1_2): 1.2 Demonstrates interest and curiosity in patient’s story

Position: 2Score Value: 12Label: Level 2 Indicator (id = pc4_2_1): 1.1 Accurately identifies patient emotions, particularly sadness, anger, and fear (4) Indicator (id = pc4_2_2): 1.2 Demonstrates interest and curiosity in patient’s story Etc.

How are use cases addressed? Map performance frameworks to competencies, including Entrustable Professional Activities – YES (Competency Reference) Publish a performance framework for use in curriculum planning and assessment – YES (Requires IDs for Components so that assessments may be tied to performance levels) Import descriptions of performance levels for use in assessment – YES Reference a Performance Framework for use in assessment – YES (Requires IDs for Components so that assessments may be tied to performance levels) Describe an individual’s current level of performance for purposes of formative or summative assessment – YES (a score and a reference to a Component should do the trick) Describe an individual’s level of performance over a longitudinal period for purposes of formative or summative assessment – YES (provided the framework is authored to support that) Associate assessment evidence with a particular level of performance in a portfolio – Do we want to associate it with a level of performance, or do we want assessment evidence to have a score that is interpreted using a particular group of performance levels defined for the relevant competency?

How are use cases addressed? Define where in a curriculum students are expected to achieve certain levels of performance – YES (will require updates to curriculum inventory, may require id on performance level) Define what level of performance is required to progress to the next phase or block within a curriculum – YES (will require updates to curriculum inventory, may require id on performance level) Use in a system that is capable of showing a learner changes in performance over time – YES View data regarding the performance levels of learners in a program for purposes of program evaluation (external) – YES (would require a spec that allows for the exchange of aggregate data) View data regarding the performance levels of learners in a program for purposes of program evaluation (internal) – YES Publish program data for public viewing – YES (would require a spec that allows for the exchange of aggregate data)