1. Applying the Performance Framework Data Model August 15, 2013

2. Performance Framework Metadata: lom, EffectiveDate, RetiredDate, Replaces, IsReplacedBy, SupportingInformation PerformanceScale: id, LeastCompetent, MostCompetent Component id Title Abbreviation Competency (reference to a competency object) Author Reviewer AdditionalInformation 1+ ComponentReference: id of a nested Component 0+ PerformanceLevelSet 0 or 1 or PerformanceScaleReference (reference to a LevelScale id) 1 PerformanceLevel 2+ DisplayOrderLabelAdditionalInformation Indicator: id, Description, AdditionalInformation 1+ Score: SingleValue OR Range (MinScore, MaxScore) 1

3. Internal Medicine Example Title: The Internal Medicine Milestone Project Identifier: – Catalogue: URI – Entry: http://www.acgme-nas.org/assets/InternalMedicineMilestones Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… Contributions: – Role: Author – Entity: William Iobst, M.D. – Role: Author – Entity: Eva Aagaard, M.D. Effective date: 2013-01-30

4. Internal Medicine Continued Supporting Information – This document presents milestones designed for programs to use … Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

5. Internal Medicine Component ID: 12345 Title: Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s). Competency: Description: http://www.example.org/im_milestones/PC1.xml Type: http://ns.medbiq.org/competencyobject/v1/http://www.example.org/im_milestones/PC1.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/im_framework.xml Type: http://ns.medbiq.org/competencyframework/v1/http://www.example.org/im_framework.xmlhttp://ns.medbiq.org/competencyframework/v1 Note: likely to change

6. Internal Medicine Component (continued) Threshold – Title: Entrustment – Description: The threshold for the entrustment decision with regard to this competency – MinimumAcceptableScore: 4

7. Performance level set Performance Scale Reference: 1 to 5 Position: 1Score: Single Value: 1 Label: Critical Deficiencies Indicator (id = i1_1): Does not collect accurate historical data Indicator (id = i1_2): Does not use physical exam to confirm history Indicator (id = i1_3): Relies exclusively on documentation of others to generate own database or differential diagnosis Indicator (id = i1_4): Fails to recognize patient’s central clinical problems Indicator (id = i1_5): Fails to recognize potentially life threatening problems This would allow indicator to be tracked as on/off.

8. Position: 2Score: Single Value: 2 Label: Indicator (id = i2_1): Inconsistently able to acquire accurate historical information in an organized fashion Indicator (id = i2_2): Does not perform an appropriately thorough physical exam or misses key physical exam findings Indicator (id = i2_3): Does not seek or is overly reliant on secondary data Indicator (id = i2_4): Inconsistently recognizes patients’ central clinical problem or develops limited differential diagnoses Position: 3Score Single Value: 3 Label: Indicator (id = i3_1): Consistently acquires accurate and relevant histories from patients Indicator (id = i3_2): Seeks and obtains data from secondary sources when needed Indicator (id = i3_3): Consistently performs accurate and appropriately thorough physical exams Indicator (id = i3_4): Uses collected data to define a patient’s central clinical problem(s)

9. Position: 4Score: Single Value: 4 Label: Ready for unsupervised practice Indicator (id = i4_1): Acquires accurate histories from patients in an efficient, prioritized, and hypothesis-driven fashion Indicator (id = i4_2): Performs accurate physical exams that are targeted to the patient’s complaints Indicator (id = i4_3): Synthesizes data to generate a prioritized differential diagnosis and problem list Indicator (id = i4_4): Effectively uses history and physical examination skills to minimize the need for further diagnostic testing Position: 5Score Single Value:5 Label: Aspirational Indicator (id = i5_1): Obtains relevant historical subtleties, including sensitive information that informs the differential diagnosis; Indicator (id = i5_2): Identifies subtle or unusual physical exam findings Indicator (id = i5_3): Efficiently utilizes all sources of secondary data to inform differential diagnosis Indicator (id = i5_4): Role models and teaches the effective use of history and physical examination skills to minimize the need for further diagnostic testing

10. Internal Medicine Component Competency: Description: http://www.example.org/im_milestones/PC.xml Type: http://ns.medbiq.org/competencyobject/v1/http://www.example.org/im_milestones/PC.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/im_framework.xml Type: http://ns.medbiq.org/competencyframework/v1/http://www.example.org/im_framework.xmlhttp://ns.medbiq.org/competencyframework/v1 ID: 23456 Title: Patient Care Additional Information (label = background): The resident is demonstrating satisfactory development of the knowledge, skill…

11. Performance levels Performance Scale Reference: 1 to 3 Position: 1Score: Single Value: 3 Label: Yes Indicator (id = pc1): The resident is demonstrating satisfactory development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care. Position: 2Score: Single Value: 1 Label: No Indicator (id = pc2): The resident is demonstrating unsatisfactory development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is not demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care.

12. Performance levels Position: 3Score: Single Value: 2 Label: Marginal Indicator (id = pc3): The resident is demonstrating marginal development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is marginally demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care.

13. Pediatrics Example Title: The Pediatrics Milestone Project Identifier: – Catalog: URI – Entry: http://www.acgme- nas.org/assets/PediatricsMilestones Contributions: – Role: Author – Entity: Carol Carraccio, M.D. – Role: Reviewer – Entity: Richard Antonelli, MD, MS Effective date: 2013-01-30

14. Pediatrics Continued Supporting Information – http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramResou rces/320_PedsMilestonesProject.pdf Performance Scale – 1 to 4 (1 least competent, 4 most competent) – 1 to 5 (1 least competent, 5 most competent) – 1 to 6 (1 least competent, 6 most competent)

15. Pediatrics Component ID: 12345 Title: Gather essential and accurate information about the patient Competency: Description: http://www.example.org/peds_milestones/PC1.xml Type: http://ns.medbiq.org/competencyobject/v1/http://www.example.org/peds_milestones/PC1.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency Description: http://www.example.org/peds_milestones.xml Type: http://ns.medbiq.org/competencyframework/v1/http://www.example.org/peds_milestones.xmlhttp://ns.medbiq.org/competencyframework/v1 Author: Daniel Schumacher, MD Additional Information (label = background): Early Development of Information-Gathering Skills In the early stages of clinical reasoning, learners must rely upon their knowledge of basic pathophysiology and …

16. Pediatrics Component Continued Additional Information (label = reference, position = 1): Schmidt HG, Norman GR, Boshuizen HPA. A cognitive perspective on medical expertise: theory and implications. Academic Medicine. 1990;65:611-621. Additional Information (label = reference, position = 2): Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: translating the Dreyfus Developmental Model to the learning of clinical skills. Academic Medicine. 2008;83:761-767. Additional Information (label = reference, position = 3): Eva K. What every teacher needs to know about clinical reasoning. Medical Education. 2004;39:98-106. Additional Information (label = reference, position = 3): Schmidt HG, Boshuizen HPA. On acquiring expertise in medicine. Educational Psychology Review. 1993;5:205-221. Etc…

17. Performance level set Performance Scale: 1 to 5 Position: 1Score: Single Value: 1 Label: Indicator (id = pc_i1): Either gathers too little information or exhaustively gathers information following a template regardless of the patient’s chief complaint, with each piece of information gathered seeming as important as the next. Recalls clinical information in the order elicited,7 with the ability to gather, filter, prioritize, and connect pieces of information being limited by and dependent upon analytic reasoning through basic pathophysiology alone. Position: 2Score Single Value: 2 Label: Indicator (id = pc_i2): Clinical experience allows linkage of signs and symptoms of a current patient to those encountered in previous patients. Still relies primarily on analytic reasoning through basic pathophysiology to gather information, but the ability to link current findings to prior clinical encounters allows information to be filtered, prioritized, and synthesized into pertinent positives and negatives as well as broad diagnostic categories. Free text reference No formatting

18. Position: 3Score: Single Value: 3 Label: Indicator (id = pc_i3): Advanced development of pattern recognition leads to the creation of illness scripts, which allow information to be gathered while it is simultaneously filtered, prioritized, and synthesized into specific diagnostic considerations. Data gathering is driven by real-time development of a differential diagnosis early in the information-gathering process.8 Position: 4Score: Single Value: 4 Label: Indicator (id = pc_i4): Well-developed illness scripts allow essential and accurate information to be gathered and precise diagnoses to be reached with ease and efficiency when presented with most pediatric problems, but still relies on analytic reasoning through basic pathophysiology to gather information when presented with complex or uncommon problems. Position: 5Score: Single Value: 5 Label: Indicator (id = pc_i5): Robust illness scripts and instance scripts (where the specific features of individual patients are remembered and used in future clinical reasoning) lead to unconscious gathering of essential and accurate information in a targeted and efficient manner when presented with all but the most complex or rare clinical problems. These illness and instance scripts are robust enough to enable discrimination among diagnoses with subtle distinguishing features.

19. Pediatrics Component ID: 12345 Title: Prescribe and perform all medical procedures Competency: Description: http://www.example.org/peds_milestones/PC8.xml Type: http://ns.medbiq.org/competencyobject/v1/http://www.example.org/peds_milestones/PC8.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/peds_milestones.xml Type: http://ns.medbiq.org/competencyframework/v1/ http://www.example.org/peds_milestones.xmlhttp://ns.medbiq.org/competencyframework/v1 Author: Patricia Hicks, MD

20. Additional Information (label = background) All of the competencies are involved in prescribing and performing medical procedures. In an integrated …

21. Additional Information Continued The component KSA of each procedure are numerous and complex. They include: Anatomy and Physiology Indications and Benefits Contra-indications and Risks Informed Consent Pain Management, Patient Psychological Preparation Specimen Handling Interpretation of Results or Outcomes Procedural Technique (multiple elements unique to procedure; common elements to all [e.g., sterile technique, situational awareness, course correction]) Post-procedure Management This approach to assessment makes some assumptions: Performance level is specific to each procedure based on the relevant components and level of responsibility of the physician. Given the variability of required components, measures of competence are based on all of the relevant components for that procedure. Performance level for a given procedure, therefore, requires reaching the desired performance level for each of the individual components.

22. Additional Information (label = reference, position = 1): Wigton R, Nicolas J, Blank L. Procedural skills of the general internist: a survey of 2500 physicians. Annals of Internal Medicine. 1989;111:1023-1034. Additional Information (label = reference, position = 2): Wigton R, Blank L, Nicolas J, Tape T. Procedural skills training in internal medicine residencies. Annals of Internal Medicine. 1989;111:932-938. Additional Information (label = reference, position = 3): Wigton R. Training internists in procedural skills. Annals of Internal Medicine. 1992;116:1091-1093. Additional Information (label = reference, position = 4): Hicks C, Gonzales R, Morton M, et al. Procedural experience and comfort level in internal medicine trainees. General Internal Medicine. 2000;15:716-722. Additional Information (label = reference, position = 5): Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler Publishers; 1998. Additional Information (label = reference, position = 6): Kirkpatrick L, Kirkpatrick JD. The four levels: an overview. In: Kirkpatrick DL, ed. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett- Koehler Publishers; 2006:21-26. Additional Information (label = reference, position = 7): Davis D. Accuracy of physician self-assessment compared with observed measures of competence. Journal of the American Medical Association. 2006:296: 1094-1102. Additional Information (label = reference, position = 8): Carbine D, Finer N, Knodel E, Rich W. Video recording as a means of evaluating neonatal resuscitation performance. Pediatrics. 2000;106:654-658. Additional Information (label = reference, position = 9): Adams K, Scott R, Perkin R, Langga L. Comparison of intubation skills between interfacility transport team members. Pediatric Emergency Care. 2000;16:5-8. Additional Information (label = reference, position = 10): Falck A, Escobedo M, Baillargeon J, et al. Proficiency of pediatric residents in performing neonatal endotracheal intubation. Pediatrics. 2003;112:1242-1247. Etc.

23. Nested Component ID: 98765 Competency: Description: http://www.example.org/peds_milestones/anatomy_physiology.xml Type: http://ns.medbiq.org/competencyobject/v1/ http://www.example.org/peds_milestones/anatomy_physiology.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/peds_milestones.xml Type: http://ns.medbiq.org/competencyframework/v1/ http://www.example.org/peds_milestones.xmlhttp://ns.medbiq.org/competencyframework/v1 Title: Anatomy and Physiology

24. Performance levels Performance Scale Reference: 1 to 4 Position: 1Score: Single Value: 1 Label: Beginning of Spectrum Indicator (id = pcap_i1): 2 SD below mean on knowledge test Position: 2Score: Single Value: 2 Label: Indicator (id = pcap_i2): 1 SD below mean on knowledge test Position: 3Score: Single Value: 3 Label: Indicator (id = pcap_i3): 1 SD above mean on knowledge test Position: 4Score: Single Value: 4 Label: Indicator (id = pcap_i4): 2 SD above mean on knowledge test

25. Pediatrics Component from http://www.acgme-nas.org/assets/pdf/Milestones/PediatricsMilestones.pdf http://www.acgme-nas.org/assets/pdf/Milestones/PediatricsMilestones.pdf ID: 90876 Title: Identify strengths, deficiencies, and limits in one’s knowledge and expertise Competency: Description: http://www.example.org/peds_milestones/PBLI1.xml Type: http://ns.medbiq.org/competencyobject/v1/http://www.example.org/peds_milestones/PBLI1.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/peds_milestones.xml Type: http://ns.medbiq.org/competencyframework/v1/ http://www.example.org/peds_milestones.xmlhttp://ns.medbiq.org/competencyframework/v1

26. Performance Levels Performance Scale Reference: 1 to 4 Position: 1Score: Single Value: 1 Label: Level 1 Indicator (id = pbli_i1): The learner acknowledges external assessments, but understanding of his performance is superficial and limited to the overall grade or bottom line; has little understanding of how the performance measure relates in a meaningful way to his specific level of Knowledge, Skills and Attitudes (KSA) Additional Information (label=example): During a semiannual review, a learner is unable to describe in any specific terms how he has performed when asked to do so by his mentor. In response, the mentor reviews and interprets the learner’s evaluations and then asks the learner to reflect on the discussion. The learner repeats the language used and recites the overall score/grade without interpretation of further meaning or inference regarding the reported performance assessment.

27. Nursing (DNP Eval) Example Title: University of San Diego Hahn School of Nursing and Health Science DNPC 630 Residency DNP NP Student Evaluation Identifier: – Catalog: URI – Entry: http://www.sandiego.edu/nursing/DNPC_630_Eval http://www.sandiego.edu/nursing/DNPC_630_Eval Performance Scale – 1 to 3 (1 least competent, 3 most competent)

28. DNP Component Competency: Description: http://www.example.org/dnp630_3.xml Type: http://ns.medbiq.org/competencyobject/v1/http://www.example.org/dnp630_3.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/dnp630.xml Type: http://ns.medbiq.org/competencyframework/v1/http://ns.medbiq.org/competencyframework/v1 ID: 5678 Title: Prepared to practice independently managing previously diagnosed and undiagnosed patients.

29. Performance Levels Performance Scale Reference: 1 to 3 Position: 1Score Single Value: 3 Label: Indicator (id = pi_i1): Met Position: 2Score Single Value: 2 Label: Indicator (id = pi_i2): In progress Position: 3Score Single Value: 1 Label: Indicator (id = pi_i3): Not Met

30. National University of Singapore Example Title: Standards of Achievement Identifier: – Catalog: URI – Entry: http://www.nus.edu.sg/standards2013http://www.nus.edu.sg/standards2013 Performance Scale – 1 to 5 (1 least competent, 5 most competent)

31. NUS Component ID: 2013_1 Title: Standards of Achievement

32. Performance Levels Performance Scale Reference: 1 to 5 Position: 1Score: Single Value: 1 Label: Indicator (id = i1): Unable to achieve outcome. Position: 2Score: Single Value: 2 Label: Indicator (id = i2): Requires a lot of guidance to achieve outcome. Position: 3Score: Single Value: 3 Label: Indicator (id = i3): Requires moderate amount of guidance to achieve outcome.

33. Performance Levels Position: 4Score: Single Value: 4 Label: Indicator (id = i4): Able to achieve outcome with little or no guidance (entrustment) Position: 5Score: Single Value: 5 Label: Indicator (id = i5): Has the ability to guide/teach others.

34. Diagnostic Radiology Example (see http://www.acgme-nas.org/assets/pdf/Milestones/DiagnosticRadiologyMilestones.pdf)http://www.acgme-nas.org/assets/pdf/Milestones/DiagnosticRadiologyMilestones.pdf Title: The Diagnostic Radiology Milestone Project Identifier: – Catalogue: URI – Entry: http://www.acgme- nas.org/assets/DiagnosticRadiologyMilestones Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… Contributions: – Role: Author – Entity: Kay Vydareny, MD – Role: Author – Entity: E. Stephen Amis Jr., MD Effective date: 2013-01-30

35. Diagnostic Radiology Continued Supporting Information – This document presents milestones designed for programs to use… Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

36. Diagnostic Radiology Component ID: 546756 Competency: Description: http://www.example.org/dr_milestones/PROF1.xml Type: http://ns.medbiq.org/competencyobject/v1/http://www.example.org/dr_milestones/PROF1.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/dr_milestones.xml Type: http://ns.medbiq.org/competencyframework/v1/http://ns.medbiq.org/competencyframework/v1 Title: Professional Values and Ethics Additional Information (label = Assessment method): End-of-Rotation Global Assessment Additional Information (label = Assessment method): 360 Evaluation/Multi- rater/Peer Additional Information (label = Assessment method): Simulation/OSCE Additional Information (label = Assessment method): Direct observation and feedback Additional Information (label = Assessment method): Conference attendance logs Additional Information (label = Assessment method): Timeliness in completing institutional and program requirements

37. Diagnostic Radiology Component continued Additional Information (label = resource): Teaching and Assessing Professionalism: A Program Director’s Guide by the ABP and APPD. See Chapter 8: Measuring Professionalism, Critical incidents, Peer assessments, Multi-source assessments, Professionalism Mini- Evaluation Exercise (P-MEX) Additional Information (label = resource): The Professionalism Mini-Evaluation Exercise: A Preliminary Investigation. Richard Cruess, Jodi Herold McIlroy, Sylvia Cruess, Shiphra Ginsburg, and Yvonne Steinert Acad Med. 2006 Oct;81(10 Suppl):S74-8 These are suggested educational tools.

38. Diagnostic Radiology Component continued Additional Information (label = resource): Text: ABRF Online Modules on Ethics and Professionalism Dcterms: references: Description: https://www.abronline.org/asp/abrf/ https://www.abronline.org/asp/abrf/

39. Performance level set Performance Scale Reference: 1 to 5 Position: 1Score: Single Value: 1 Label: Level 1 Indicator (id = p1_1): recognizes the importance and priority of patient care and advocates for patient interests Indicator (id = p1_2): fulfills work-related responsibilities Indicator (id = p1_3): is truthful Indicator (id = p1_4): recognizes personal limitations and seeks help when appropriate Indicator (id = i1_5): recognizes personal impairment and seeks help when needed Etc…

40. Position: 2Score: Single Value: 2 Label: Level 2 Indicator (id = p2_1): Is an effective health care team member Indicator (id = p2_2): Demonstrates professional behaviors listed in the second column Position: 3Score: Single Value: 3 Label: Level 3 Indicator (id = p3_1): Is an effective health care team leader, promoting primacy of patient welfare, patient autonomy, and social justice Indicator (id = p3_2): Demonstrates professional behaviors listed in the second column Position: 4Score: Single Value: 4 Label: Level 4 Indicator (id = p4_1): Serves as a role model for professional behavior Indicator (id = p4_2): Demonstrates professional behaviors listed in the second column

41. Position: 5Score: Single Value: 5 Label: Level 5 Indicator (id = p5_1): Participates in local and national organizations to advance professionalism in radiology Indicator (id = p5_2): Mentors others regarding professionalism and ethics

42. Psychiatry Example (see http://www.acgme-nas.org/assets/pdf/Milestones/PsychiatryMilestones.pdf ) http://www.acgme-nas.org/assets/pdf/Milestones/PsychiatryMilestones.pdf Title: The Psychiatry Milestone Project Identifier: – Catalogue: URI – Entry: http://www.acgme-nas.org/assets/PsychiatryMilestones Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… Contributions: – Role: Author – Entity: Sheldon Benjamin, M.D. – Role: Author – Entity: Adrienne L. Bentman, M.D Effective date: 2013-01-30

43. Psychiatry Continued Supporting Information – This document presents milestones designed for programs to use… Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

44. Psychiatry Component ID: 546756 Competency: Description: http://www.example.org/psy_milestones/pc4.xml Type: http://ns.medbiq.org/competencyobject/v1/ http://www.example.org/psy_milestones/pc4.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/psy_milestones.xml Type: http://ns.medbiq.org/competencyframework/v1/http://www.example.org/psy_milestones.xmlhttp://ns.medbiq.org/competencyframework/v1 Title: Psychotherapy Refers to 1) the practice and delivery of psychotherapies, including but not limited to psychodynamic (1), cognitive-behavioral (2), and supportive therapies (3); 2) exposure to couples, family, and group therapies; and 3) integrating psychotherapy with psychopharmacology Additional Information (label = note, position = 1): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, and to understand the concepts of resistance/defenses, transference/countertransference

45. Psychiatry Component (continued) Additional Information (label = note, position = 2): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, including behavior change, skills acquisition, and addressing cognitive distortions Additional Information (label = note, position = 3): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, and to strengthen the patient’s adaptive defenses, resilience, and social supports

46. Performance level set Performance Scale Reference: 1 to 5 Position: 1Score: Single Value: 1 Label: Level 1 Indicator (id = pc4_1_1): 1.1 Accurately identifies patient emotions, particularly sadness, anger, and fear (4) Additional Infomration (label =note): This thread, consisting of the first item at each level, regarding the development of empathy across residency is adapted from the AADPRT Psychotherapy Workgroup’s document “Benchmarks for Psychotherapy Training.” Indicator (id = pc4_1_2): 1.2 Demonstrates interest and curiosity in patient’s story

47. Position: 2Score: Single Value: 12 Label: Level 2 Indicator (id = pc4_2_1): 1.1 Accurately identifies patient emotions, particularly sadness, anger, and fear (4) Indicator (id = pc4_2_2): 1.2 Demonstrates interest and curiosity in patient’s story Etc.

48. Score ranges

49. Performance level set Performance Scale Reference: 1 to 10 Position: 1Score: Range MinScore: 1 MaxScore: 3 Label: Below Minimum Indicator (id = i1_1): Performs inaccurate or grossly incomplete PE Indicator (id = i1_2): Misses major or obvious findings in most cases Position: 2Score: Range MinScore: 4 MaxScore: 6 Label: Marginal Performance Indicator (id = i2_1): Frequent significant omissions or inaccuracies Position: 3Score: Range MinScore: 7 MaxScore: 9 Label: Expected Performance Indicator (id = i3_1): Occasional omissions or inaccuracies Position: 4Score: Single value: 10 Label: Exceeds Expectations Indicator (id = i4_1): Complete, accurate PE; rare omissions or inaccuracies

50. TIME Example Title: Transformation in Medical Education Identifier: – Catalogue: URI – Entry: http://www.utsystem.edu/initiatives/time/http://www.utsystem.edu/initiatives/time/ Description: This document elaborates the initial vision for competency- based education (CBE) in the University of Texas System's Transformation in Medical Education (TIME) initiative. CBE is one of the four principal elements of the TIME model of predoctoral physician education… Contributions: – Role: Author – Entity: Pedro Reyes, Ph.D. – Role: Author – Entity: Steve Lieberman, M.D. Effective date: 2013-01-30

51. TIME Continued Performance Scale – 1 to 7 (1 least competent, 7 most competent)

52. TIME Component ID: 56789 Title: Demonstrate compassion and respect for patient Competency: Description: http://www.example.org/time_milestones/PC1_1_3.xml Type: http://ns.medbiq.org/competencyobject/v1/http://www.example.org/time_milestones/PC1_1_3.xmlhttp://ns.medbiq.org/competencyobject/v1 Competency: Description: http://www.example.org/time_framework.xml Type: http://ns.medbiq.org/competencyframework/v1/http://www.example.org/time_framework.xmlhttp://ns.medbiq.org/competencyframework/v1

53. TIME Component (continued) Threshold – Title: Milestone for transition to medical campus – compassion and respect – Description: At the time of transition from the undergraduate to the medical campus, the student will have demonstrated the ability to demonstrate compassion and respect for a patient in a stressful encounter when faced with a purpose‐designed standardized patient based simulation challenge at the following level. – MinimumAcceptableScore: 4

54. Performance level set Performance Scale Reference: 1 to 7 Position: 1Score: Single Value: 1 Label: I Indicator (id = pc1_1_3_1): Demonstrates decontextualized skills, knowledge, behaviors Position: 2Score: Single Value: 2 Label: II Indicator (id = pc1_1_3_2): Demonstrates application of knowledge, skills, behavior to relevant health care problems Position: 3Score: Single Value: 3 Label: II+ Indicator (id = pc1_1_3_3): Competency demonstration is required in non‐clinical settings as a transition milestone. The skill/behavior is expected whenever the student is in clinical settings, but its demonstration is not required as a transition milestone.

55. Position: 4Score: Single Value: 4 Label: III Indicator (id = pc1_1_3_4): Demonstrates competency in simulated environments. Position: 5Score: Single Value: 5 Label: IV Indicator (id = pc1_1_3_5): Consistently demonstrates competency in a clinical setting under direct supervision. Position: 6Score: Single Value: 6 Label: V Indicator (id = pc1_1_3_6): Consistently demonstrates competency in a clinical setting with supervision immediately available. Position: 7Score: Single Value: 7 Label: VI Indicator (id = pc1_1_3_7): Consistently demonstrates competency in a clinical setting without supervision (“entrustable”).

56. How are use cases addressed? Map performance frameworks to competencies, including Entrustable Professional Activities – YES (Competency Reference) Publish a performance framework for use in curriculum planning and assessment – YES (Requires IDs for Components so that assessments may be tied to performance levels) Import descriptions of performance levels for use in assessment – YES Reference a Performance Framework for use in assessment – YES (Requires IDs for Components so that assessments may be tied to performance levels) Describe an individual’s current level of performance for purposes of formative or summative assessment – YES (a score and a reference to a Component should do the trick) Describe an individual’s level of performance over a longitudinal period for purposes of formative or summative assessment – YES (provided the framework is authored to support that) Associate assessment evidence with a particular level of performance in a portfolio – Change: assessment evidence to have a score that is interpreted using a particular group of performance levels defined for the relevant competency.

57. How are use cases addressed? Define where in a curriculum students are expected to achieve certain levels of performance – YES (will require updates to curriculum inventory, may require id on performance level) Define what level of performance is required to progress to the next phase or block within a curriculum – YES (will require updates to curriculum inventory, may require id on performance level) Use in a system that is capable of showing a learner changes in performance over time – YES View data regarding the performance levels of learners in a program for purposes of program evaluation (external) – YES (would require a spec that allows for the exchange of aggregate data) View data regarding the performance levels of learners in a program for purposes of program evaluation (internal) – YES Publish program data for public viewing – YES (would require a spec that allows for the exchange of aggregate data)