MedBiquitous 2016: Technical Challenges in Uploading to the Curriculum Inventory Terri Cameron, MA Director, Curriculum Programs, AAMC

We will cover: Quick Introduction to Curriculum Inventory Terminology AAMC Business Rules for Curriculum Inventory data What curriculum management system data should be included in the XML export to the Curriculum Inventory Vendor / Sender Onboarding Process Staging Data Upload Process Data Verification Process Challenges

What is the Curriculum Inventory An international repository of curriculum information that can be used to report on what is being taught in medical schools, how content is being taught, how content is being assessed, the resources used to teach and assess, and the competencies taught in conjunction with the content.

Who Participates in the CI? 2015: 135 Schools 127 USMD 5 CAMD 3 DO (Pilot) 2014: 120 Schools 115 USMD 5 CAMD Inaugural Year (2013): 90 Schools 85 USMD 5 CAMD

Curriculum Inventory Academic Levels 1-6, predominantly, 1 Average Number of Events: 75 Median Number of Events: 53 Top 5 Instructional Methods: Laboratory Lecture Computer-Based Instruction Case-based Learning Discussion, Small Group Top 5 Assessment Methods Exam - Institutionally Developed, Written/ Computer-based Practical (Lab) Participation Exam - Nationally Normed/Standardized, Subject Self-Assessment

Terminology Previous Academic Year (LCME model – if current year is , previous academic year is ) Academic Level / Phase (formerly called ‘year of curriculum’) Sequence Block /Nested Sequence Block (Course, Module, Unit, Block, Clerkship) Integration Block (Theme, Vertical / Horizontal Integration) Event (Instructional or Assessment Session) (e.g., Lecture, Lab, Discussion, Assessment) Expectations / Competency Framework (Competency / Objective / Milestone / EPA) MedBiquitous (

Standardized Vocabulary  Use local terms for institutional data entry and reporting; match to standardized vocabulary to upload to CIR for aggregate reporting  Instructional Methods  Assessment Methods  Resources  UIDs available at  Detailed descriptions available at

Curriculum Inventory: Standardized Vocabulary Update To be implemented in 2018? To be implemented in 2016, with or without IDs AM15 Renamed: AM19: Exam – Institutionally Developed, Laboratory Practical New!

Physician Competency Reference Set (PCRS)  Compared/mapped healthcare profession competency sets to create a set of ‘core’ competencies for aggregate reporting  ACGME (including RRCs)  CanMEDS  Scottish Doctor  Tomorrow’s Doctors  Healthcare Professions  Specialty Objectives (ABMS, Clerkships)  Process and complete list of competencies published in August 2013 Academic Medicine  URI available at

Physician Competency Reference Set (PCRS) Partial URI Physician Competency Reference Set (AAMC 2013) aamc-pcrs-comp-c0100 PATIENT CARE: Provide patient-centered care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health aamc-pcrs-comp-c0101 Perform all medical, diagnostic, and surgical procedures considered essential for the area of practice aamc-pcrs-comp-c0102 Gather essential and accurate information about patients and their condition through history-taking, physical examination, and the use of laboratory data, imaging, and other tests aamc-pcrs-comp-c0103 Organize and prioritize responsibilities to provide care that is safe, effective, and efficient aamc-pcrs-comp-c0104 Interpret laboratory data, imaging studies, and other tests required for the area of practice aamc-pcrs-comp-c0105Make informed decisions about diagnostic and therapeutic interventions based on patient information and preferences, up-to- date scientific evidence, and clinical judgment

Matching School Expectations to PCRS Program-Level (School) Expectations must be mapped to PCRS competencies May NOT be mapped to PCRS domains ‘Other’ provided for instances where there is not a direct match between expectations Event- and Sequence Block-Level Expectations may NOT be mapped to PCRS Event and Sequence Block Expectations may be mapped to each other or to Program Expectations or not at all This mapping is pulled from your curriculum management system PCRSProgram Sequence Block Event

Matching Program Objectives to PCRS (Crosswalk) PCRSProgram Objectives 2. Knowledge for Practice: Demonstrate knowledge of established and evolving biomedical, clinical, epidemiological and social-behavioral sciences, as well as the application of this knowledge to patient care. 2.1 Demonstrate an investigatory and analytic approach to clinical situations 2.2 Apply established and emerging bio-physical scientific principles fundamental to health care for patients and populations 2.3 Apply established and emerging principles of clinical sciences to diagnostic and therapeutic decision- making, clinical problem-solving, and other aspects of evidence-based health care 2.4 Apply principles of epidemiological sciences to the identification of health problems, risk factors, treatment strategies, resources, and disease prevention/health promotion efforts for patients and populations 2.5 Apply principles of social-behavioral sciences to provision of patient care, including assessment of the impact of psychosocial and cultural influences on health, disease, care-seeking, care compliance, and barriers to and attitudes toward care 2.6 Contribute to the creation, dissemination, application, and translation of new health care knowledge and practices 2.99 Other knowledge for practice 2.Demonstrate a broad working knowledge of the fundamental science, principles, and processes basic to the practice of medicine. 2.1 Understand the clinical relevance of scientific inquiry. 2.2 Apply this knowledge in a judicious and consistent manner to prevent common health problems and achieve effective and safe patient care. 2.3 Demonstrate the ability to evaluate emerging knowledge and research as it applies to diagnosis, treatment and the prevention of disease. 2.4 Utilize state of the art information technology and tools to retrieve, manage and use biomedical information in the care of individuals. 2.5 Utilize state of the art information technology and tools to retrieve, manage and use biomedical information in the care of populations. 2.6 Understand the indications, contraindications, and potential complications of common clinical procedures. 2.7 Perform the basic clinical procedures expected of a new PGY-1.

Competencies vs Objectives Objectives Competencies Expectations Milestones EPAs Competencies, Learning Objectives, Milestones, and Entrustable Professional Activities (EPAs) are all separate concepts with explicit definitions and purposes. For the Curriculum Inventory, they are all “Expectations”. The category element can be used to distinguish whether an Expectation is a learning objective, competency, etc., but there are no requirements to use that option at this time.

AAMC Business Rules Core Rules (CR) Core rules must be met before additional rules for the competency framework and the curriculum inventory structure are applied. If it is detected that a submission does not meet a core rule: a) processing of the submission discontinues immediately and b) an error notification is generated that describes the invalid core rule.

AAMC Business Rules Core Rules (CR) CR01: Approved Sender Senders must be certified via AAMC’s on- boarding process before the web service will allow them to send submissions. CR02: Selected Sender – Schools must select their Sender via the School CI Portal ‘Manage Vendor’ page. o Vendor o School

AAMC Business Rules Core Rules (CR) CR03: One Inventory for One School A submission, or XML file sent to the web service, must include data for only one curriculum inventory for one school. CR04: Institution ID The must match an AAMC institution ID for a school. AAMC EIS Code

AAMC Business Rules Core Rules (CR) CR05: Submission File Size The file size cannot exceed 19mg An error will be generated if data in any field exceeds the record length Zip files accepted CR06: Data Dictionary Each field in the XML has a specific length limit, which is documented in the AAMC Business Rules. An error will be generated if data in any field exceeds its maximum allowable length

AAMC Business Rules Competency Framework and Competency Object Rules (CF) These rules are evaluated only once all core rules have been met. These rules and their application are covered in CI Developers Workshop Part II.

AAMC Business Rules Curriculum Structure Rules (CS) These rules are evaluated only once all core rules have been met.

AAMC Business Rules Curriculum Structure Rules (CS) CS01: Unique Report ID The sub-element must be unique for each of a school’s submissions. If a Curriculum Inventory Upload is rejected, the next uploaded file must not have the same ReportID. The sub-element must be alphanumeric The Report ID must not include dashes, underscores or other special characters

AAMC Business Rules Curriculum Structure Rules (CS) CS02: Previous Academic Year Submissions are checked to ensure they contain information for only the previous academic year (AY). Schools must submit data with and between July (of the previous year) and June (of the current year). Note: Dates at the sequence block or event level are not checked to ensure they are within the previous academic year. Dates specified at these levels that fall outside of the date range for a previous academic year will not cause an error or rejection.

AAMC Business Rules Curriculum Structure Rules (CS) CS03: Number of Academic Levels The number of academic levels, or phases, are defined by. Each academic level is defined as a within the tag. CS05: Academic Level Numbers sub-elements of the tag must have sequential numbers (i.e., the number attribute) starting with 1.

AAMC Business Rules Curriculum Structure Rules (CS) CS04: Academic Levels and Sequence Blocks A that is not defined within the tag cannot be referenced by a sequence block. CS06: Academic Levels and Sequence Blocks Each academic level must be referenced by at least one sequence block.

AAMC Business Rules Sequence Blocks CS07: tags must have unique identifiers (i.e., the id attribute). CS08: tags must be associated to a defined academic level using the required sub-element. CS09: The sub-element of all tags with a ClerkshipModel attribute must specify and. Document start and end dates for entire iteration period (e.g., 07/01/12 – 06/30/13). CS16: If it is provided, the duration of a sequence block must be specified as a number of days. For clerkships, 5 days = 1 week

AAMC Business Rules Curriculum Structure Rules (CS) CS10: Nested Sequence Blocks When nesting sequence blocks, must not result in circular references. See “Note about Hierarchical Conflict” section (pp ) of the Competency Framework Specification, version 1.0.

AAMC Business Rules Curriculum Structure Rules (CS) Events CS11: tags must have unique identifiers (i.e., the id attribute). CS12: Within tags, the and tags must reference a unique ID from the Curriculum Inventory Standardized Vocabulary. CS13: At a minimum, each must reference at least one or. CS14: If one or more tags are provided, one and only one must be denoted as primary. CS15: All tags must be referenced by at least one tag within a.

Curriculum Inventory Process Portal Manage Data (Home Screen) Download Verification Report After successful data upload Reject or Verify Data Manage Users Schools will be encouraged add Vendor with CI Viewer role Manage Vendors Vendor must be selected by school in Portal or data will not upload

Curriculum Inventory Process Data Submit Data ‘Portal Open’ Notification sent 08/01 System Check for (MB CI Standard, Terminology, Competencies) Verify Data Verification Report available as soon as data is successfully submitted Deadline for verifying data is 09/30 ‘Portal Closed’ Notification sent 10/01 Create Snapshot of data Reports process begins immediately

Curriculum Inventory Process Reminders and Notifications Curriculum Dean / Primary CI Admin Dean Ramifications No access to CI Reports Continuous Quality Improvement / Accreditation Documentation Portal Only available actions: Download Verification Report Manage Users Manage Vendors

Upload Process Vendor / Sender Dependent CMS exports XML file File is manually uploaded by School (or Vendor if Vendor has CI Primary Admin role) Web services upload from Sender to AAMC File is uploaded via a function within the CMS

Web Services

Web Services / Vendor Onboarding Process School Developers and Vendors who wish to use Web Services must complete an Electronic Information Exchange Agreement (System Developer section of Uploads from Web Services must be from IP address range provided in EIEA SOAP UI must be used to upload via web services (set up documentation in System Developer section of

Staging Web Services or Manual Upload via Staging School Portal Web Services Same process as Production – EIEA must be completed for Web Services, and SOAPUI must be used to upload data Manual Upload via Staging School Portal Each school has a Staging Site Contact for Credentials and Change Sender options based on web services or manual upload School does not need to be involved No Verification option, but Verification Report is created Staging is open all year; converts to next upload year in February

Curriculum Inventory Process An activity log displays all activity – data upload attempts and status, Verification Report downloads, and data rejections and verifications.

Curriculum Inventory Process

The data and Verification Report will remain available until the upload process for the next academic year begins

CI Verification Report Institution (Program Information) All Academic Levels (Dates) All Sequence Blocks Expectations / Expectation Mapping (if provided) All Events Expectation Mapping (if available) Instructional Methods Primary Instructional Methods Assessments Events with Assessment Method(s) only

CI Verification Report

Pearls A Sequence Block Week is 5 Days Clerkships are Sequence Blocks with Clerkship Model (Rotation or Longitudinal) rotation Check date / duration formats PT2H PT90M PT1H30M P13D P12M

Pearls Events must have Instructional Method OR Assessment Method(s) OR Both Use Instructional Method and Assessment Method UIDs!!!! IM013 AM012 Can have multiple Instructional Methods but one must be Primary Can have multiple Assessment Methods – no Primary Must have Duration Formative / Summative Assessment Events No Instructional Methods

Pearls Country Code: 3-character code (USA, CAN) USA

Pearls Program Code: 2-character code (MD, DO, PA) M.D Parallel Tracks Denote same cohort of students in separate tracks (should not be used to designate concurrent Academic Levels for multiple cohorts of students)

Pearls Special Characters: When data is copied from a website and pasted into Curriculum Management Systems, the special characters that make up the HTML can cause problems either in the upload process OR in reporting. In addition, these special characters sometimes cause unexpected truncation because they fill the character limit for the field. Solution: when possible, use the paste function ‘text only’ when pasting text into a CMS

Pearls Extra spaces: = 'E395'] This issue often makes it through the error checking and results in a successful upload, but the Verification Report is not generated

Issues Found in Data Analysis Issue 1: copying one course over to all other courses Issue 2: keywords that just have numbers Issue 3: synonyms Issue 4: no titles for competency objects Issue 5: Uploading only one event for all sequence blocks Issue 6: Uploading only one academic level

12345 NoSuch University School of Medicine Main Street Baltimore MD US M.D NoSuch School of Medicine Curriculum en-us The NoSuch curriculum reframes the context of health and illness to encourage students to explore a larger, integrated system.

Clerkships Multiple Small Groups Electives Tracks Mapping Terminology Nested Sequence Blocks Competency Levels Challenges

Gateway Exams Release School Name CEPAER Documentation Initiatives

Best Practice for ‘Gateway Exams’ Gateway Exams: Major Exams for Promotion or Graduation Often multi-day exams Not related to a particular course or clerkship Often a series of unqueued cases Examples: Senior Clinical Competency Exam; CCX; Post-Clerkship Exam

Best Practice for ‘Gateway Exams’ Documenting Gateway Exams: Separate Sequence Block Duration expected for average student Include all Assessment Methods as Single Event (e.g., All Cases) or Split into Multiple Events (e.g., Case 1, Case 2.) No Instructional Methods Include Keywords for Content if possible Reporting is post-exam and aggregate If exam is split into multiple Academic Levels, reference the AL where most of the exam occurs Use Sequence Block Description to provide innovations, information such as year exam was implemented, etc.

Using Program Description to Designate Approval of School Name linked to CI Reports A Brief History CurrMIT allowed schools to run reports to find schools offering particular content or using particular instructional or assessment methods or resources Received both positively and negatively To expedite implementation of CI and ensure highest participation rate, schools were told all reports would be aggregate Received both positively and negatively

Using Program Description to Designate Approval of School Name linked to CI Reports Solution: Allow schools to choose whether they wish to have their name available for benchmarking and research School name will be in a separate data file – no link to actual data in report Schools Included in This Report link will include: School Name, Curriculum Dean Contact Information, Primary CI Contact Information

Using Program Description to Designate Approval of School Name linked to CI Reports Implementation August 1, 2016, for Data: Use Program Description to include the text: ReleaseName=Y OR ReleaseName=N No ReleaseName text will be the same as ReleaseName=N This should not require changes to programs – simply adding text to a field For systems that do not have this option, an can be sent to will need to be sent each year.

Best Practices for CEPAER Documentation CEPAER: Core EPAs for Entering Residency 13 EPAs Linked to PCRS Competencies In Pilot Phase (10 schools) Not a requirement, but being documented at many schools Document as CEPAER-01 – CEPAER-13 Detailed information (included PCRS mapping) can be found at:

Schedule Staging available now School Portal Opens: August 1 School Portal Closes: September 30 Status sent to CI Primary Admin, Curriculum Dean, and School Dean: September 30 Data clean-up/reporting starts: November 1

Resources Curriculum Inventory Implementers’ Listserv: Competency Framework Specs and Schemas: AAMC Curriculum Inventory Developer Resources: Physician Competency Reference Set URIs: MedBiquitous: Curriculum Inventory Working Group: Competencies Working Group:

Questions? List of upcoming Curriculum Inventory Webinars and materials from today’s session:

Reports Webinars and training Register for Curriculum Deans and Admins Webinar April 30! Information about Curriculum Management System vendors Tools for Developers Curriculum Inventory School Portal For more information

Questions???