Preparing for the Curriculum Inventory Terri Cameron, MA Director, Curriculum Programs AAMC

We will cover: Quick Introduction to Curriculum Inventory MedBiquitous Curriculum Inventory Standard AAMC Business Rules for Curriculum Inventory data What curriculum management system data should be included in the XML export to the Curriculum Inventory Vendor / Sender Onboarding Process Data Upload Process Data Verification Process

What is the Curriculum Inventory An international repository of curriculum information that can be used to report on what is being taught in medical schools, how content is being taught, how content is being assessed, the resources used to teach and assess, and the competencies taught in conjunction with the content.

Who Participates in the CI? 2015: 135 Schools 127 USMD 5 CAMD 3 DO (Pilot) 2014: 120 Schools 115 USMD 5 CAMD Inaugural Year (2013): 90 Schools 85 USMD 5 CAMD

Why participate in the CI? The Curriculum Inventory is the culmination of a long history of the AAMC collecting information about medical education content, structure, and pedagogy to support: Medical Education Research (medical school faculty and external researchers) Continuous Quality Improvement / Benchmarking Support of Curriculum Committees Reporting on new trends Responding to legislative inquiries or providing data when important issues are being discussed Respond to media inquiries The Curriculum Inventory is only effective when the data represents most, if not all, medical schools participate. Only participating schools can request detailed Curriculum Inventory Reports.

Continuous Quality Improvement and Accreditation Curriculum Inventory Reports MedBiquitous Data Exchange Standard Curriculum data mapped to a competency hierarchy (competencies to course objectives to session objectives) are uploaded to the Curriculum Inventory (CI) from school systems using the MedBiquitous CI data exchange standard ( Standardized Vocabulary for Instructional Methods, Assessment Methods, and Resources Institutionally- developed competencies will be matched to Physician Competency Reference Set (PCRS) Vendor and School – developed Curriculum Management Systems Curriculum Inventory Part II

Questions? Please use Q & A option to submit questions.

Terminology

Previous Academic Year (LCME model – if current year is , previous academic year is ) Academic Level / Phase (formerly called ‘year of curriculum’) Sequence Block /Nested Sequence Block (Course, Module, Unit, Block, Clerkship) Integration Block (Theme, Vertical / Horizontal Integration) Event (Instructional or Assessment Session) (e.g., Lecture, Lab, Discussion, Assessment) Expectations / Competency Framework (Competency / Objective / Milestone / EPA) MedBiquitous ( Part II

Standardized Vocabulary  Use local terms for institutional data entry and reporting; match to standardized vocabulary to upload to CIR for aggregate reporting  Instructional Methods  Assessment Methods  Resources  UIDs available at  Detailed descriptions available at

Curriculum Inventory Standardized Vocabulary Update Renamed Renamed: Exam – Institutionally Developed, Laboratory Practical

Matching Local Terminology to CI Standardized Terminology Local Terminology: Instructional Methods CI Standardized Terminology Clinical SkillsClinical Experience - Outpatient Procedure TrainingClinical Experience - Inpatient Resource SessionLecture Standardized PatientSimulation (with Resource: Standardized Patient) Wrap-Up SessionDiscussion, Small Group The MedBiquitous Curriculum Inventory Standardized Terminology Subcommittee reviews the terminology each year, based on literature and terminology schools report as difficult to match (challenges) Submit matching ‘challenges’ to or post comments in

Physician Competency Reference Set (PCRS)  Compared/mapped healthcare profession competency sets to create a set of ‘core’ competencies for aggregate reporting in MedAPS and MedEdPORTAL  ACGME (including RRCs)  CanMEDS  Scottish Doctor  Tomorrow’s Doctors  Healthcare Professions  Specialty Objectives (ABMS, Clerkships)  Process and complete list of competencies published in August 2013 Academic Medicine  URI available at

Physician Competency Reference Set (PCRS) Partial URI Physician Competency Reference Set (AAMC 2013) aamc-pcrs-comp-c0100 PATIENT CARE: Provide patient-centered care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health aamc-pcrs-comp-c0101 Perform all medical, diagnostic, and surgical procedures considered essential for the area of practice aamc-pcrs-comp-c0102 Gather essential and accurate information about patients and their condition through history-taking, physical examination, and the use of laboratory data, imaging, and other tests aamc-pcrs-comp-c0103 Organize and prioritize responsibilities to provide care that is safe, effective, and efficient aamc-pcrs-comp-c0104 Interpret laboratory data, imaging studies, and other tests required for the area of practice aamc-pcrs-comp-c0105Make informed decisions about diagnostic and therapeutic interventions based on patient information and preferences, up-to- date scientific evidence, and clinical judgment

Matching School Expectations to PCRS Program-Level (School) Expectations must be mapped to PCRS competencies May NOT be mapped to PCRS domains ‘Other’ provided for instances where there is not a direct match between expectations Event- and Sequence Block-Level Expectations may NOT be mapped to PCRS Event and Sequence Block Expectations may be mapped to each other or to Program Expectations or not at all This mapping is pulled from your curriculum management system PCRSProgram Sequence Block Event

Matching Program Objectives to PCRS (Crosswalk) PCRSProgram Objectives 2. Knowledge for Practice: Demonstrate knowledge of established and evolving biomedical, clinical, epidemiological and social-behavioral sciences, as well as the application of this knowledge to patient care. 2.1 Demonstrate an investigatory and analytic approach to clinical situations 2.2 Apply established and emerging bio-physical scientific principles fundamental to health care for patients and populations 2.3 Apply established and emerging principles of clinical sciences to diagnostic and therapeutic decision- making, clinical problem-solving, and other aspects of evidence-based health care 2.4 Apply principles of epidemiological sciences to the identification of health problems, risk factors, treatment strategies, resources, and disease prevention/health promotion efforts for patients and populations 2.5 Apply principles of social-behavioral sciences to provision of patient care, including assessment of the impact of psychosocial and cultural influences on health, disease, care-seeking, care compliance, and barriers to and attitudes toward care 2.6 Contribute to the creation, dissemination, application, and translation of new health care knowledge and practices 2.99 Other knowledge for practice 2.Demonstrate a broad working knowledge of the fundamental science, principles, and processes basic to the practice of medicine. 2.1 Understand the clinical relevance of scientific inquiry. 2.2 Apply this knowledge in a judicious and consistent manner to prevent common health problems and achieve effective and safe patient care. 2.3 Demonstrate the ability to evaluate emerging knowledge and research as it applies to diagnosis, treatment and the prevention of disease. 2.4 Utilize state of the art information technology and tools to retrieve, manage and use biomedical information in the care of individuals. 2.5 Utilize state of the art information technology and tools to retrieve, manage and use biomedical information in the care of populations. 2.6 Understand the indications, contraindications, and potential complications of common clinical procedures. 2.7 Perform the basic clinical procedures expected of a new PGY-1.

Competencies vs Objectives Objectives Competencies Expectations Milestones EPAs Competencies, Learning Objectives, Milestones, and Entrustable Professional Activities (EPAs) are all separate concepts with explicit definitions and purposes. For the Curriculum Inventory, they are all “Expectations”. The category element can be used to distinguish whether an Expectation is a learning objective, competency, etc., but there are no requirements to use that option at this time.

Questions? Please use Q & A option to submit questions.

The Curriculum Inventory Standard

® Mission: To develop and promote technology standards for the health professions that advance lifelong learning, continuous improvement, and better patient outcomes. Not-for-profit, member-driven, standards development organization

Building the Foundation: The MedBiquitous Architecture ( * American National Standards † Candidate standard

MedBiquitous Curriculum Inventory Working Group Susan Albright and Hugh Stoddard, co-chairs Launched in December 2010 Charter:  “to develop XML standards for the exchange of curriculum data for benchmarking and educational research.” (Not just for AAMC Curriculum Inventory) Reasons for Undertaking Work: Enable local curriculum management systems to upload data to a central Curriculum Inventory Identify curricular trends Support curriculum benchmarking and reform

A E B Curriculum Inventory: Events A curriculum is made up of many education and assessment events Events have metadata, descriptive information about the events F C G DH I J K L

Curriculum Inventory: Expectations Curricula may reference objectives, competencies, or learning outcomes

Curriculum Inventory: Expectations and Events An event may be intended to address one or more expectation (learning objectives, competencies, etc.) An expectation may be associated with many events G K

Curriculum Inventory: Sequence Blocks Events can be organized into blocks of instruction, called Sequence Blocks (think course/module/year/phase/clerk ship etc) A recurring event may be associated with more than one Sequence Block Sequence blocks may be associated with expectations (competencies, objectives, etc.) Sequence blocks have metadata, too A E B F C G DH I J K L A 1 2 7

Curriculum Inventory: Integration Blocks Curricular themes, or Integration Blocks, may apply to many sequence blocks, events, or expectations and describe how the curriculum is semantically integrated An Integration Block may link to many events, blocks or objects A E B F C G DH I J K L A M E O N C G SH I J P R Q 1 3 2

Curriculum Inventory: Sequence Sequence blocks can be put together to form the main structure of the curriculum (a sequence of courses, modules, phases, clerkships) A E B F C G DH I J K L A M E O N C G SH I J P R Q 1 2 3

Includes Competency objects and frameworks Curriculum Inventory: Expectations

Instructional and / or Assessment methods Resources (virtual patient, etc.) Keywords Objectives (expectations) Curriculum Inventory:

Questions? Please use Q & A option to submit questions.

AAMC Business Rules for the Curriculum Inventory

AAMC Business Rules Are designed for specific use of the MedBiquitous Curriculum Inventory Standard for use in the AAMC Curriculum Inventory Are not designed to replicate the MedBiquitous Curriculum Inventory Standard or the MedBiquitous Curriculum Inventory Implementation Guidelines ( Should be used as a major part of the CI Developers Toolkit, but not in isolation

AAMC Business Rules Core Rules (CR) Core rules must be met before additional rules for the competency framework and the curriculum inventory structure are applied. If it is detected that a submission does not meet a core rule: a) processing of the submission discontinues immediately and b) an error notification is generated that describes the invalid core rule.

AAMC Business Rules Core Rules (CR) CR01: Approved Sender Senders must be certified via AAMC’s on- boarding process before the web service will allow them to send submissions. CR02: Selected Sender – Schools must select their Sender via the School CI Portal ‘Manage Vendor’ page. o Vendor o School

AAMC Business Rules Core Rules (CR) CR03: One Inventory for One School A submission, or XML file sent to the web service, must include data for only one curriculum inventory for one school. CR04: Institution ID The must match an AAMC institution ID for a school. AAMC EIS Code

AAMC Business Rules Core Rules (CR) CR05: Submission File Size The file size cannot exceed 19mg An error will be generated if data in any field exceeds the record length Zip files accepted CR06: Data Dictionary Each field in the XML has a specific length limit, which is documented in the AAMC Business Rules. An error will be generated if data in any field exceeds its maximum allowable length

AAMC Business Rules Competency Framework and Competency Object Rules (CF) These rules are evaluated only once all core rules have been met. These rules and their application are covered in CI Developers Workshop Part II.

AAMC Business Rules Curriculum Structure Rules (CS) These rules are evaluated only once all core rules have been met.

AAMC Business Rules Curriculum Structure Rules (CS) CS01: Unique Report ID The sub-element must be unique for each of a school’s submissions. If a Curriculum Inventory Upload is rejected, the next uploaded file must not have the same ReportID. The sub-element must be alphanumeric The Report ID must not include dashes, underscores or other special characters

AAMC Business Rules Curriculum Structure Rules (CS) CS02: Previous Academic Year Submissions are checked to ensure they contain information for only the previous academic year (AY). Schools must submit data with and between July (of the previous year) and June (of the current year). Note: Dates at the sequence block or event level are not checked to ensure they are within the previous academic year. Dates specified at these levels that fall outside of the date range for a previous academic year will not cause an error or rejection.

AAMC Business Rules Curriculum Structure Rules (CS) CS03: Number of Academic Levels The number of academic levels, or phases, are defined by. Each academic level is defined as a within the tag. CS05: Academic Level Numbers sub-elements of the tag must have sequential numbers (i.e., the number attribute) starting with 1.

AAMC Business Rules Curriculum Structure Rules (CS) CS04: Academic Levels and Sequence Blocks A that is not defined within the tag cannot be referenced by a sequence block. CS06: Academic Levels and Sequence Blocks Each academic level must be referenced by at least one sequence block.

AAMC Business Rules Sequence Blocks CS07: tags must have unique identifiers (i.e., the id attribute). CS08: tags must be associated to a defined academic level using the required sub- element. CS09: The sub-element of all tags with a ClerkshipModel attribute must specify and. Document start and end dates for entire iteration period (e.g., 07/01/12 – 06/30/13). CS16: If it is provided, the duration of a sequence block must be specified as a number of days. For clerkships, 5 days = 1 week

AAMC Business Rules Curriculum Structure Rules (CS) CS10: Nested Sequence Blocks When nesting sequence blocks, must not result in circular references. See “Note about Hierarchical Conflict” section (pp ) of the Competency Framework Specification, version 1.0.

AAMC Business Rules Curriculum Structure Rules (CS) Events CS11: tags must have unique identifiers (i.e., the id attribute). CS12: Within tags, the and tags must reference a unique ID from the Curriculum Inventory Standardized Vocabulary. CS13: At a minimum, each must reference at least one or. CS14: If one or more tags are provided, one and only one must be denoted as primary. CS15: All tags must be referenced by at least one tag within a.

Questions? Please use Q & A option to submit questions.

Curriculum Inventory Process

Portal Manage Data (Home Screen) Download Verification Report After successful data upload Reject or Verify Data Manage Users Schools will be encouraged add Vendor with CI Viewer role Manage Vendors Vendor must be selected by school in Portal or data will not upload

Curriculum Inventory Process Data Submit Data ‘Portal Open’ Notification sent 08/01 System Check for (MB CI Standard, Terminology, Competencies) Verify Data Verification Report available as soon as data is successfully submitted Deadline for verifying data is 09/30 ‘Portal Closed’ Notification sent 10/01 Create Snapshot of data Reports process begins immediately

Curriculum Inventory Process Reminders and Notifications Curriculum Dean / Primary CI Admin Dean Ramifications MedAPS Reports LCME Portal Only available actions: Download Verification Report Manage Users Manage Vendors

Upload Process Vendor / Sender Dependent CMS stores data on School server CMS exports XML file File is manually uploaded by School (or Vendor if Vendor has CI Primary Admin role) CMS stores data on Vendor server Web services upload from Sender to AAMC

Questions? Please use Q & A option to submit questions.

School Portal

Curriculum Inventory Process Portal Manage Data (Home Screen) Download Verification Report After successful data upload Reject or Verify Data Manage Users (Four Roles) Curriculum Dean (Pre-populated) Primary CI Admin (Pre-Populated) CI Admin CI Viewer Schools are encouraged to add Vendor with CI Viewer role (work with vendor to create ) Manage Vendors Vendor must be selected by school in Portal or data will not upload

Web Services

Curriculum Inventory Process An activity log displays all activity – data upload attempts and status, Verification Report downloads, and data rejections and verifications.

Curriculum Inventory Process

The data and Verification Report will remain available until the upload process for the next academic year begins

CI Verification Report Institution (Program Information) All Academic Levels (Dates) All Sequence Blocks Expectations / Expectation Mapping (if provided) All Events Expectation Mapping (if available) Instructional Methods Primary Instructional Methods Assessments Events with Assessment Method(s) only

CI Verification Report

Questions? Please use Q & A option to submit questions.

Pearls A Sequence Block Week is 5 Days Clerkships are Sequence Blocks with Clerkship Model (Rotation or Longitudinal) rotation Check date / duration formats PT2H PT90M PT1H30M P13D P12M

Pearls Events must have Instructional Method OR Assessment Method(s) OR Both Use Instructional Method and Assessment Method UIDs!!!! IM013 AM012 Can have multiple Instructional Methods but one must be Primary Can have multiple Assessment Methods – no Primary Must have Duration Formative / Summative Assessment Events No Instructional Methods

Pearls Country Code: 3-character code (USA, CAN) USA

Pearls Program Code: 2-character code (MD, DO, PA) M.D Parallel Tracks Denote same cohort of students in separate tracks (should not be used to designate concurrent Academic Levels for multiple cohorts of students)

Pearls Special Characters: When data is copied from a website and pasted into Curriculum Management Systems, the special characters that make up the HTML can cause problems either in the upload process OR in reporting. In addition, these special characters sometimes cause unexpected truncation because they fill the character limit for the field. Solution: when possible, use the paste function ‘text only’ when pasting text into a CMS

Pearls Extra spaces: /CurriculumInventory/Events/Eve = 'E395'] This issue often makes it through the error checking and results in a successful upload, but the Verification Report is not generated

Issues Found in Data Analysis Issue 1: copying one course over to all other courses Issue 2: keywords that just have letters or punctuation Issue 3: synonyms Issue 4: no titles for competency objects Issue 5: Uploading only one event for all sequence blocks Issue 6: Uploading only one academic level

12345 NoSuch University School of Medicine Main Street Baltimore MD US M.D NoSuch School of Medicine Curriculum en-us The NoSuch curriculum reframes the context of health and illness to encourage students to explore a larger, integrated system.

Questions? Please use Q & A option to submit questions.

Schedule Staging available now School Portal Opens: August 1 School Portal Closes: September 30 Status sent to CI Primary Admin, Curriculum Dean, and School Dean: September 30 Data clean-up/reporting starts: November 1

Resources Curriculum Inventory Implementers’ Listserv: Competency Framework Specs and Schemas: AAMC Curriculum Inventory Developer Resources: Physician Competency Reference Set URIs: ollaborative/resource/793 ollaborative/resource/793 MedBiquitous: Curriculum Inventory Working Group: Competencies Working Group:

Questions? List of upcoming Curriculum Inventory Webinars and materials from today’s session: