Curriculum Inventory Administrators’ Group December 13, 2017 Terri Cameron, MA Director, Curriculum Programs

Agenda AAMC Medical Education Update CI 2016-2017 Upload Statistics CI System Changes for 2017-2018 Enhancements to the Verification Report Medical School Highlight: Michigan State University Review of the CI Update given at Learn Serve Lead in November November CI in Context: Transition to Residency / Boot Camps December CI in Context: Three-Year Medical Schools Next meeting: Wednesday, January 10, 1 pm ET

AAMC Medical Education Cluster Update Director, Curriculum Programs Screening in process; interviews to be scheduled soon

Review of 2016-2017 Upload Cycle 2012-2013 2013-2014 2014-2015 2015-2016 2016- 2017 90 Schools 120 Schools (82 of 90 retained; 28 new schools) 135 Schools (114 retained; 21 new schools) 141 Verified (10 new schools) 133 Verified (4 new schools) (only 1 US school has not uploaded in at least 1 cycle)   2012-2013 2013-2014 2014-2015 2015- 2016 2016- 2017 US Medical Schools 85 115 127 134 131 Canadian Medical Schools 5 2 US Osteopathic Schools (Pilot 2015-2016) n/a 3

Verification Report Table Numbering Current: Proposed: Program Expectations Mapped to PCRS 1. Program Expectations Mapped to PCRS Primary Methods of Instruction 2. Primary Methods of Instruction: Courses (LCME DCI Tables 6.0-1 and 6.0-2) Clerkships 3. Primary Methods of Instruction: Clerkships (LCME DCI Table 6.0-3) Methods of Assessment 4. Methods of Assessment: Courses (LCME DCI Table 9.0-1 and 9.0-2) Clerkship 5. Methods of Assessment: Clerkships (LCME DCI Table 9.0-3) All Instructional Methods 6. All Instructional Methods Documented in CI Upload Summative Assessment Method 7. Summative Assessment Method Formative Assessment Method 8. Formative Assessment Methods Program Expectations mapped to Events with Assessment Methods 9. Program Expectations mapped to Events with Assessment Methods Sequence Block Details 10. Sequence Block (Course/Clerkship) Details Documented in CI Upload Event Details 11. Event Details Documented in CI Upload

Overview of our New Curriculum

Our Goal: Excellence on July 1 Graduates who are… READY WILLING ABLE

Shared Discovery Curriculum Overview

CHM 5: Key Design Features of Our New Curriculum Fully integrated basic, social, disease and patient care sciences over 4 years Very early, ongoing patient care experience in authentic patient care environments Progress testing: Constant feedback on how you are doing but “tests that count” only twice a semester JustInTimeMedicine software provides road map and drives the curriculum Learning Society structure has small Scholar Groups to support student success

Reflects how doctors function in the real world Fully Integrated Basic, Social and Clinical Sciences Reflects how doctors function in the real world Not just “undergrad on steroids” Moving learning from memorization to internalization Knowledge placed into context; students know why something is important (or not) USMLE study is built on an integrated foundation “Patients are the best mnemonic”

Curriculum organized around 93 Chief Complaints and Concerns Fully Integrated Basic, Social and Clinical Sciences Curriculum organized around 93 Chief Complaints and Concerns Curriculum define the end-competencies expected of graduating students What patients complain about What doctors are concerned about Competency template for each patient complaint models the clinical problem solving process Data gathering/Problem Identification/Management Planning/Necessary Science/Complexities and Controversies

USEFULNESS EXPERIENCE Students learn while doing Early, Ongoing, Meaningful Patient Care Experience USEFULNESS Students learn while doing Students care for patients as a part of a team Work-place based application of knowledge and skills EXPERIENCE M1: Outpatient setting 2 half-days per week by Wk 9 M2: 2 and 4 Week inpatient rotations throughout Weekly simulation and clinical skills experiences for M1 and M2 students

Aimed at graduate-level student competence Progress Testing: Knowledge and Skills Aimed at graduate-level student competence Aligned with our SCRIPT competency framework Goal is continuous improvement with abundant coaching and formative assessment Focused on success by participating in the curriculum and monitoring progress

SCRIPT Competency Model Service Care of Patients (ACGME PC and ICS) Progress Testing: Aligned with Our Competency Model SCRIPT Competency Model Service Care of Patients (ACGME PC and ICS) Rationality (ACGME PBL and I) Integration (ACGME System Based Practice) Professionalism Transformation (ACGME Medical Knowledge)

Progress Testing: Suite of Learner Assessments KNOWLEDGE Basic science, social science, disease and patient care science (multiple choice/short answer/essay) SKILLS Patient care skills (observed with real and simulated patients) Knowledge applied to patient care (oral/written questions) MULTISOURCE FEEDBACK Teamwork , professionalism, responding to feedback PORTFOLIO Individual learning plans and developing personal goals Weekly assignments and work products(evidence in your portfolio)

Competencies Map to Progress Suite

JustinTimeMedicine: Curriculum Management www.justintimemedicine.com

JustInTimeMedicine Platform for curriculum delivery, organizing and integrating weekly content by patient concern Provides student calendar, preparation resources as well as content related to specific learning experiences

Provides individual student progress dashboard JustInTimeMedicine Provides individual student progress dashboard Links to digital learning resources available through MSU library Access to course syllabus, student manual, Academic Support resources Provides searchable content of 93 Chief Complaint and Concern documents Working to make downloads compatible for uploads to CI

MEDICAL SCHOOL IS CHALLENGING Learning Societies Learning is more effective when it takes place within a challenging and supportive environment MEDICAL SCHOOL IS CHALLENGING MEDICAL SCHOOL is not often characterized as “a supportive environment” Learning societies create a dedicated core teaching faculty engaging clinical, basic and social science faculty to educate and support students

Four learning societies --subdivided into scholar groups of 7 to 8 students and one lead clinician --parallel structures in two “pre-clerkship” campuses Year 1: two 2-hour meetings per week Year 2: one 2-hour meeting per week

What Happens in the Learning Societies? Inside Small Groups Outside Small Groups Debrief your clinical experiences Meet with Learning Society Fellows to: Develop individualized learning plans Work through modified Problem Based Learning cases Enhance performance through coaching and mentoring Review portfolio Access resources

A Week in the Life of the Early Clinical Experience (M1)   Mon Tues Wed Thurs Fri AM Large group Activity (e.g., TBL) Clinic Simulation Guided Independent Learning PM Learning Society Small Group Lab-2 Lab-1

A Week in the Life of the Middle Clinical Experience (M2) Example 1 Mon Tues Wed Thurs Fri Sat AM Clinical activity: IM or FM service PM Guided independent learning Clinical Team Rotational group Large group activity (e.g., TBL) Simulation or Workshop Example 2   M Tu W Th F Sa AM Clinical activity: Nutrition Self-structured learning Nutrition Rotational Small Group PM Learning Society Small Group Large group activity (e.g., TBL) Simulation or Workshop

Small group: Necessary science A Week in the Life of the Late Clinical Experience (M3&4)   M Tu W Th F Sa AM Clinical activity PM Simulation Small group: Necessary science

Michigan State University College of Human Medicine Experiences – Relationships -- Data Michigan State University College of Human Medicine Dedicated to excellence in medical education A history of pushing students to better-than-predicted performance A curriculum delivered by talented and passionate faculty and staff A curriculum driven by innovation: the CHM 5 Faculty and Students engaged in Shared Discovery

Curriculum Inventory Update Terri Cameron, MA, Director of Curriculum Programs Walter Fitz-William, MPP, Senior Program Operations Specialist

Objectives: Share AAMC Medical Education Cluster Update Provide update on 2016-2017 Curriculum Inventory Cycle Describe new CI Custom Report Three-Year Format Demonstrate CI Report Search function Discuss CI Challenges Present options for CI Support of CQI and Accreditation Report activities of CI Research Group Share goals for 2017

AAMC Medical Education Cluster Update Senior Director, Medical Education Digital Resources and Scholarship Anne Farmakidis, MPS (November 1, 2017) Anne has been working in Publications at AAMC since 2001 and is currently the Senior Director for Publications Responsible for Curriculum Inventory, MedEdPORTAL, Academic Medicine Senior Director, Medical Education Research Dorothy Andriole, MD (January 8, 2018) Currently Associate Professor of Surgery and Assistant Dean for Medical Education at Washington University School of Medicine in St. Louis, with a long-standing educational research interest in medical education outcomes and medical graduate career paths. Responsible for developing a research agenda focused on educational outcomes across the continuum of medical education and will assist the Core EPA project in ongoing program evaluation. Director, Curriculum Programs

Review of 2016-2017 Upload Cycle 2012-2013 2013-2014 2014-2015 2015-2016 2016- 2017 90 Schools 120 Schools (82 of 90 retained; 28 new schools) 135 Schools (114 retained; 21 new schools) 141 Verified (10 new schools) 133 Verified (4 new schools) (only 1 US school has not uploaded in at least 1 cycle)   2012-2013 2013-2014 2014-2015 2015- 2016 2016- 2017 US Medical Schools 85 115 127 134 131 Canadian Medical Schools 5 2 US Osteopathic Schools (Pilot 2015-2016) n/a 3

Challenging Documentation Areas: Clerkships Current documentation efforts making it difficult to analyze / report on clerkship activities Several groups will be looking at the issue and working with schools that have begun to develop innovative documentation practices Best current option: Create ‘model’ clerkship in curriculum management system and document each event (no event summaries), with metadata Tracks Working with schools that documented tracks to develop Best Practices Integration Blocks Streamers such as Medical Humanities, Medical Informatics, Genetics, Geriatrics Should these streamers include Expectations?

Upload Issues: Curriculum Detail What does it mean to upload an entire curriculum? Required Courses Required Events for all Required Courses Instructional Methods for all ‘Instructional Events’ Include Assessment Methods in ‘Instructional Events’ to show how content in that event WILL be assessed in course assessments Assessment Methods for all ‘Assessment Events’ Shows how content across the course is assessed AND populates the ‘Number of Exams’ column in the Course Assessment Methods Table Resources Documents use of important resources such as Standardized Patients, Virtual Patients Is it possible to upload every detail of a curriculum? No! But the more detail included, the better reporting at the local level and the national level

Upload Issues: Curriculum Detail How consistent is curriculum data across institutions? Do Academic Levels matter? What is a course? What is a clerkship? To what level are clerkship activities documented? Is every event in every clerkship documented? How are multiple instructional methods documented/reported? What does it mean to link an assessment method to an ‘Instructional Event’? Are we capturing every possible version of a term or concept? What curriculum data is captured in other systems and does not get included in a school’s curriculum upload?

Upload / Verification Issues: Verification Report Purpose: Display tables that show details of uploaded data in a format Curriculum Deans/Admins can read Show how CI data could populate LCME Data Collection Instrument Assessment Methods Tables (more later) Instructional Methods Tables Mapping CI IMs to LCME IMs (Crosswalk) All Instructional Methods (Add additional columns and show all IMs, not just Primary; will be investigating how to deal with Primary IM in LCME DCI tables

Academic Level (1 to 4 only) Required Sequence Blocks: 2016-2017 Required SB Integrated Rotation Course Academic Level (1 to 4 only) Schools Sequence Blocks 1 2 120 1250 9 56 109 981 3 17 57 79 378 80 340 4 10 51 181 68 501

25th 1262 Med. 995 Avg. 949 75th 705

2013 2014 2015 2016 2017 Lecture 54% 52% 49%

Data Validation Efforts Do the sequence blocks look about the same? The same number of events or referencing the same events? If the participant is only providing one academic level, did they only provide one last season as well? Are there fewer academic levels this year than previous years? Are the start and end dates for the sequence blocks too far apart? Are titles and descriptions approaching the 4000 character limit? Do any sequence blocks have an unusual number of events? Are there repeating expectation titles?

New Search Query / Content Report Update The Content Search examines terms provided by participants in the sequence block title, description, expectations, and the event title, description, keywords, and expectations. It does this by breaking each entry in each field into individual words, which allows the report author to examine terms independently, finding alternative spellings and conjugations, and to connect those individual words using Boolean logic, expanding the potential captured events and sequence blocks for our queries. Currently adds significant time to each query, but yields much better results.

New Custom Content Report Template: Three year view of: Participation/Documentation of Content Content Documentation by Academic Level Instructional Methods Assessment Methods Resources Competencies

Enhanced Search Results for Online CI Reports (Searches keywords, as well as title text)

Assessment Tables in Verification Report Two tables meant to replicate LCME DCI tables, including ‘summarizing’ assessment methods into an abbreviated list of LCME terms (see Crosswalk at www.aamc.org/cir under Training and Resources: Methods of Assessment – Courses Methods of Assessment – Clerkships Two tables to show all Assessment Methods listed in CI Standardized Vocabulary document: Summative Assessment Methods Formative Assessment Methods

Assessment Tables in Verification Report Issue in previous Verification Reports: double-counting of Assessment Events in Summative Assessment Methods Solution: System Change for 2016-2017: Changed tallies of assessment methods in the Summative and Formative Assessment tables so events are counted in one or the other. Basically, Methods of Assessment Table counts only Summative AMs in Assessment Events and Summative Assessment Table does NOT count Assessment Events

Assessment Tables in Verification Report Unexpected Consequences: Schools that were not experiencing the ‘double-counting’ issue or had adjusted data entry to avoid the issue are not seeing all of their Summative Assessment Methods counted Miscommunication resulted in this change being made to the Formative Assessment Table in error Since Formative Assessment tends not to be Summative, schools are reporting that the Formative Assessment table is not being populated as expected. We are investigating both of these issues and will be working with you as you find problems. We apologize for the confusion.

Verification Report User Guide Methods of Assessment: This table lists all non-Clerkship SBs and shows how Summative Assessment methods linked to Events in those SBs are mapped to the list of LCME DCI Educational Methods (see Crosswalk). The Total Number of Exams is calculated by totaling the number of Events for that SB that have Summative Assessment Methods in Assessment Events (Events that have Assessment Methods but do not have Instructional Methods). The Formative Column is checked If at least one event contains one or more Assessment Methods tagged as “Formative.”

Verification Report User Guide Methods of Assessment: Clerkships: Sequence Blocks will only be included in this table if they are tagged as clerkships. Summative Assessment methods linked to Assessment Events in those SBs are mapped to the list of LCME DCI Assessment Methods (see Crosswalk). Academic Level is based on the AL referenced by each SB in the CI Upload.

Verification Report User Guide Summative Assessment Methods: This table shows the number of Events* linked to CI Assessment Methods tagged as Summative. For Events to be counted in this table, both an Instructional Method AND an Assessment Method must be linked. Assessment Events are not shown in this Table.

Verification Report User Guide Formative Assessment Methods: This table shows the number of Events* linked to CI Assessment Methods tagged as Formative. For Events to be counted in this table, both an Instructional Method AND an Assessment Method must be linked. Assessment Events are not shown in this Table.

Adding Additional Tables to Verification Report Reviewing DCI to determine what additional tables to include Agenda item for next CI Advisory Board Meeting Reviewing LCME AQ II to determine what additional tables we could include in VR Issue: Generally uploading curriculum data does not always result in the same data that is provided when a specific question is asked

LCME AQII Data CI Data

LCME AQII Data CI Data

Required Rotations with the word “Ambulatory” LCME AQII Data CI Data Required Rotations with the word “Ambulatory” TITLE Schools Ambulatory - Fam & Comm Med 1 Ambulatory - Internal Medicine Ambulatory Internal Medicine Ambulatory Internal Medicine Clerkship Ambulatory Medicine 2 Ambulatory Medicine Core Clerkship Ambulatory Rotation Adult Ambulatory Care MED 652 Ambulatory Medicine MED 673 Ambulatory Internal Medicine in Economically Underserved Areas of Mississippi Medicine Ambulatory Clerkship Year 4 - Ambulatory Combined Ambulatory Medicine and Pediatrics Clerkship (CAMPC) 2016 OB/GYN 659 Ambulatory Care Pediatrics Ambulatory Clerkship AMFMC_1516: Ambulatory Internal / Family Medicine Clerkship (Class of 2017) Family Medicine Ambulatory Clerkship PED 651 Pediatric Ambulatory Care Family Medicine and Ambulatory Care Clerkship IDC-IDC301-Amb Med-Primary Care Clerkship-Ambulatory Medicine [4 weeks/4 credits]

LCME AQII Data CI Data

LCME AQII Data CI Data Ehr Ehrs Emr Or 1. Electronic near 2. Heal Healh Healt Healtcare Healtchare Health Health3Health4 Health6 Healtha Med Medi Medical 3. Rec Reco Recor Record Record2012 Recorded Recorder Recorders Recording Recordings Records Recored 58 Schools mention HER/EMR in their curriculum management system

LCME AQII Data CI Data 12 schools indicate night call as part of their curricular maps

LCME DCI LCME AQII Data CI Data 7.2 ORGAN SYSTEMS/LIFE CYCLE/PRIMARY CARE/PREVENTION/WELLNESS/ SYMPTOMS/SIGNS/DIFFERENTIAL DIAGNOSIS, TREATMENT PLANNING, IMPACT OF BEHAVIORAL AND SOCIAL FACTORS   NARRATIVE RESPONSE Describe the location(s) in the pre-clerkship and clinical curriculum in which objectives related to the subjects listed below are taught and assessed. Refer to the Supporting Data and Documentation for Standard 6 in the responses. Normal human development Adolescent medicine Geriatrics Continuity of care End-of-life care CI Data Only 49 schools feature “End” “Life” “Care” documented in the expectations, 72 in any field.

Accomplishments this year Enhanced Data Validations Implemented More Complex Search Query for Custom Content Reports Revised CI Custom Content Report Template Review of CI Website and Reports by Board CI Verification Report Manual Reorganization of CI Reports by Name Enhanced Search Results for Online CI Reports

Planned changes for 2017-2018: Increase the length of the Academic Level Title New AAMC Business Rule: Competency Object titles will be required. New AAMC Business Rule: Event ID references cannot have spaces. New AAMC Business Rule: The year value for date will need to be greater within 6 years of either direction of the report start and end dates. New AAMC Business Rule: If Resources are documented, UIDs will be required (as they currently are for Instructional and Assessment methods). New AAMC Business Rule: Special characters will not be accepted (the Business Rule will provide more information). CompetencyObjects that are deeply placed in the competency framework will appear in the verification report. The Curriculum Inventory will make use of the new System Access Management system at the AAMC, an important step in rewarding participation.

Documenting EPAs in the CI No requirement to document EPA’s For research purposes only Best Practice for Documentation: CoreEPA01_Gather a history and perform a physical examination Timeline: Document for upcoming AY: 2017-2018 Upload August 1, 2018

PCRS Update / Competency-Based Medical Education Terminology Group Physician Competency Reference Set (PCRS) Aggregation Tool used to report on competency coverage across medical schools Needs updated in light of update of IPEC competencies Is PCRS a break in the continuum because GME focuses on ACGME, or is PCRS necessary because there is no common set of ACGME sub-competencies below the 6 competencies? AAMC Common Language for CBME Task Force Goals Develop a common taxonomy Discuss competencies across the continuum -- PCRS / ACGME We are waiting to see the outcome of the AAMC CBME Task Force Ramifications for research re: competencies in UGME?

National Keyword Terminology Would not be required and could be considered a ‘base’ list that schools could build on Pros: Would improve reporting locally and nationally Could save schools time and effort in developing lists Issues: How long should the list be? Who develops the list? Who maintains the list? Will the list affect medical education? Solution: Research / Discussion with Curriculum Deans

CI for CQI and Accreditation (CICA) CICA will resume meetings in November. Priorities: Determining specifically each DCI Standard/Question CI could support Developing CI Charts that show national responses to those questions for benchmarking and CQI purposes Adding as many of these charts as possible to the Verification Report May take a couple of years to add all the tables Reviewing options for other formats (spreadsheets?)

Curriculum Inventory Research Group October 4 Retreat resulted in kick-off of three research projects: What is a clerkship? (Brian Mavis and Tanya Horsley) What is replacing lectures in medical education? (Colleen Grochowski and Carolyn DuFault) What is the life expectancy of curriculum? What are facilitators and barriers to reform? (Brian Mavis and Jeanne Farnan) Dorothy Andriole will be taking on leadership of this group as she begins her role as Senior Director for Medical Education Research in January

Goals for 2017-2018 Implementation of remainder of CIAB Website / Report Recommendations Enhanced Verification Report (new DCI tables, add columns to ‘All Instructional Methods table to show additional IMs, and add LCME/CI Terminology Crosswalk to Report) Improved Search Options on CI Report page Guide for Documenting Curricula in CI for CQI and Accreditation CQI and Accreditation ‘Manual’ Pilot CI Report Login Screen Discipline Reports Documentation of AAMC Core EPAs CI Publication(s) from CI Research Group Work with MedBiquitous to Develop National Keyword Terminology?

CI in Context: November 2017 Transitions to Residency in US Medical Schools Bridget O’Brien, PhD, University of California – San Francisco

CI in Context: December 2017 Three-Year Medical Schools Joan Cangiarella, MD, New York University Medical College

Proposals for Spring/Summer Education Meetings CGEA: Is the lecture being replaced? (Carolyn DuFault, Washington University; Sarah Lang, Indiana University; Dean Parmelee, Wright State) Medical Education Research Toolkit (Brian Mavis, Michigan State; Carolyn DuFault, Washington University; Jeanne Farnam, University of Chicago) Simulation Center Use at Medical Schools (Gerald Wickham, University of Illinois – Peoria; Dwyer, Anthony Michael, University of Illinois – Peoria; Mary Aiello, Southern Illinois; MaryKay Smith, Michigan State University) The Next Generation of Medical School Curriculum: Exploring Curricular Innovation and Change (Brian Mavis, Michigan State; Elizabeth Baker, Rush Medical Giulia Bonaminio,University of Kansas; Abbas Hyderi, University of Illinois – Chicago; Dianne Wagner, Michigan State University)

Proposals for Spring/Summer Education Meetings SGEA: Best Practices for Reporting on Clerkship Content (Joshua Thornhill, University of South Carolian; Josev Lopez, Texas Tech; Julie Bridges, Eastern Virginia) Is the lecture being replaced? (Colleen Grochowski, Duke; Tanis Hog, Texas Tech; Sarah Wood, Florida Atlantic)

Proposals for Spring/Summer Education Meetings NEGEA: Best Practices for Reporting on Clerkship Content (Joshua Thornhill, University of South Carolian; Josev Lopez, Texas Tech; Julie Bridges, Eastern Virginia) Is the lecture being replaced? (Nancy Hueppchen, Johns Hopkins; Rebecca Keller, Albany; Brian Reid, Dartmouth) WGEA: Best Practices for Reporting on Clerkship Content (Irina Russell, Stanford; Tomo Ito, Oregon Health Sciences University; Cinda Stone, Arizona-Phoenix)

Next meeting: Wednesday, January 10, 1 pm ET (Second Wednesday of each month, 1 pm ET) Registration Links posted in Training and Resources section of www.aamc.org/cir Please send agenda items to ci@aamc.org