1. Curriculum Inventory Administrators’ Group December 13, 2017
Terri Cameron, MA
Director, Curriculum Programs

2. Agenda AAMC Medical Education Update CI 2016-2017 Upload Statistics
CI System Changes for
Enhancements to the Verification Report
Medical School Highlight: Michigan State University
Review of the CI Update given at Learn Serve Lead in November
November CI in Context: Transition to Residency / Boot Camps
December CI in Context: Three-Year Medical Schools
Next meeting: Wednesday, January 10, 1 pm ET

3. AAMC Medical Education Cluster Update
Director, Curriculum Programs
Screening in process; interviews to be scheduled soon

4. Review of 2016-2017 Upload Cycle
90 Schools
120 Schools (82 of 90 retained; 28 new schools)
135 Schools (114 retained; 21 new schools)
141 Verified
(10 new schools)
133 Verified (4 new schools)
(only 1 US school has not uploaded in at least 1 cycle)
US Medical Schools 85
115
127
134
131
Canadian Medical Schools 5 2
US Osteopathic Schools (Pilot )
n/a 3

5. Verification Report Table Numbering
Current:
Proposed:
Program Expectations Mapped to PCRS
1. Program Expectations Mapped to PCRS
Primary Methods of Instruction
2. Primary Methods of Instruction: Courses (LCME DCI Tables and 6.0-2)
Clerkships
3. Primary Methods of Instruction: Clerkships (LCME DCI Table 6.0-3)
Methods of Assessment
4. Methods of Assessment: Courses (LCME DCI Table and 9.0-2)
Clerkship
5. Methods of Assessment: Clerkships (LCME DCI Table 9.0-3)
All Instructional Methods
6. All Instructional Methods Documented in CI Upload
Summative Assessment Method
7. Summative Assessment Method
Formative Assessment Method
8. Formative Assessment Methods
Program Expectations mapped to Events with Assessment Methods
9. Program Expectations mapped to Events with Assessment Methods
Sequence Block Details
10. Sequence Block (Course/Clerkship) Details Documented in CI Upload
Event Details
11. Event Details Documented in CI Upload

6. Overview
of our
New Curriculum

7. Our Goal: Excellence on July 1
Graduates who are…
READY
WILLING
ABLE

8. Shared Discovery Curriculum Overview

9. CHM 5: Key Design Features of Our New Curriculum
Fully integrated basic, social, disease and patient care sciences over 4 years
Very early, ongoing patient care experience in authentic patient care environments
Progress testing: Constant feedback on how you are doing but “tests that count” only twice a semester
JustInTimeMedicine software provides road map and drives the curriculum
Learning Society structure has small Scholar Groups to support student success

10. Reflects how doctors function in the real world
Fully Integrated Basic, Social and Clinical Sciences
Reflects how doctors function in the real world
Not just “undergrad on steroids”
Moving learning from memorization to internalization
Knowledge placed into context; students know why something is important (or not)
USMLE study is built on an integrated foundation
“Patients are the best mnemonic”

11. Curriculum organized around 93 Chief Complaints and Concerns
Fully Integrated Basic, Social and Clinical Sciences
Curriculum organized around 93 Chief Complaints and Concerns
Curriculum define the end-competencies expected of graduating students
What patients complain about
What doctors are concerned about
Competency template for each patient complaint models the clinical problem solving process
Data gathering/Problem Identification/Management Planning/Necessary Science/Complexities and Controversies

12. USEFULNESS EXPERIENCE Students learn while doing
Early, Ongoing, Meaningful Patient Care Experience
USEFULNESS
Students learn while doing
Students care for patients as a part of a team
Work-place based application of knowledge and skills
EXPERIENCE
M1: Outpatient setting 2 half-days per week by Wk 9
M2: 2 and 4 Week inpatient rotations throughout
Weekly simulation and clinical skills experiences for M1 and M2 students

13. Aimed at graduate-level student competence
Progress Testing: Knowledge and Skills
Aimed at graduate-level student competence
Aligned with our SCRIPT competency framework
Goal is continuous improvement with
abundant coaching and formative assessment
Focused on success by participating in the
curriculum and monitoring progress

14. SCRIPT Competency Model Service Care of Patients (ACGME PC and ICS)
Progress Testing: Aligned with Our Competency Model
SCRIPT Competency Model
Service
Care of Patients (ACGME PC and ICS)
Rationality (ACGME PBL and I)
Integration (ACGME System Based Practice)
Professionalism
Transformation (ACGME Medical Knowledge)

15. Progress Testing: Suite of Learner Assessments
KNOWLEDGE
Basic science, social science, disease and patient care science (multiple choice/short answer/essay)
SKILLS
Patient care skills (observed with real and simulated patients)
Knowledge applied to patient care (oral/written questions)
MULTISOURCE FEEDBACK
Teamwork , professionalism, responding to feedback
PORTFOLIO
Individual learning plans and developing personal goals
Weekly assignments and work products(evidence in your portfolio)

16. Competencies Map to Progress Suite

17. JustinTimeMedicine: Curriculum Management

18. JustInTimeMedicine
Platform for curriculum delivery, organizing and integrating weekly content by patient concern
Provides student calendar, preparation resources as well as content related to specific learning experiences

19. Provides individual student progress dashboard
JustInTimeMedicine
Provides individual student progress dashboard
Links to digital learning resources available through MSU library
Access to course syllabus, student manual, Academic Support resources
Provides searchable content of 93 Chief Complaint and Concern documents
Working to make downloads compatible for uploads to CI

20. MEDICAL SCHOOL IS CHALLENGING
Learning Societies
Learning is more effective when it takes place within a challenging and supportive environment
MEDICAL SCHOOL IS CHALLENGING
MEDICAL SCHOOL is not often characterized as
“a supportive environment”
Learning societies create a dedicated core teaching faculty engaging clinical, basic and social science faculty to educate and support students

21. Four learning societies --subdivided into scholar groups of
7 to 8 students and one lead clinician
--parallel structures in two “pre-clerkship”
campuses
Year 1: two 2-hour meetings per week
Year 2: one 2-hour meeting per week

22. What Happens in the Learning Societies?
Inside Small Groups
Outside Small Groups
Debrief your clinical experiences
Meet with Learning Society Fellows to:
Develop individualized learning plans
Work through modified Problem Based Learning cases
Enhance performance through coaching and mentoring
Review portfolio
Access resources

24. A Week in the Life of the Early Clinical Experience (M1)
Mon
Tues
Wed
Thurs
Fri AM
Large group Activity
(e.g., TBL)
Clinic
Simulation
Guided Independent Learning PM
Learning Society Small Group
Lab-2
Lab-1

25. A Week in the Life of the Middle Clinical Experience (M2)
Example 1
Mon
Tues
Wed
Thurs
Fri
Sat AM
Clinical activity: IM or FM service PM
Guided independent learning
Clinical Team
Rotational group
Large group activity
(e.g., TBL)
Simulation or Workshop
Example 2 M Tu W Th F Sa AM
Clinical activity: Nutrition
Self-structured learning
Nutrition
Rotational Small Group PM
Learning Society Small Group
Large group activity
(e.g., TBL)
Simulation or Workshop

26. Small group: Necessary science
A Week in the Life of the Late Clinical Experience (M3&4) M Tu W Th F Sa AM
Clinical activity PM
Simulation
Small group: Necessary science

27. Michigan State University College of Human Medicine
Experiences – Relationships -- Data
Michigan State University College of Human Medicine
Dedicated to excellence in medical education
A history of pushing students to better-than-predicted performance
A curriculum delivered by talented and passionate faculty and staff
A curriculum driven by innovation: the CHM 5
Faculty and Students engaged in Shared Discovery

29. Curriculum Inventory Update
Terri Cameron, MA, Director of Curriculum Programs
Walter Fitz-William, MPP, Senior Program Operations Specialist

30. Objectives: Share AAMC Medical Education Cluster Update
Provide update on Curriculum Inventory Cycle
Describe new CI Custom Report Three-Year Format
Demonstrate CI Report Search function
Discuss CI Challenges
Present options for CI Support of CQI and Accreditation
Report activities of CI Research Group
Share goals for 2017

31. AAMC Medical Education Cluster Update
Senior Director, Medical Education Digital Resources and Scholarship
Anne Farmakidis, MPS (November 1, 2017)
Anne has been working in Publications at AAMC since 2001 and is currently the Senior Director for Publications
Responsible for Curriculum Inventory, MedEdPORTAL, Academic Medicine
Senior Director, Medical Education Research
Dorothy Andriole, MD (January 8, 2018)
Currently Associate Professor of Surgery and Assistant Dean for Medical Education at Washington University School of Medicine in St. Louis, with a long-standing educational research interest in medical education outcomes and medical graduate career paths.
Responsible for developing a research agenda focused on educational outcomes across the continuum of medical education and will assist the Core EPA project in ongoing program evaluation.
Director, Curriculum Programs

32. Review of 2016-2017 Upload Cycle
90 Schools
120 Schools (82 of 90 retained; 28 new schools)
135 Schools (114 retained; 21 new schools)
141 Verified
(10 new schools)
133 Verified (4 new schools)
(only 1 US school has not uploaded in at least 1 cycle)
US Medical Schools 85
115
127
134
131
Canadian Medical Schools 5 2
US Osteopathic Schools (Pilot )
n/a 3

33. Challenging Documentation Areas:
Clerkships
Current documentation efforts making it difficult to analyze / report on clerkship activities
Several groups will be looking at the issue and working with schools that have begun to develop innovative documentation practices
Best current option: Create ‘model’ clerkship in curriculum management system and document each event (no event summaries), with metadata
Tracks
Working with schools that documented tracks to develop Best Practices
Integration Blocks
Streamers such as Medical Humanities, Medical Informatics, Genetics, Geriatrics
Should these streamers include Expectations?

34. Upload Issues: Curriculum Detail
What does it mean to upload an entire curriculum?
Required Courses
Required Events for all Required Courses
Instructional Methods for all ‘Instructional Events’
Include Assessment Methods in ‘Instructional Events’ to show how content in that event WILL be assessed in course assessments
Assessment Methods for all ‘Assessment Events’
Shows how content across the course is assessed AND populates the ‘Number of Exams’ column in the Course Assessment Methods Table
Resources
Documents use of important resources such as Standardized Patients, Virtual Patients
Is it possible to upload every detail of a curriculum?
No! But the more detail included, the better reporting at the local level and the national level

35. Upload Issues: Curriculum Detail
How consistent is curriculum data across institutions?
Do Academic Levels matter?
What is a course?
What is a clerkship?
To what level are clerkship activities documented?
Is every event in every clerkship documented?
How are multiple instructional methods documented/reported?
What does it mean to link an assessment method to an ‘Instructional Event’?
Are we capturing every possible version of a term or concept?
What curriculum data is captured in other systems and does not get included in a school’s curriculum upload?

36. Upload / Verification Issues:
Verification Report
Purpose:
Display tables that show details of uploaded data in a format Curriculum Deans/Admins can read
Show how CI data could populate LCME Data Collection Instrument
Assessment Methods Tables (more later)
Instructional Methods Tables
Mapping CI IMs to LCME IMs (Crosswalk)
All Instructional Methods (Add additional columns and show all IMs, not just Primary; will be investigating how to deal with Primary IM in LCME DCI tables

38. Academic Level (1 to 4 only)
Required Sequence Blocks:
Required SB
Integrated
Rotation
Course
Academic Level (1 to 4 only)
Schools
Sequence Blocks 1 2
120
1250 9 56
109
981 3 17 57 79
378 80
340 4 10 51
181 68
501

39. 25th
Med. 995
Avg
75th 705

43. 2013
2014
2015
2016
2017
Lecture
54%
52%
49%

44. Data Validation Efforts
Do the sequence blocks look about the same? The same number of events or referencing the same events?
If the participant is only providing one academic level, did they only provide one last season as well? Are there fewer academic levels this year than previous years?
Are the start and end dates for the sequence blocks too far apart?
Are titles and descriptions approaching the 4000 character limit?
Do any sequence blocks have an unusual number of events?
Are there repeating expectation titles?

45. New Search Query / Content Report Update
The Content Search examines terms provided by participants in the sequence block title, description, expectations, and the event title, description, keywords, and expectations.
It does this by breaking each entry in each field into individual words, which allows the report author to examine terms independently, finding alternative spellings and conjugations, and to connect those individual words using Boolean logic, expanding the potential captured events and sequence blocks for our queries.
Currently adds significant time to each query, but yields much better results.

46. New Custom Content Report Template:
Three year view of:
Participation/Documentation of Content
Content Documentation by Academic Level
Instructional Methods
Assessment Methods
Resources
Competencies

47. Enhanced Search Results for Online CI Reports
(Searches keywords, as well as title text)

48. Assessment Tables in Verification Report
Two tables meant to replicate LCME DCI tables, including ‘summarizing’ assessment methods into an abbreviated list of LCME terms (see Crosswalk at under Training and Resources:
Methods of Assessment – Courses
Methods of Assessment – Clerkships
Two tables to show all Assessment Methods listed in CI Standardized Vocabulary document:
Summative Assessment Methods
Formative Assessment Methods

49. Assessment Tables in Verification Report
Issue in previous Verification Reports: double-counting of Assessment Events in Summative Assessment Methods
Solution: System Change for :
Changed tallies of assessment methods in the Summative and Formative Assessment tables so events are counted in one or the other.
Basically, Methods of Assessment Table counts only Summative AMs in Assessment Events and Summative Assessment Table does NOT count Assessment Events

50. Assessment Tables in Verification Report
Unexpected Consequences:
Schools that were not experiencing the ‘double-counting’ issue or had adjusted data entry to avoid the issue are not seeing all of their Summative Assessment Methods counted
Miscommunication resulted in this change being made to the Formative Assessment Table in error
Since Formative Assessment tends not to be Summative, schools are reporting that the Formative Assessment table is not being populated as expected.
We are investigating both of these issues and will be working with you as you find problems.
We apologize for the confusion.

51. Verification Report User Guide
Methods of Assessment: This table lists all non-Clerkship SBs and shows how Summative Assessment methods linked to Events in those SBs are mapped to the list of LCME DCI Educational Methods (see Crosswalk).
The Total Number of Exams is calculated by totaling the number of Events for that SB that have Summative Assessment Methods in Assessment Events (Events that have Assessment Methods but do not have Instructional Methods).
The Formative Column is checked If at least one event contains one or more Assessment Methods tagged as “Formative.”

52. Verification Report User Guide
Methods of Assessment: Clerkships:
Sequence Blocks will only be included in this table if they are tagged as clerkships. Summative Assessment methods linked to Assessment Events in those SBs are mapped to the list of LCME DCI Assessment Methods (see Crosswalk).
Academic Level is based on the AL referenced by each SB in the CI Upload.

53. Verification Report User Guide
Summative Assessment Methods:
This table shows the number of Events* linked to CI Assessment Methods tagged as Summative.
For Events to be counted in this table, both an Instructional Method AND an Assessment Method must be linked. Assessment Events are not shown in this Table.

54. Verification Report User Guide
Formative Assessment Methods:
This table shows the number of Events* linked to CI Assessment Methods tagged as Formative.
For Events to be counted in this table, both an Instructional Method AND an Assessment Method must be linked. Assessment Events are not shown in this Table.

55. Adding Additional Tables to Verification Report
Reviewing DCI to determine what additional tables to include
Agenda item for next CI Advisory Board Meeting
Reviewing LCME AQ II to determine what additional tables we could include in VR
Issue: Generally uploading curriculum data does not always result in the same data that is provided when a specific question is asked

56. LCME AQII Data
CI Data

57. LCME AQII Data
CI Data

58. Required Rotations with the word “Ambulatory”
LCME AQII Data
CI Data
Required Rotations with the word “Ambulatory”
TITLE
Schools
Ambulatory - Fam & Comm Med 1
Ambulatory - Internal Medicine
Ambulatory Internal Medicine
Ambulatory Internal Medicine Clerkship
Ambulatory Medicine 2
Ambulatory Medicine Core Clerkship
Ambulatory Rotation
Adult Ambulatory Care
MED 652 Ambulatory Medicine
MED 673 Ambulatory Internal Medicine in Economically Underserved Areas of Mississippi
Medicine Ambulatory Clerkship
Year 4 - Ambulatory
Combined Ambulatory Medicine and Pediatrics Clerkship (CAMPC) 2016
OB/GYN 659 Ambulatory Care
Pediatrics Ambulatory Clerkship
AMFMC_1516: Ambulatory Internal / Family Medicine Clerkship (Class of 2017)
Family Medicine Ambulatory Clerkship
PED 651 Pediatric Ambulatory Care
Family Medicine and Ambulatory Care Clerkship
IDC-IDC301-Amb Med-Primary Care Clerkship-Ambulatory Medicine [4 weeks/4 credits]

59. LCME AQII Data
CI Data

60. LCME AQII Data CI Data Ehr Ehrs Emr Or 1. Electronic near
2. Heal Healh Healt Healtcare Healtchare Health Health3Health4 Health6 Healtha Med Medi Medical
3. Rec Reco Recor Record Record2012 Recorded Recorder Recorders Recording Recordings Records Recored
58 Schools mention HER/EMR in their curriculum management system

61. LCME AQII Data
CI Data
12 schools indicate night call as part of their curricular maps

62. LCME DCI LCME AQII Data CI Data
7.2 ORGAN SYSTEMS/LIFE CYCLE/PRIMARY CARE/PREVENTION/WELLNESS/ SYMPTOMS/SIGNS/DIFFERENTIAL DIAGNOSIS, TREATMENT PLANNING, IMPACT OF BEHAVIORAL AND SOCIAL FACTORS
NARRATIVE RESPONSE
Describe the location(s) in the pre-clerkship and clinical curriculum in which objectives related to the subjects listed below are taught and assessed. Refer to the Supporting Data and Documentation for Standard 6 in the responses.
Normal human development
Adolescent medicine
Geriatrics
Continuity of care
End-of-life care
CI Data
Only 49 schools feature “End” “Life” “Care” documented in the expectations, 72 in any field.

63. Accomplishments this year
Enhanced Data Validations
Implemented More Complex Search Query for Custom Content Reports
Revised CI Custom Content Report Template
Review of CI Website and Reports by Board
CI Verification Report Manual
Reorganization of CI Reports by Name
Enhanced Search Results for Online CI Reports

64. Planned changes for :
Increase the length of the Academic Level Title
New AAMC Business Rule: Competency Object titles will be required.
New AAMC Business Rule: Event ID references cannot have spaces.
New AAMC Business Rule: The year value for date will need to be greater within 6 years of either direction of the report start and end dates.
New AAMC Business Rule: If Resources are documented, UIDs will be required (as they currently are for Instructional and Assessment methods).
New AAMC Business Rule: Special characters will not be accepted (the Business Rule will provide more information).
CompetencyObjects that are deeply placed in the competency framework will appear in the verification report.
The Curriculum Inventory will make use of the new System Access Management system at the AAMC, an important step in rewarding participation.

65. Documenting EPAs in the CI
No requirement to document EPA’s
For research purposes only
Best Practice for Documentation:
CoreEPA01_Gather a history and perform a physical examination
Timeline:
Document for upcoming AY:
Upload August 1, 2018

66. PCRS Update / Competency-Based Medical Education Terminology Group
Physician Competency Reference Set (PCRS)
Aggregation Tool used to report on competency coverage across medical schools
Needs updated in light of update of IPEC competencies
Is PCRS a break in the continuum because GME focuses on ACGME, or is PCRS necessary because there is no common set of ACGME sub-competencies below the 6 competencies?
AAMC Common Language for CBME Task Force
Goals
Develop a common taxonomy
Discuss competencies across the continuum -- PCRS / ACGME
We are waiting to see the outcome of the AAMC CBME Task Force
Ramifications for research re: competencies in UGME?

67. National Keyword Terminology
Would not be required and could be considered a ‘base’ list that schools could build on
Pros:
Would improve reporting locally and nationally
Could save schools time and effort in developing lists
Issues:
How long should the list be?
Who develops the list?
Who maintains the list?
Will the list affect medical education?
Solution:
Research / Discussion with Curriculum Deans

68. CI for CQI and Accreditation (CICA)
CICA will resume meetings in November.
Priorities:
Determining specifically each DCI Standard/Question CI could support
Developing CI Charts that show national responses to those questions for benchmarking and CQI purposes
Adding as many of these charts as possible to the Verification Report
May take a couple of years to add all the tables
Reviewing options for other formats (spreadsheets?)

69. Curriculum Inventory Research Group
October 4 Retreat resulted in kick-off of three research projects:
What is a clerkship? (Brian Mavis and Tanya Horsley)
What is replacing lectures in medical education? (Colleen Grochowski and Carolyn DuFault)
What is the life expectancy of curriculum? What are facilitators and barriers to reform? (Brian Mavis and Jeanne Farnan)
Dorothy Andriole will be taking on leadership of this group as she begins her role as Senior Director for Medical Education Research in January

70. Goals for
Implementation of remainder of CIAB Website / Report Recommendations
Enhanced Verification Report (new DCI tables, add columns to ‘All Instructional Methods table to show additional IMs, and add LCME/CI Terminology Crosswalk to Report)
Improved Search Options on CI Report page
Guide for Documenting Curricula in CI for CQI and Accreditation
CQI and Accreditation ‘Manual’ Pilot
CI Report Login Screen
Discipline Reports
Documentation of AAMC Core EPAs
CI Publication(s) from CI Research Group
Work with MedBiquitous to Develop National Keyword Terminology?

72. CI in Context: November 2017
Transitions to Residency in US Medical Schools
Bridget O’Brien, PhD, University of California – San Francisco

73. CI in Context: December 2017
Three-Year Medical Schools
Joan Cangiarella, MD, New York University Medical College

74. Proposals for Spring/Summer Education Meetings
CGEA:
Is the lecture being replaced? (Carolyn DuFault, Washington University; Sarah Lang, Indiana University; Dean Parmelee, Wright State)
Medical Education Research Toolkit (Brian Mavis, Michigan State; Carolyn DuFault, Washington University; Jeanne Farnam, University of Chicago)
Simulation Center Use at Medical Schools (Gerald Wickham, University of Illinois – Peoria; Dwyer, Anthony Michael, University of Illinois – Peoria; Mary Aiello, Southern Illinois; MaryKay Smith, Michigan State University)
The Next Generation of Medical School Curriculum: Exploring Curricular Innovation and Change (Brian Mavis, Michigan State; Elizabeth Baker, Rush Medical Giulia Bonaminio,University of Kansas; Abbas Hyderi, University of Illinois – Chicago; Dianne Wagner, Michigan State University)

75. Proposals for Spring/Summer Education Meetings
SGEA:
Best Practices for Reporting on Clerkship Content (Joshua Thornhill, University of South Carolian; Josev Lopez, Texas Tech; Julie Bridges, Eastern Virginia)
Is the lecture being replaced? (Colleen Grochowski, Duke; Tanis Hog, Texas Tech; Sarah Wood, Florida Atlantic)

76. Proposals for Spring/Summer Education Meetings
NEGEA:
Best Practices for Reporting on Clerkship Content (Joshua Thornhill, University of South Carolian; Josev Lopez, Texas Tech; Julie Bridges, Eastern Virginia)
Is the lecture being replaced? (Nancy Hueppchen, Johns Hopkins; Rebecca Keller, Albany; Brian Reid, Dartmouth)
WGEA:
Best Practices for Reporting on Clerkship Content (Irina Russell, Stanford; Tomo Ito, Oregon Health Sciences University; Cinda Stone, Arizona-Phoenix)

77. Next meeting: Wednesday, January 10, 1 pm ET
(Second Wednesday of each month, 1 pm ET)
Registration Links posted in Training and Resources section of
Please send agenda items to