Creating Meaningful Benchmarking Reports Using Data from a Flexible Data Standard MedBiq 2016 Terri Cameron, MA Director, Curriculum Programs

© 2015 AAMC. May not be reproduced without permission. Objectives Update on AAMC Implementation of CI Data Exchange Standard Describe reason for MedBiquitous CI Standard Review need for flexibility Examine data validation issues

© 2015 AAMC. May not be reproduced without permission. Curriculum Inventory Participation Schools120 Schools (82 of 90 retained; 28 new schools) 135 Schools (114 retained; 21 new schools) 9 Schools that participated in either or did not participate in due to changes in curriculum management systems, changes in administration, and/or accreditation preparation US Medical Schools Canadian Medical Schools 555 US Osteopathic Schools (Pilot)n/a 3

© 2015 AAMC. May not be reproduced without permission. Accomplishments this year: Increased CI participation by 15 schools (12%). Presented CI to Federation of Associations of Schools of the Health Professions (FASHP) Implemented a pilot of Osteopathic schools. Data analysis for anomalies and potential need for additional data checking / Business Rules occurs weekly. Charts based on CI Data Statistics have been added to A report request that produces detailed content reports (see CI Report Sample) has been added to A Curriculum Inventory Task Force has been established to identify best ways to query and display CI data. A CI Research Group has been established to review CI data create a set of research projects based on CI data.

© 2015 AAMC. May not be reproduced without permission. Need for Data Exchange Standard Avoid duplicate data entry Upload data directly from existing curriculum management systems to national Curriculum Inventory Easier transition from one curriculum management system to another Standardized Vocabulary Instructional Methods / Assessment Methods / Resources Physician Competency Reference Set Curriculum Mapping Terminology ? Expectations Competencies / Learning Objectives / Outcome Objectives

© 2015 AAMC. May not be reproduced without permission. Flexibility in MedBiquitous CI Data Exchange Standard Structure Academic Level Sequence Blocks Nested Sequence Blocks Events Can reference multiple Sequence Blocks Tracks Terminology Local instructional and assessment methods crosswalked to Standardardized Vocabulary Local keywords, SB and Event names and expectations matched using synonym process

© 2015 AAMC. May not be reproduced without permission. Academic Level Distribution by Length (Month Increments) Academic Level Number of Schools 0_6 Months 7_12 Months 13_18 Months 19_24 Months >24 Months %89%6%1% %86%9%2%1% %72%24%0%1% 490 6%76%14%2% 52 0%100%0% %0% Schools are less likely to report higher academic levels, 90 as opposed to 135, (even excusing the 6 newer schools) and this is looking like a trend (67% of participants provided AL 4 data in 2015, compared to 72% in 2014 and 73% in 2013, and 83% provided AL 3 data, compared with 87% and 88% in 2015, 2014, and 2013 respectively).

© 2015 AAMC. May not be reproduced without permission. Course Details In academic level 1, we see most schools offering between 6 to 14 required non-clerkships and 2 to 9 optional non-clerkships. Of the total population, 37 offered both required and optional non-clerkships, 88 offered only required non-clerkships, and 9 offered only optional non-clerkships. One school, not shown here, offered only clerkships.

© 2015 AAMC. May not be reproduced without permission. Course Details In academic Level 2, we see most schools offering between 5 and 13 required non-clerkships and 2 to 13 optional non-clerkships. Of the total population in academic level 2, 39 offered both required and optional non-clerkships, 78 offered only required non-clerkships, and the same 9 offered only optional non-clerkships. Not all schools provided academic level 2 data.

© 2015 AAMC. May not be reproduced without permission. Course Details Academic level 3, however, presents a very different story. Most schools offer only 1 to 6 required non-clerkships and 1 to 13 optional non-clerkships. We have some very clear outliers (82, 90, and 208). Removing those brings the range or optional non-clerkships to 1 to 9 instead, which all needs to be weighed against the fact that only 36 schools are offering any optional non-clerkships. Of the total population in academic level 3, 20 offered both required and optional non-clerkships, 63 offered only required non-clerkships, and 16 offered only optional non-clerkships. Not all schools provided academic level 3 data.

© 2015 AAMC. May not be reproduced without permission. Course Details For Academic Level 4, we included Clerkships and non-clerkships together, but kept them broken apart by Required and Optional. Most schools offer 5 or fewer required sequence blocks and 12 or fewer sequence blocks. We are not certain what to say about the high counts of required sequence blocks at some schools.

© 2015 AAMC. May not be reproduced without permission. Clerkship Details In , 94 schools provided Clerkship data. As can be seen in the graph above, across their entire curriculum, schools had on average, 10.4 clerkships (the median is 9). If we use the interquartile range of 5 (the difference between “7” for the 25 th percentile and “12” for the 75 th percentile) to help us set the bounds for extreme outliers, we find we may drop the 35 and 52. This brings the average down to 9.7 clerkships per curriculum.

© 2015 AAMC. May not be reproduced without permission. Pre-Clerkship Academic Level Average Number of Sequence Blocks Median Number of Sequence Blocks Average Number of Events Median Number of Events We defined Pre-Clerkship as the “ Clerkship Model ” being left as NULL and the academic level as 3 or less. This table does not use the “ Required ” value as a separator.

© 2015 AAMC. May not be reproduced without permission. Post-Clerkship Sequence Blocks (Courses) Academic Level Average Number of Sequence Blocks Median Number of Sequence Blocks Average Number of Events Median Number of Events We defined Post-Clerkship as the “ Clerkship Model ” being left as NULL and the academic level as 4 or above. This table does not use the “ Required ” value as a separator. The percent of participants reporting clerkships in each AL has been stable. However, the average number of clerkships reported in Academic Level 4 in particular has been on the rise.

© 2015 AAMC. May not be reproduced without permission. Reports Requested Jan – March 2016 Adverse Childhood ExperiencesEpidemiologyOrgan System AnatomyEthicsOrthopaedics Behavioral ScienceEBMPain Management BiochemistryGeneticsPathology BiostatisticsGun SafetyPathophysiology Breast ExamHistologyPelvic Exam Cell BiologyImmunologyPharmacology Clinical DiagnosisIPEPhysiatry Clinical SkillsMental HealthPhysical Medicine Communication and EthicsMicrobiologyPhysiology Correctional HealthMindfullnessRadiology DisclosureMusculoskeletalRheumatology Early Clinical ExperienceNeuroscienceSports Medicine Eating DisordersNutritionStandardized/Virtual Patient Elder AbuseOphthalmologySubstance Abuse EmbryologyOpioids

© 2015 AAMC. May not be reproduced without permission. Reports Requested April – May 2016 Anatomy Biostatistics CEPAER Community Service Cost Awareness Disability Endocrine Giving Bad News Health Policy History of Medicine Medical Marijuana Ophthalmology Palliative Care Service Learning Substance Abuse

© 2015 AAMC. May not be reproduced without permission. CI Data Analysis Issue 1: Copying one course over to all other courses Issue 2: Keywords that just have numbers Issue 3: Developing a Synonym-Matching Process Issue 4: Missing Titles for Competency Objects Issue 5: Uploading only One Event for All Sequence Blocks Issue 6: Uploading only One Academic Level Issue 7: Use of Nested Sequence Blocks Issue 8: Use of Tracks

© 2015 AAMC. May not be reproduced without permission CI Upload Portal opens August 1; closes September 30. Staging available now Preparations: Updating Curriculum Deans and Primary CI Admins Webinars / Training Goals: US medical schools: 100 percent participation (148) Canadian medical schools: 50 percent participation (9) Osteopathic School Pilot: 30 percent participation (10)

© 2015 AAMC. May not be reproduced without permission. Questions???