Curriculum Mapping Workshop Terri Cameron, MA Director of Curriculum Programs May 19, 2016

We will cover: Curriculum Data Necessary for a Successful CI Upload Curriculum Blueprinting vs Curriculum Mapping Mapping Terminology Options Best Practices for Curriculum Documentation Curriculum Administration Issues

CI Process Validate XML against MedBiq CI Standard Upload to CI Successful submission results in ‘Submitted Status’ Log into School Portal Download and Review Verification Report Download results in ‘Review Status’ Reject or Verify Data Rejection returns Status to ‘No Response’ Verification results in ‘Verified Status’

CI Verification Report Purpose: Confirm that data from CMS uploaded to CI as expected Review how data will be displayed in key LCME tables

CI Verification Report Glossary / Consistent Terminology across Tables New Summary Tables: Program Expectations to PCRS Table All Instructional Methods with number of events for each All Assessment Methods with number of events for each Program Expectations mapped to Assessment Methods PCRS column removed from both the Sequence Block and Event Tables Use of ID numbers where appropriate to reduce amount of text in cells Margin Reductions / Rotated Column Headings Will re-evaluate ‘LCME Tables’

CI Verification Report Institution (Program Information) All Academic Levels (Dates) All Sequence Blocks Expectations / Expectation Mapping (if provided) All Events Expectation Mapping (if available) Instructional Methods Primary Instructional Methods Assessments Events with Assessment Method(s) only

CI Verification Report

CI Verification Report

CI Verification Report

CI Verification Report

CI Verification Report

CI Verification Report

CI Verification Report

Curriculum Data Necessary for a Successful CI Upload Program Expectations Competencies Learning Objectives Outcome Objectives Milestones EPAs Mapped to PCRS

Sequence Blocks ALL Sequence Blocks: Courses, Clerkships, Electives Duration in Days AND / OR Begin / End Dates Clerkships in Days AND Begin / End Dates Days for single iteration Begin / End Dates for Iteration Period Expectations (Learning Objectives / Outcome Objectives / Competencies) Academic Level Nested Sequence Blocks (Optional) Organ Systems Integrated Clerkships / Longitudinal Clerkships Selectives Integration Blocks (Optional) Themes Vertical Integration / Horizontal Integration Curriculum Data Necessary for a Successful CI Upload

Curriculum Data Necessary for a Successful CI Upload Events Duration in hours and / or minutes Instructional Methods Can have multiple IMs but ONE must be designated Primary Assessment Methods Summative vs Formative Assessment Events Resources (Optional, but recommended) Expectations (Learning Objectives / Outcome Objectives / Competencies) Keywords (School-generated, LCME “Hot Topics,” curriculum content terms for LCME documentation, USMLE Content Outline, MESH, UMLS, etc.)

Mapping Requirements for CI Upload Program Expectations to PCRS Optional, but highly recommended: Sequence Block Expectations SB Expectations mapped to Program Expectations Event Expectations Event Expectations maped to Sequence Block OR Program Expectations OR both Keywords

Curriculum Inventory: Standardized Vocabulary To be implemented in 2018? To be implemented in 2016, with or without IDs AM15 Renamed: AM19: Exam – Institutionally Developed, Laboratory Practical New!

Curriculum Blueprinting vs. Curriculum Mapping Curriculum Blueprinting Planning for a new or revised curriculum Begin with Program Expectations and plan curriculum to meet those expectations Course and Event Expectations Keywords Vertical Integration Horizontal Integration Integration Themes Course Directors and Teaching Faculty build content based on Blueprint

Curriculum Blueprinting vs. Curriculum Mapping Curriculum Mapping Document existing curriculum Develop mechanism to find where content and competencies are covered Course and Event Expectations Keywords Vertical Integration Horizontal Integration Integration Themes Course Directors and Teaching Faculty assist in documenting mapping

Mapping Options: Expectations Developed centrally, often available as drop- down menus as faculty/staff document curricula Pool of non-hierarchical expectations that faculty/staff choose from Hierarchy of expectations Developed by different entities and may or may not be hierarchical or set up to be mapped Program: Curriculum Committee Course: Course Directors Events: Teaching Faculty

Mapping Options: Keywords Developed centrally by Curriculum Administration or Curriculum Committee Nationally-developed terminology MeSH / UMLS / SnoMed tline.pdf tline.pdf kstart.html kstart.html Licensure Exam Content Outline Clinical Conditions / Presenting Situations Combination of multiple terminology sets Accreditation content areas Continuum content areas (GME, CME)

USMLE Content Outline General Principles of Foundational Science Immune System Blood & Lymphoreticular System Behavioral Health Nervous System & Special Senses Skin & Subcutaneous Tissue Musculoskeletal System Cardiovascular System Respiratory System Gastrointestinal System Renal & Urinary System Pregnancy, Childbirth, & the Puerperium Female Reproductive System & Breast Male Reproductive System Endocrine System Multisystem Processes & Disorders Biostatistics, Epidemiology/Population Health, & Interpretation of the Medical Literature Social Sciences

Unified Medical Language System (UMLS) What is the UMLS? The UMLS, or Unified Medical Language System, is a set of files and software that brings together many health and biomedical vocabularies and standards to enable interoperability between computer systems. You can use the UMLS to enhance or develop applications, such as electronic health records, classification tools, dictionaries and language translators.

SNOMED The Systematized Nomenclature of Medicine (SNOMED) is a systematic, computer-processable collection of medical terms, in human and veterinary medicine, to provide codes, terms, synonyms and definitions which cover anatomy, diseases, findings, procedures, microorganisms, substances, etc.

Mapping Options: Keywords Developed centrally by Curriculum Administration or Curriculum Committee Locally-developed terminology Centrally-developed list of terms based on Program Expectations, School Mission, Government Regulations, and other faculty- generated terms Faculty generated, with or without standardization

Keyword List Example: ETSU (197 Terms Total)

Keyword List Example: Oakland (477 Terms Total)

Keyword List Example: UCSF (477 Terms Total)

Keyword List Example: Phoenix (154 Terms Total)

Top Keywords in the CI Pharmacology Anatomy Pathology Physiology Education, Medical, Undergraduate11416 Communication Skills9509 Educational Methods : Lecture9402 Clinical Skills8615 Pediatrics84224 Epidemiology83934 Professionalism79712 Communication76518 Physical Examination73317 Nutrition66922 Biochemistry63920 Genetics60621 Microbiology59719 Neurology57521 Infectious Disease5537 Histology53621 Psychiatry47918 Radiology47720 Diagnostic Imaging47418 Immunology44710 Communication skills43010 Clinical Reasoning4297 Self-Directed Learning4122 Medical History Taking40814 Clinical Competence37211 Patient Care3728 Evidence-based medicine3707 Cell Biology3668 Endocrinology36016 Pathophysiology3602 Patient Safety33322 Gastroenterology32212 Medical Genetics32010 Clinical Pathology3204 Educational Methods : Discussion3182 Neuroscience3156 Biostatistics30325 Cardiology30115 Surgery29612 Family Medicine29510 Hematology28815 Research Methods28811 Emergency Medicine28717 Evidence-Based Medicine28717 Geriatrics28028 Medical Ethics27211 Embryology27017 Oncology2699 Women's Health26515 Prevention/Health Maintenance2645 Reproductive System2572 Ambulatory Care25511 Internal Medicine25313 Metabolism25119 Ethics24518 pathophysiology2453 Diagnosis, Differential23212 Pain Management21517 Health Disparities21014 Physician-Patient Relations20812 Kidney20716 Hypertension20623 Education, Medical2039 Community Health1959 Human Development / Life Cycle1953 Diagnosis19412 Public Health19318 Nephrology1939 Gross Anatomy1932 Liver19118 Clinical/Translational Research1916 Dermatology18719 History1837 AAMC/LCME Hot Topics : Pharmacology1782 Diabetes Mellitus17718 Health Care Systems17611 Evidence Based Medicine1755 Human Development/Life Cycle1735 Substance Abuse17110 Substance Abuse / Addiction1712 Palliative Care16722 Disease Management16311 Anemia16220 Educational Methods : Self-Directed Learning1602 Pulmonary1593 Diversity1583 Pregnancy15517 Behavior15213 Medical ethics1526 Clinical Medicine1498 Heart14517 Pharmacokinetics14516 Obesity14321 Cardiovascular System14215 Infection14214 Preventive Medicine14213 Patient Health Education1424 Lung14018 Decision Making14012 Obstetrics and Gynecology1393

Reports Requested Jan – March 2016 Adverse Childhood ExperiencesEpidemiologyOrgan System AnatomyEthicsOrthopaedics Behavioral ScienceEBMPain Management BiochemistryGeneticsPathology BiostatisticsGun SafetyPathophysiology Breast ExamHistologyPelvic Exam Cell BiologyImmunologyPharmacology Clinical DiagnosisIPEPhysiatry Clinical SkillsMental HealthPhysical Medicine Communication and EthicsMicrobiologyPhysiology Correctional HealthMindfullnessRadiology DisclosureMusculoskeletalRheumatology Early Clinical ExperienceNeuroscienceSports Medicine Eating DisordersNutritionStandardized/Virtual Patient Elder AbuseOphthalmologySubstance Abuse EmbryologyOpioids

Reports Requested April – May 2016 Anatomy Biostatistics CEPAER Community Service Cost Awareness Disability Endocrine Giving Bad News Health Policy History of Medicine Medical Marijuana Ophthalmology Palliative Care Service Learning Substance Abuse

Best Practices for Curriculum Documentation Centralized Review syllabi and educational materials Interview faculty Distribute worksheets to faculty Provide an online tool for faculty Hire medical students Search software Natural language processing

Best Practices for Curriculum Documentation Distributed Faculty document as educational materials are uploaded to LMS Faculty provide course coordinators with mapping terminology Course Directors collect and document all mapping terminology for their courses

Curriculum Administration Issues Vendor System Challenges/Solutions Training Expectation options Keyword options Data Entry Reporting Data Entry Issues Alignment of CMS fields to CI fields Tedious vs Batch Mapping Time

Curriculum Administration Issues Data Collection Issues Busy faculty – Faculty Development to come to common understanding Faculty and staff time to document Lack of administrative support -- too many other issues

Curriculum Administration Issues Using Curriculum Data to Support Curriculum Committees) Annual Review of coverage of Program Expectations Annual Review of coverage of Mapped Content USMLE Reports Graduation Questionnaire Instructional / Assessment Methods Vertical / Horizontal Integration Course Reviews Content Initiatives Continuous Quality Improvement

Release School Name ( ) Gateway Exams ( ) CEPAER Documentation ( ) New Data Initiatives

Using Program Description to Designate Approval of School Name linked to CI Reports A Brief History CurrMIT allowed schools to run reports to find schools offering particular content or using particular instructional or assessment methods or resources Received both positively and negatively To expedite implementation of CI and ensure highest participation rate, schools were told all reports would be aggregate Received both positively and negatively

Using Program Description to Designate Approval of School Name linked to CI Reports Solution: Allow schools to choose whether they wish to have their name available for benchmarking and research School name will be in a separate data file – no link to actual data in report Schools Included in This Report link will include: School Name, Curriculum Dean Contact Information, Primary CI Contact Information

Using Program Description to Designate Approval of School Name linked to CI Reports Implementation August 1, 2016, for Data: Use Program Description to include the text: ReleaseName=Y OR ReleaseName=N No ReleaseName text will be the same as ReleaseName=N This should not require changes to programs – simply adding text to a field For systems that do not have this option, an can be sent to will need to be sent each year.

Gateway Exams: Major Exams for Promotion or Graduation Often multi-day exams Not related to a particular course or clerkship Often a series of unqueued cases Examples: Senior Clinical Competency Exam; CCX; Post-Clerkship Exam Best Practice for ‘Gateway Exams’

Best Practice for ‘Gateway Exams’ Documenting Gateway Exams: Separate Sequence Block Duration expected for average student Include all Assessment Methods as Single Event (e.g., All Cases) or Split into Multiple Events (e.g., Case 1, Case 2.) No Instructional Methods Include Keywords for Content if possible Reporting is post-exam and aggregate If exam is split into multiple Academic Levels, reference the AL where most of the exam occurs Use Sequence Block Description to provide innovations, information such as year exam was implemented, etc.

Best Practices for CEPAER Documentation CEPAER: Core EPAs for Entering Residency 13 EPAs Linked to PCRS Competencies In Pilot Phase (10 schools) Not a requirement, but being documented at many schools Document as CEPAER-01 – CEPAER-13 Detailed information (included PCRS mapping) can be found at:

Preparing for CI School Portal Opens August 1 Closes September 30 Staging (for testing uploads) available now Vendors automatically have access Available to school developers by request School Portal Opens: August 1 School Portal Closes: September 30 Status sent to CI Primary Admin, Curriculum Dean, and School Dean: September 30 Data clean-up/reporting starts: November 1

Resources R.M. Harden. “Curriculum mapping: a tool for transparent and authentic teaching and learning.” Medical Teacher, Vol. 23, No. 2, Judith A. Kitzes, Renate D. Savich, Summers Kalishman, John C. Sander, Arti Prasad, Christine R. Morris & Craig Timm. “Fitting it all in: integration of 12 cross-cuttingthemes into a School of Medicine curriculum.” Medical Teacher 29: 437–442, Terri Cameron, Kevin Krane, Genevieve Moineau, Joshua L. Jacobs, Ronald Harden. Mapping Medical Education Curricula. Learn Serve Lead: The AAMC 2012 Annual Meeting. Robby Reynolds, Cindi Hasit, Tomo Ito, Rebecca Keller, Lynn Romrell, Terri Cameron. Mapping Medical Education Curricula. Learn Serve Lead: The AAMC 2013 Annual Meeting.

Willett TG, Marshall KC, Broudo M, Clarke M. It’s about TIME: A general-purpose taxonomy of subjects in medical education. Med Educ. 2008; 42(4): Bell CE, Ellaway RH, Rhind SM. Getting started with curriculum mapping in a veterinary degree program. J Vet Med Educ. 2009; 36(1): Wong RY, Roberts JM. Real time curriculum map for internal medicine residency. BMC Med Educ. 2007; 7:42. Watson EG, Moloney PJ, Toohey SM, et al. Development of eMed: A comprehensive, modular curriculum- management system. Acad Med. 2007; 82(4): Miller WL. The clinical hand: A curricular map for relationship-centered care. Fam Med. 2004; 36(5):

Resources Susan M. Kies. “Curriculum Mapping: Knowing Where You are Going and How You are Going to Get There.” Medical Science Educator, Volume 20: No. 2. Cottrell S, Linger B, Shumway J. Using information contained in the curriculum management information tool (CurrMIT) to capture opportunities for student learning and development. Med Teach. 2004; 26(5): Kathol DD, Geiger ML, Hartig JL. Clinical correlation map. A tool for linking theory and practice.Nurse Educ. 1998; 23(4): Komenda M, Víta M, Vaitsis C, Schwarz D, Pokorná A, Zary N, et al. “Curriculum Mapping with Academic Analytics in Medical and Healthcare Education.” PLoS ONE 10(12): e , general

Open Q&A What questions do you have that have not been discussed? What information/assistance do you need?

Questions?

©