1. Perspectives on Curriculum Mapping – Local and National Impacts and Outcomes  Terri Cameron, MA, Director, Curriculum Programs, Association of American Medical Colleges  Katharine Agnew, BA, Curriculum Coordinator, Council on Curriculum University of Missouri Kansas City School of Medicine  Stefanie Ellison, MD, Associate Dean and Chair, Council on Curriculum University of Missouri Kansas City School of Medicine  Robin Rivest, M.Ed., MBA, Director of Curriculum Data Management, Oakland University William Beaumont School of Medicine  Robert Noiva, PhD, Associate Dean for Medical Education Oakland University William Beaumont School of Medicine

2. CGEA 2016: Innovation & Impact: Modern Medical Education Disclosures  None of the session faculty have anything to disclose…

3. CGEA 2016: Innovation & Impact: Modern Medical Education Learning Objectives:  By the end of this workshop, participants will be able to:  Discuss the issues involved in choosing a curriculum management system and the criteria that should be considered in choosing a system  Describe the challenges and solutions for engaging faculty in curriculum documentation efforts  Provide examples of terminology sets used to map curricula for horizontal and vertical integration, benchmarking, and accreditation and discuss the pros and cons of each  Explain how standardized vocabulary is used to facilitate reporting of curriculum content, pedagogy, and competencies  Review how benchmarking reports can be used with outcomes data for continuous improvement

4. CGEA 2016: Innovation & Impact: Modern Medical Education Format of Activities:  Introduction of Panel Discussion Issues by speakers (30 minutes)  Focus group discussions for session questions (30 minutes)  All groups answer all questions (7 mins per questions / 2 min wrap-up):  What are the best practices for choosing a curriculum management system? What criteria should be considered?  What terminology sets are being used and/or considered for mapping curricula? What are the pros and cons of each?  How can/should faculty be engaged in the curriculum documentation process?  What are other challenges that curriculum leaders face in documenting their curricula in curriculum management systems for local and benchmarking needs?  Report back to the whole group (20 minutes) (Groups)  Questions/Wrap-up/summary (10 minutes)

5. CGEA 2016: Innovation & Impact: Modern Medical Education Discussion Questions:  What are the best practices for choosing a curriculum management system? What criteria should be considered?  What terminology sets are being used and/or considered for mapping curricula? What are the pros and cons of each?  How can/should faculty be engaged in the curriculum documentation process?  What are other challenges that curriculum leaders face in documenting their curricula in curriculum management systems for local and benchmarking needs?

6. CGEA 2016: Innovation & Impact: Modern Medical Education Challenges and perspectives:  Developing a curriculum that will have the most impact on the health of the nation in today’s healthcare system;  Determining how to document innovative curricula so that it is captured in systems that report aggregate data;  Defining terminology to map throughout curricula for vertical and horizontal integration, benchmarking, and accreditation;  Deciding what pedagogy should be used to document instructional and assessment methods;  Documenting innovative curriculum efforts in a manner that can be aggregated in an international benchmarking system.

7. CGEA 2016: Innovation & Impact: Modern Medical Education Curriculum Management Systems:  Must match the needs of the medical school for ‘local’ curriculum administration, reporting, and CQI.  Must meet the needs of administrators, faculty, and students.  Must support major initiatives such as continuous quality improvement, curriculum renewal, accreditation, support of curriculum committees, reporting, accreditation, and medical education research.  Must have options for reports that provide appropriate data for faculty decisions.  Must be able to report local level to national database to facilitate benchmarking and contribute to national and international discourse on medical education trends and research, and how medical schools are preparing future physicians to function in the ever-changing health care environment.  Using the system efficiently often requires dedicated staff, standardized terminology, and faculty engagement.

8. CGEA 2016: Innovation & Impact: Modern Medical Education Multiple Perspectives:  Curriculum leaders assist faculty and curriculum committees in keeping up with current literature, trends and best practices in medical education, accreditation policies and procedures and developing innovations to implement curriculum change.  Curriculum staff work with faculty to develop best practices for organizing and documenting curricula and developing reports that will provide the best impact, both at the local level where faculty review the data as they make decisions, and for how those decisions impact graduates, and, ultimately, patients of those graduates.  Faculty use curriculum management system data to find peers covering similar content to coordinate appropriate redundancy and scaffolding, to make informed decisions at curriculum committee meetings, and to conduct research.

9. © 2015 AAMC. May not be reproduced without permission. From Local Data to National Impacts / Outcomes

10. © 2015 AAMC. May not be reproduced without permission. From Local Data to National Impacts / Outcomes

11. © 2015 AAMC. May not be reproduced without permission. From Local Data to National Impacts / Outcomes

12. www.med.umkc.edu

13. System Selection Essentials Logistics Bonus Features Data Server Integration Technical Support User Friendly Cost Database Population Training Mapping Structure Content Rollover Mobile Capabilities Evaluations Lottery Scheduling Faculty Activity Summary

14. Curriculum Management Vendors All of E Solutions Entrada E*Value Ilious LCMS+ MedHub Medtrics New Innovations OASIS one45 OpalQM TUSK 4iQ Solutions https://wwww.aamc.org/download/249510/data/cipvendordocument.pdf

15. System Benefits Reporting Curriculum Renewal Support of Curriculum Committees Medical Education Research National and International Discourse

16. Building the System System Selected Presented updates regularly Presented system needs System Populated Disseminated quarterly to courses and clerkships Held training sessions during regular meetings System QA and QI Curriculum leaders assigned to key courses and clerkships Review of each CIR and meetings with directors

17. Faculty Engagement: Challenges and Solutions Create submission deadlines, use standing meetings for in- person training, schedule one-on-one sessions Utilize course / clerkship coordinators to expedite process Director’s Contributions Obtain course / clerkship content to pre-populate database Introduce required curriculum mapping elements gradually Provide examples of ideal mapping structure and final reports Extensive Amounts of Data Share current literature, trends, best practices in medical education, accreditation, and curriculum innovation Highlight curriculum management data use in committee decisions and research projects Lack of Buy-In

18. Standardized Vocabulary Facilitates Reporting of: – Curriculum Content – Competencies UMKC Competencies mapped to PCRS for CIR purposes All sessions are mapped to UMKC Competencies (year level specific) and IPE Competencies (if applicable) – Pedagogy All session objectives are mapped using MedBiquitous standardized vocabulary – Instructional Methods – Assessment Methods – Resource Types

19. Curriculum Inventory Report

20. Terminology Sets Access to Information Alternative Health Care Community Health Computer Assisted Decision Making Critical Judgment Decision Analysis Differential Diagnosis DisparitiesEpidemiology Evidence- Based Medicine Genetics Health Care Systems Health Status Healthcare Financing Informatics IPELGBTQIANeurologyOrgan SystemsPatient Safety Problem- Solving Public Policy Publication Components Quality Improvement Scholarly Work /Opportunities Social Determinants Socioeconomic Sciences Statistics Study Characteristics Translational Research 30 key words were compiled from the following: ACGME Hot Topics DCI LCME Standards MeSH Program Specific Step 1/Step 2 Content Outlines UMLS

21. Reporting and Outcomes Horizontal and Vertical Integration – Pros: eliminate redundancies, implement new curricula – Cons: variations of terms missed Benchmarking – Pros: compare curricula at local, national, and international levels – Cons: identifying trends vs. long term integration Accreditation – Pros: expedites queries – Cons: addition/removal of terms from standards, questionnaires, etc.

22. Content Report Example: Benchmarking in QI As the Curriculum Inventory table demonstrates, medical school consistently cite Patient Safety and Quality Improvement in the form of required and elective curricula since 2011. It appears that medical schools are answering the call and are increasing educational activities in these important areas. Curriculum Inventory in Context – Vol.2, No.3 March 2015

23. Continuous Quality Improvement for Curriculum Management Examples of Curriculum Priorities: – Curriculum Quality and Accuracy – Instructional Methods – Clinical Conditions – Content Reviews – Link to outcomes Step 1, Step 2 CK, Step 2 CS, GQ, pass rates, NBME Subject examination scores, student evaluations

24. Curriculum Management CQI Example Search Results for Radiology in 2014-2015:

25. oakland.edu/medicine

26. Designing the Curriculum Inventory Process What does a new program need? How can we help to meet those needs?

27. Designing the Curriculum Inventory Process Need: Deliver an outstanding curriculum meeting the learner’s needs Assess (Analyze, Anticipate ) DesignDevelopImplementEvaluateImprove Curriculum Inventory

28. Deliverables 1.Generate a Curriculum Map 2.Meet LCME accreditation requirements 3.Collect and deliver AAMC Curriculum Inventory Report (CIR) data 4.Provide data sets for local curriculum oversight – session and assessment documentation – content evaluation – faculty workload – resource utilization 5.Provide an archival repository Designing the Curriculum Inventory Process What is the job to be done? Provide a useable Curriculum Inventory

29. Designing the Curriculum Inventory Process Datasets and Tools Faculty!

30. Engage those stakeholders in the process Curriculum leadership – Curriculum committees – Office of Medical Education – Course/Clerkship Directors – Provost’s Office – Dean suite Faculty and staff Resource oversight (Business office)? AAMC, LCME and Higher Learning Commission Students Designing the Curriculum Inventory Process Who are the stakeholders?

31. Conduct a needs assessment – explain requirements – share information – ask questions – anticipate requests Meet one-on-one with each Course and Clerkship Director – develop working vs. adversarial relationship – inform and listen! Actively support Committee and Sub-Committee Meetings – become a go-to resource Follow up to ensure curriculum data needs are met – is the curriculum data being used (as anticipated)? – is the necessary data in a useable format? – what additional data/reports should we provide? Designing the Curriculum Inventory Process Engaging the stakeholders

32. CI and Standardized Vocabulary Competencies (PCRS) – 8 Domains Program Learning Objectives and Core Competencies Course and Clerkship Objectives – K, S, A MedBiquitous Standardized Vocabulary: Instructional and Assessment Methods and Resources Designing the Curriculum Inventory Process Selecting a common language 61 Discipline s 34 LCME Hot Topics 388 Topics 483 Keywords From “Anatomy” to “Work-Related Health Issues”

33. Course data first, then Clerkship data – Manageable process, working out the bugs – Tailoring the training process along the way – Re-designing forms and reports based on feedback from stakeholders Timely reporting to AAMC and LCME – Quickly determined we needed to audit data collection/reporting – Refining the dataset is an iterative process – Importance of testing scenarios to develop process Workload and resource data has been useful – Quick turnaround for budget data – Reports to Departmental Chairs are invaluable Designing the Curriculum Inventory Process How has the process worked?

34. Samples of the Work Benchmarking Report Weeks of instruction Contact hours Types of instruction Types of assessment Sequencing Graduation requirements

35. Samples of the Work Adhoc Content Report Topic Reports: Abuse content in the curriculum – Request driven by faculty/leadership/Provost Office proposal – Data collection process involves Office of Medical Education and Curriculum Integration subcommittee – Content of report Instruction on Abuse in the OUWB Curriculum Abuse curriculum benchmarked at other medical schools Bibliography of literature on abuse curricula in Medical and Health Sciences education

36. Samples of the Work Course Summary Data Shows curriculum mapping at all levels Annual reports to the Curriculum Evaluation Committee – Documents types of instruction, assessment, and course content – CI data pre-populates data tables in the form Validates data accuracy and completion Ensures data consistency

37. Samples of the Work Workload Report Instructor workload – Contact hours – Type of instruction – Travel – Preparation – Leadership – Sortable by course, discipline, department Space utilization Scheduling

38. Successes – Meeting requirements on time – Effective collaboration with stakeholders – Software support has been outstanding – iSeek – Integration analyses for curriculum review – Ad hoc curriculum content reports – Course/clerkship summary report data – Workload reporting to chairs Challenges – Identifying useful keywords – Too much data – useful filters – need for faculty oversight Designing the Curriculum Inventory Process Successes and Challenges

39. CGEA 2016: Innovation & Impact: Modern Medical Education Discussion Questions: 30 Minutes: 7 Mins per Question; 2 Min Wrap-Up  What are the best practices for choosing a curriculum management system? What criteria should be considered?  What terminology sets are being used and/or considered for mapping curricula? What are the pros and cons of each?  How can/should faculty be engaged in the curriculum documentation process?  What are other challenges that curriculum leaders face in documenting their curricula in curriculum management systems for local and benchmarking needs?

40. CGEA 2016: Innovation & Impact: Modern Medical Education Session Wrap-Up  What are some ‘take-home’ messages from this session?  What action items will you take back to your institution?  What resources do you need to assist you with this process?  What colleagues have you just met who could work with you to implement new strategies or help you with resources?