1. Perspectives on Curriculum Mapping – Local and National Impacts and Outcomes Terri Cameron, Director of Curriculum Programs, Association of American Medical Colleges Susan LeDoux, Associate Dean for Medical Education and Professor University of South Alabama College of Medicine Josev Lopez, Associate Director of Academic Technology, Texas Tech University Health Sciences Center at El Paso - Paul L. Foster School of Medicine Alvaro Perez, Director of Student Assessment and Program Evaluation and Assistant Professor in Biochemistry/Pharmacology, San Juan Bautista School of Medicine

2. Disclosures None of the session faculty have anything to disclose…

3. Learning Objectives: By the end of this workshop, participants will be able to: Discuss the issues involved in choosing a curriculum management system and the criteria that should be considered in choosing a system Describe the challenges and solutions for engaging faculty in curriculum documentation efforts Provide examples of terminology sets used to map curricula for horizontal and vertical integration, benchmarking, and accreditation and discuss the pros and cons of each Explain how standardized vocabulary is used to facilitate reporting of curriculum content, pedagogy, and competencies Review how benchmarking reports can be used with outcomes data for continuous improvement

4. Format of Activities: Introduction of Panel Discussion Issues by speakers (30 minutes) Focus group discussions for session questions (30 minutes) All groups answer all questions (7 mins per questions / 2 min wrap-up) What are the best practices for choosing a curriculum management system? What criteria should be considered? What terminology sets are being used and/or considered for mapping curricula? What are the pros and cons of each? How can/should faculty be engaged in the curriculum documentation process? What are other challenges that curriculum leaders face in documenting their curricula in curriculum management systems for local and benchmarking needs? Reports back to the whole group (20 minutes) (Groups) Questions/Wrap-up/summary (10 minutes)

5. Discussion Questions: What are the best practices for choosing a curriculum management system? What criteria should be considered? What terminology sets are being used and/or considered for mapping curricula? What are the pros and cons of each? How can/should faculty be engaged in the curriculum documentation process? What are other challenges that curriculum leaders face in documenting their curricula in curriculum management systems for local and benchmarking needs?

6. Challenges and perspectives: Developing a curriculum that will have the most impact on the health of the nation in today’s healthcare system; Determining how to document innovative curricula so that it is captured in systems that report aggregate data; Defining terminology to map throughout curricula for vertical and horizontal integration, benchmarking, and accreditation; Deciding what pedagogy should be used to document instructional and assessment methods; Documenting innovative curriculum efforts in a manner that can be aggregated in an international benchmarking system.

7. Curriculum Management Systems: Must match the needs of the medical school for ‘local’ curriculum administration, reporting, and CQI. Must meet the needs of administrators, faculty, and students. Must support major initiatives such as continuous quality improvement, curriculum renewal, accreditation, support of curriculum committees, reporting, accreditation, and medical education research. Must have options for reports that provide appropriate data for faculty decisions. Must be able to report local level to national database to facilitate benchmarking and contribute to national and international discourse on medical education trends and research, and how medical schools are preparing future physicians to function in the ever-changing health care environment. Using the system efficiently often requires dedicated staff, standardized terminology, and faculty engagement.

8. Multiple Perspectives: Curriculum leaders assist faculty and curriculum committees in keeping up with current literature, trends and best practices in medical education, accreditation policies and procedures and developing innovations to implement curriculum change; Curriculum staff work with faculty to develop best practices for organizing and documenting curricula and developing reports that will provide the best impact, both at the local level where faculty review the data as they make decisions, and for how those decisions impact graduates, and, ultimately, patients of those graduates. Faculty use curriculum management system data to find peers covering similar content to coordinate appropriate redundancy and scaffolding, to make informed decisions at curriculum committee meetings, and to conduct research.

9. From Local Data to National Impacts / Outcomes

10. © 2015 AAMC. May not be reproduced without permission. From Local Data to National Impacts / Outcomes

11. Curriculum Development and Curriculum Mapping at the University of South Alabama Susan P. LeDoux, Ph.D. Associate Dean, Medical Education and Student Affairs University of South Alabama College of Medicine

12. DEVELOP AND ADOPT COMPETENCY-BASED COM OBJECTIVES DESIGNATE THE SEQUENCE, TIME ALLOCATION AND STUDENT WORKLOAD FOR AN INTEGRATED SYSTEMS-BASED CURRICULUM FOR YEARS 1 AND 2 IDENTIFY MODULE DIRECTORS AND THREAD LEADERS Process of Curriculum Development and Curriculum Mapping DEVELOP SPECIFIC LEARNING OBJECTIVES WITHIN EACH M1-M2 SYSTEM MODULE, M3 CLERKSHIP AND M4 ROTATION IDENTIFICATION AND DEVELOPMENT OF CONTENT TO FULFILL OBJECTIVES AND ASSESSMENT TOOLS TO MEASURE THEM FACULTY DEVELOPMENT IN OBJECTIVE WRITING

13. DEVELOPMENT OF METHODS FOR LINKING OBJECTIVES

14. IDENTIFICATION AND DEVELOPMENT OF METHODS FOR CONTENT DELIVERY Process of Curriculum Development and Curriculum Mapping DEVELOPMENT OF METHODS FOR LINKING OBJECTIVES IDENTIFY VENDOR TO UPLOAD INFORMATION INTO CURRICULUM INENTORY PORTAL

15. GOALS Develop a database that could be uploaded into the Curriculum Inventory Portal Develop a searchable database that could used by students, faculty and administrators

16. CHALLENGES Obtaining information from faculty in a timely fashion Received an excessive number of objectives Manual entering of objectives was very labor intensive Manual mapping of objectives was also labor intensive

17. LESSONS LEARNED Develop template for standardized objectives Limit number of module objectives (20-50) Identify another software program for searching for content

18. Curriculum Mapping Using Internal Re-Design of Curriculum Management System Jose Lopez Associate Director of Academic Technology Texas Tech University Health Sciences Center-Paul L. Foster School of Medicine

19. Challenges Our Curriculum Structure was difficult to represent through our Curriculum Management System Inefficient reporting of Clerkship information. Cross course elements inappropriately represented Our Curriculum Structure did not map easily to the Curriculum Inventory Model Large Amount of time spent proofing the CI Report and reclassification of mismatch data

20. Curriculum Management System iLios v.1 Acquired software code from UCSF in 2006 Written in classic ASP-Legacy programming Highly tailored to our integrated curriculum (not easily replaced) CHAMP (Curriculum Hub And Management Platform) Design a new CMS that aligns to our Curriculum Inventory and curricular mapping

21. MedBiquitous Influences Adopted the following: Concepts and Vocabularies for: Instructional Methods Assessment Method Resource Types Mapped Institutional Learning Objectives to Physicians Competency Reference Set Flexibility of using Parent-Child /nested structures for Courses

22. Nested Course Structure Integrated Curricular Elements-ICE (parent course) 1.Curricular elements (child course) CEYE multi-course summative assessment CBSE formative assessments Yr2 and Yr3 OSCE summative assessments Longitudinal Surveys – ‘hidden curriculum monitoring’ re. social determinants of health, empathy Self-directed learning time 2.Non-curricular elements (child course) Orientations Holidays

23. Valuable Resources Use AAMC Staging Site to proof our Curriculum Verification Report. CI Working group to identify Curriculum reporting issues. CHAMP Steering Committee o VP/CIO (oversees all TTUHSCEP IT functions) o Sr. Director, Academic Technology (TTUHSCEP IT project oversight) o Director of Assessment and Evaluation (accreditation data management) o Assoc. Dean for Med Ed o Assist. Dean for Med Ed o Assoc. Director for Pre-Clerkship (course coordination expert) o Director of Technology Services (software development) o Assoc. Director for Academic Technology (operational oversight) o Lead Analyst for Academic Technology (testing, manuals, training) o 3 Programmer/Analyst (software development)

24. CMS Redesign

25. CMS Development Timeline

26. Best practices for choosing a curriculum management system Alvaro Perez, MD

27. What are the best practices for choosing a curriculum management system? What criteria should be considered?

28. Remember that this is not just a transaction, but a relationship that will potentially continue for years it is very important to involve IT department staff in all discussions from the very beginning Ash K. (2013) How to Choose the Right Learning Management System. Education Week, Vol 06(03), 25-27 What has been your biggest frustration with your current CM? If you had to do this all over again, what would you do differently?

29. Steps in the selection of the best CMS Analyze needs Define Requirements Organize options Evaluate the product Take the decision

30. Analyze needs Define Requirements Organize options Evaluate the product Take the decision 1. Institutional needs: Part of a LMS or separated product? 2. Operability 3. Technical considerations

31. Analyze needs Define Requirements Organize options Evaluate the product Take the decision 1. Functionality: Configuration Accessibility 2. Technical requirements: Security Number of users 3. Costs Prioryze your requirements What is most important for your organization?

32. Analyze needs Define Requirements Organize options Evaluate the product Take the decision The vendors that have agreed to upload to the Curriculum Inventory include: Curriculum Management System Solutions and Vendors* Medtrics 4iQ Solutions MedHub AllofE New Innovations E*Value OASIS Entrada One45 Ilios OpalQM Knowledge4You TUSK LCMS+

33. Features to be considered when building a comparison matrix Capability to report to CI Reporting Stand alone or Suite application Easy migration Manual filling Linking process Customizable Price Benchmarking

34. Feature Medtrics 4iQ Solutions MedHub AllofE New Innovations E*Value OASIS Entrada One45 Ilios OpalQM Knowledge4You TUSK LCMS+ Easy Migration 0.1 Easy Tagging 0.1 Customize Key words 0.1 USMLE Tags included 0.1 High Quality Reports 0.25 Custom In-House Integration 0.02 Stand alone 0.2 Price 0.13 Score CM Features Rating Matrix (sample) Adapted from: Berking P, and Gallgher S. Choosing a Learning Management System (2016)

35. 0= does not have that feature 1= feature included but…. 2= has implemented this feature to the fullest extent possible 5=Automatic (built-in, out of the box feature) 4=Semi-automatic (mostly built-in, but requires some programming or customization to activate) 3=Semi-custom (partially available. The system can be adapted to implement this feature through moderate customization) 2=Custom (not available but can be added, possibly at high cost, with programming) 1=Not available (would be impossible or cost-prohibitive to customize the system to add the feature due to incompatibilities with system architecture, etc.) Suggested Scales Brandon-Hall (Brandon-Hall Group, 2010)

36. Analyze needs Define Requirements Organize options Evaluate the product Take the decision YOUR FINALISTS Trials Comparisons from literature Demonstrations

37. "Sometimes you may get more bells and whistles with more money, but you may not need more bells and whistles," Is the system truly optimized to perform certain functions that are important for your School? How user-friendly are the features? What is the total cost of ownership for this system? What feedback did they have about their experiences? Was the system hard to navigate? Were the users able to complete the tasks given to them? Was the company representative receptive to answering difficult questions during the demonstration? What kind of technical support and professional development are included in the pricing? How much will this CM cost the School over five years? Ash K. (2013) How to Choose the Right Learning Management System. Education Week, Vol 06(03), 25-27 Evaluation Questions

38. Play an active role when viewing and exploring CM product demonstrations "Make sure it's not the company saying, 'Let me show you the five things we think are really cool,' but it's the school saying, 'Here are the five things we want it to be able to do. Walk us through how to do it.' " Ash K. (2013) How to Choose the Right Learning Management System. Education Week, Vol 06(03), 25-27 Live Demonstration

39. Discussion Questions: 30 Minutes: 7 Mins per Question; 2 Min Wrap-Up What are the best practices for choosing a curriculum management system? What criteria should be considered? What terminology sets are being used and/or considered for mapping curricula? What are the pros and cons of each? How can/should faculty be engaged in the curriculum documentation process? What are other challenges that curriculum leaders face in documenting their curricula in curriculum management systems for local and benchmarking needs?

40. Session Wrap-Up What are some ‘take-home’ messages from this session? What action items will you take back to your institution? What resources do you need to assist you with this process? What colleagues have you just met who could work with you to implement new strategies or help you with resources?