1. Curriculum Inventory Administrators’ Group June 8, 2016 Terri Cameron, MA Director, Curriculum Programs

2. Presentations from the Spring Educational Meetings Best Practices for Choosing a Curriculum Management System by Alvaro Perez, MD, Universidad Central del Caribe Data Visualization by Christopher Vaughan, Boston University June CI in Context: Service Learning in US Medical Schools July CI in Context: USMLE Timing in US Medical Schools (Effect on Curriculum) Featured Report: Difficulty in Finding Inpatient Clinical Clerkship Placements Updating Contact Information in the Curriculum Inventory CI Data Analysis / Reviews of Key Documentation Issues Using Program Description to Designate Approval of School Name linked to CI Reports January – June Report Requests Curriculum Inventory Task Force / CI Research Group New Curriculum Mapping Section on the CI Website Planning for Spring 2017 Education Meetings Medical School Highlight: University at Buffalo by Alan Lesse, MD Next meeting: Wednesday, July 13, 1 pm ET Agenda

3. Best practices for choosing a curriculum management system ALVARO PEREZ, MD-MS ASSISTANT DEAN FOR CURRICULUM

4. What are the best practices for choosing a curriculum management system? What criteria should be considered?

5. Remember that this is not just a transaction, but a relationship that will potentially continue for years it is very important to involve IT department staff in all discussions from the very beginning Ash K. (2013) How to Choose the Right Learning Management System. Education Week, Vol 06(03), 25-27 What has been your biggest frustration with your current CM? If you had to do this all over again, what would you do differently?

6. Steps in the selection of the best CMS Analyze needs Define Requirements Organize options Evaluate the product Make the decision

7. Analyze needsDefine Requirements Organize options Evaluate the product Make the decision Institutional needs: ◦High academic administration level only ◦Curriculum Committee ◦Course and clerkship coordinators and directors ◦Accreditation requirement???

8. Analyze needsDefine Requirements Organize options Evaluate the product Make the decision 1.Functionality: ◦Configuration ◦Accessibility 2.Technical requirements: ◦Reliability? Security? Maintenance? ◦Inclusion as part of the LMS or as a stand alone product? ◦user friendly or specialized skills required? ◦Number of users 3.Costs Prioritize your requirements What is most important for your organization?

9. Analyze needsDefine Requirements Organize options Evaluate the product Make the decision The vendors that have agreed to upload to the Curriculum Inventory include: Curriculum Management System Solutions and Vendors* Medtrics 4iQ Solutions MedHub AllofE New Innovations E*Value OASIS Entrada One45 Ilios OpalQM Knowledge4You TUSK LCMS+

10. Play an active role when viewing and exploring CM product demonstrations "Make sure it's not the company saying, 'Let me show you the five things we think are really cool,' but it's the school saying, 'Here are the five things we want it to be able to do. Walk us through how to do it.' " Ash K. (2013) How to Choose the Right Learning Management System. Education Week, Vol 06(03), 25-27 Live Demonstration

11. "Sometimes you may get more bells and whistles with more money, but you may not need more bells and whistles," Is the system truly optimized to perform certain functions that are important for your School? How user-friendly are the features? What is the total cost of ownership for this system? What feedback did they have about their experiences? Was the system hard to navigate? Were the users able to complete the tasks given to them? Was the company representative receptive to answering difficult questions during the demonstration? What kind of technical support and professional development are included in the pricing? How much will this CM cost the School over five years? Ash K. (2013) How to Choose the Right Learning Management System. Education Week, Vol 06(03), 25-27 Evaluation Questions

12. Features to be considered when building a comparison matrix  Capability to report to CI  Reporting  Stand alone or Suite application  Easy migration  Manual filling  Linking process (tagging)  Customizable  Price  Benchmarking

13. Feature Medtrics 4iQ Solutions MedHub AllofE New Innovations E*Value OASIS Entrada One45 Ilios OpalQM Knowledge4You TUSK LCMS+ Easy Migration 0.1 Easy Tagging 0.1 Customize Key words 0.1 USMLE Tags included 0.1 High Quality Reports 0.25 Custom In-House Integration 0.02 Stand alone 0.2 Price 0.13 Score CM Features Rating Matrix (sample) Adapted from: Berking P, and Gallgher S. Choosing a Learning Management System (2016)

14. 0= does not have that feature 1= feature included but…. 2= has implemented this feature to the fullest extent possible 5=Automatic (built-in, out of the box feature) 4=Semi-automatic (mostly built-in, but requires some programming or customization to activate) 3=Semi-custom (partially available. The system can be adapted to implement this feature through moderate customization) 2=Custom (not available but can be added, possibly at high cost, with programming) 1=Not available (would be impossible or cost-prohibitive to customize the system to add the feature due to incompatibilities with system architecture, etc.) Suggested Scales Brandon-Hall (Brandon-Hall Group, 2010)

15. Analyze needsDefine Requirements Organize options Evaluate the product Make the decision YOUR FINALISTS Trials Comparisons from literature Demonstrations

16. Analyze needsDefine Requirements Organize options Evaluate the product Make the decision Now you got your 3 finalists to be presented to the Committee or Dean for final decision Remember that this is not just a transaction, but a relationship that will potentially continue for years and you will not be working alone

17. Curriculum Management at BUSM Christopher Vaughan Director, Evaluation and Curriculum Management Office of Medical Education Office of Academic Affairs

18. Curriculum Management at BUSM Fundamentally we are looking at two things in this process: Our Institutional Learning Objectives Are they appropriate? Are we meeting our goals? Our Core Content Is it appropriate? Are there gaps or unplanned redundancies?

19. Curriculum Assessment as a Quality Improvement (QI) Based on: Using the Results of Assessment: Lessons from the Wabash Study Charles Blaich and Kathy Wise, Wabash Center for Inquiry Boston University Office of the Provost Program Learning Outcomes Assessment Workshop October 2015

20. Curriculum Assessment Should: Lead to experiments Data  Evidence  Experiment Data = values (quantitative, qualitative variables) Evidence = when a group of faculty have made sense and come to some agreement about what the data are telling us Experiment = when you try something new (can be small), see what happens, and talk about what you learned with your community

21. 4GeriatricsHome Visits20 Demonstrate effective and empathic interviewing and examination of elderly patients 3 B,C 1,2,3,6,8,9,10,11,12,13, 14,15,16,17,19 Clinical Experience Ambulatory, Preceptorship Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; EMR Professionalism; patient safety; transitions of care; medical finance; home health care; geriatrics/aging/elder care, communication skills, cultural competencies, patient & student safety, primary, prevention, geriatrics, bias, ethics, skill assess, reason assess, professionalism 4GeriatricsHome Visits Develop management plans that considers patient functional status and preferences for common geriatric issues 3 B,U 1,2,3,6,8,9,10,11,12,13, 14,15,16,17,19 Clinical Experience Ambulatory, Preceptorship Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; EMR Professionalism; patient safety; transitions of care; medical finance; home health care; geriatrics/aging/elder care, communication skills, cultural competencies, patient & student safety, primary, prevention, geriatrics, bias, ethics, skill assess, reason assess, professionalism 4GeriatricsNursing Home Visits12 Describe the roles of the multi disciplinary team in the nursing home. 3 S 1,2,3,5,8,9,10,11,12, 13,14,15,16,17,19 Clinical Experience Ambulatory, Preceptorship, patient presentation--Learner Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; Rehabilitation/care of the disabled interprofessional communication; team work; geriatrics/aging/elder care 4GeriatricsNursing Home Visits Compare the level of care available in the nursing home compared to home care or the hospital 3 S 1,2,3,5,8,9,10,11,12, 13,14,15,16,17,20 Clinical Experience Ambulatory, Preceptorship, patient presentation--Learner Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; Rehabilitation/care of the disabled interprofessional communication; team work; geriatrics/aging/elder care 4GeriatricsNursing Home Visits Develop a management plan for a patient that considers patient functional status and goals of care 3 U,S 1,2,3,5,8,9,10,11,12, 13,14,15,16,17,19 Clinical Experience Ambulatory, Preceptorship, patient presentation--Learner Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; Rehabilitation/care of the disabled interprofessional communication; team work; geriatrics/aging/elder care 4GeriatricsClinic Visits5 Demonstrate effective and empathic interviewing and examination of elderly patients 3 B,U,C 1,2,3,8,9,10,11,12, 13,14,15,16,19 Clinical Experience Ambulatory, Preceptorship, patient presentation--Learner Initial Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; EMR Clinical Problem- solving/Decision-making; geriatrics/aging/elder care YearModule/Course Session TitleContact HoursLearning ObjectivePCRS LO Type BU CARES Match to Course LOs Instructional Methods Assessment Methods ResourcesKeywords

23. Clerkships Course/ModuleFamily MedicineGeriatricsMedicine 1Medicine 2NeurologyOb/GynPediatricsPsychiatryRadiologySurgeryEOTYA Summative Assessment Method(s) Clinical Documentation Review (Pt Write-Ups) X X X Clinical Performance Rating/Checklist (CSEF, Passport, EOYA etc.) XXXXXXXXXXX Exam Inst Dev Clinical PerformanceX X X Exam Inst Dev Written Computer Based X X Exam Inst Dev Oral Exam - Licensure - Clinical (USMLE) Exam - Licensure - Written (USMLE) Exam - Nationally Normed - ShelfX X XXXX XX Multisource Assessment (365 Evaluation) X X XXX XX Narrative Assessment XX X Oral Patient PresentationX XX Participation (Level) X Peer Assessment X Practical (Lab/BioLucida) Research or Project AssessmentXX XXX X Self-Assessment (For Self Directed Learning) X Data Source: 15-16 OME Annual Survey Completed by Course Director Assessment Method Source: AAMC MedBiquitous Curriculum Inventory Standards

24. Identify the assessment items within each tool provided that can provide evidence of student attainment of your ILO "letter". Assessment MethodILO - E - PreclerkshipILO - E - Clerkship Clinical Documentation Review (Pt Write-Ups) xx Clinical Performance Rating/Checklist (CSEF, Passport, EOYA, Etc.) xx Exam - Inst Dev, Clinical Performance (OSCE, EOYA) xx Exam - Licensure, Clinical Performance (USMLE) x Exam - Nationally Normed/Standardized (Shelf) x Multisource Assessment (360 Evaluation) x Narrative Assessment xx Oral Pt Presentation xx Participation (Level) x Peer Assessment x Research or Project Assessment x Self-Assessment (For SDL) xx Where is your letter assessed? What course or clerkship? When in the curriculum? Course/Clerkship/ModuleILO - E- PreclerkshipILO - E- Clerkship IP 1 x ICM 1 A&B x IP 2 x ICM 2 x Family Medicine x Geri x Med 1 x Med 2 x Neurology x Ob/Gyn x Surgery x EOTYA x

27. C - PrelerkshipWhat are we doing well?What needs improvement? EOFYA, EOSYA ?Are our rubrics adequate to assess communication skills? EG evaluating write-ups - are the rubrics clear Some written exams (e.g. is this an open ended questions, are you using interviewing principles - in ?EPH, HBM Quantity of assessment - NOT adequate for this letter EG PCRS 3.8, 7.3, 4.1 IP Oral Presentations Need Peer Assessment: is this a good method> Need to be explicit about how we do this if we do. Portfolio based assessment C - ClerkshipWhat are we doing well?What needs improvement? Some reasonable assessments going on in a number of formats: Oral presentations, EOTYA, USMLE, OSCE For the rest (not on list above) – not a lot of validation and need more Difference for more rigorous summative evaluation (as opposed to formative), moving towards entrustability Need more formative assessment as well Needs to be more systematic Are teaching a lot of these areas, but are not evaluating many of them Need more evaluations on: more workplace evaluations, simulations, OSCEs Use 4 th year, make it count, take them to the EPA level we would like them to have for internship We are teaching lots of areas, but we are not evaluating them (e.g. death/dying, bad diagnoses, etc.) Each clerkship could take a piece, but then need to connect them and evaluate CSEF weakness: Doesn’t assess interaction with hospital staff (e.g. nursing), other than that it measures a lot of communication skills More direct faculty observation of students with patients is VERY needed

28. CI in Context: June 2016 Service Learning in US Medical Schools Rebecca Keller, PhD and Ingrid Allard, MD Albany Medical College

29. CI in Context: July 2016 USMLE Timing in US Medical Schools R. J. Canterbury, MD, University of Virginia School of Medicine

30. Featured CI Report: Difficulty in Finding Inpatient Clinical Clerkship Placements

31. Updating Contact Information in the Curriculum Inventory Kick-off and status e-mails for 2015-2016 Upload begin August 1 Sent to Curriculum Dean and CI Primary Admin Check current designations in ‘Manage Users’ tab or send e- mail to ci@aamc.orgci@aamc.org OR send current contacts to ci@aamc.orgci@aamc.org Last year, two schools were disappointed to learn that they had missed the deadline because contacts had not been updated

32. CI Data Analysis Issue 8: Clerkship Start Month To respond to the question of how early clerkships are beginning in medical school MONTH AL 1 AL 2 AL 3 AL 4 AL 5 AL 6 11134 2115 311202 41126 511207 6391721 7224626 821911 92165 102 3 11132 1251

33. Key Documentation Issues for 2015-2016 Upload Using Program Description to Designate Approval of School Name linked to CI Reports Use Program Description to include the text: ReleaseName=Y OR ReleaseName=N For systems that do not have this option, an e-mail can be sent to ci@aamc.org (e-mail will need to be sent each year)ci@aamc.org Documenting Gateway Exams Use updated Resource List using IDs if possible, text if not

34. Curriculum Inventory: Standardized Vocabulary Update To be implemente d in 2018? To be implemented in 2016, with or without IDs AM15 Renamed: AM19: Exam – Institutionally Developed, Laboratory Practical New!

35. Reports Requested Jan – June 2016 Adv Childhood ExpEarly Clin ExpHistory of MedicinePathology AnatomyEating DisordersImmunologyPathophysiology AspirinElder AbuseIPEPelvic Exam Behavioral ScienceEmbryologyMedical MarijuanaPharmacology BiochemistryEndocrineMedical SocioeconomicsPhysiatry BiostatisticsEpidemiologyMental HealthPhysical Medicine Breast ExamEthicsMicrobiologyPhysician Impairment CEPAEREBMMindfullnessPhysiology Cell BiologyGeneticsMusculoskeletalPopulation Health Clinical DiagnosisGiving Bad NewsNeurosciencePublic Health Issues Clinical SkillsGun SafetyNutritionRadiology Comm / EthicsHealth Care FinancingOphthalmologyRheumatology Comm ServiceHealth Care Quality ImprovOpioidsService Learning Cost AwarenessHealth Care SystemsOrgan SystemSimulation Correctional HealthHealth DisparitiesOrthopaedicsSports Medicine DisabilityHealth PolicyPain ManagementStand/Virtual Pt DisclosureHistologyPalliative CareSubstance Abuse

36. CI Task Force Update: May 25 th Meeting  Reviewed specific responses to questions in Standard 7  Reviewed cases of AAMC calculations for 6.0-1,2, and 4 for Task Force participants  Determined “granular” data will be necessary for output  During our next meeting, participant-provided answers will be compared to first-pass AAMC calculations Next Meeting is June 22

37. CI Research Group Task Force Steering Committee Members: Jorie Colbert-Gertz, PhD; University of Utah (SDRME) Carolyn DuFault, PhD; Washington University in St. Louis Steven J. Durning, M.D., PhD, FACP Rachel Ellaway, PhD; University of Calgary Jeanne Farnan, MD, MHPE; University of Chicago Colleen Grochowski, PhD; Duke University Tanya Horsley, PhD, Royal College of Physicians and Surgeons of Canada (RIME) Brian Mavis, PhD; Michigan State University Martin Pusic, MD, PhD, New York University (MESRE) Valerie Smothers, MS, MedBiquitous (ex officio) Final member has been invited; awaiting response

38. CI Webinar Schedule Complete for season! Recordings available on website

39. CI at 2016 Medical Education Meetings Complete list of meetings on CI Website: https://www.aamc.org/initiatives/cir/423222/conferences.html https://www.aamc.org/initiatives/cir/423222/conferences.html Slides are being posted as they become available Presenters will be asked to present at upcoming CIAG webinars

40. CI at 2017 Medical Education Meetings Seeking schools with to present at Spring/Summer 2017 meetings (complete list will be posted to website in near future): Innovative approach to curriculum documentation Faculty Development Technical Process Data Collection Process Use of Curriculum Inventory Resources Support of Curriculum Committee(s) Benchmarking Continuous Quality Improvement Support of Faculty Research Collaboration

41. Curriculum at Jacobs School of Medicine and Biomedical Sciences Alan J. Lesse, M.D. Senior Associate Dean for Medical Curriculum

42. Curriculum Outline

43. History Started in this position Jan 2015 School had used CurrMIT extensively in past and was trying to move the CI through conversion at E*Value About a year with no results, followed by the unexpected death of the Curriculum Dean Terri Cameron and Scott Kroyer from E*Value visited

44. Curricular Index Started from no baseline Utilized the online calendar webpage to record all the events in first two years CSV file moved to Excel for first pass.

46. CI Data moved to database (Filemaker™) with moderate amount of scrubbing and cleaning Use of restricted vocabulary in Medbiquitous for as many terms as possible Names were a bear

47. Entry into E*Value Got a special break somehow – Scott Kroyer (no longer at E*Value) – E*Value took the exported Excel file from the database and imported it in the background, and… IT WORKED!!! Also mapped our Educational Objectives to the PCRS

48. This Year Trying to map course level objectives to Schools’ Educational Objectives Map Session level detail to course objectives

49. New School of Medicine and Biomedical Sciences Main Street Allen Street RPCI IHI /Dept Biomed Informatics Conventus MOB WCHOB BGH High St RIA

50. New Medical School – 1 st Floor Small Group Learning and Medical Admissions

53. Ground Floor and Atrium of the New School of Medicine and Biomedical Sciences

54. Trifecta Strategic Retreat for Curricular Redesign – August 2016 LCME Self-Study Move First Class on Jan 17, 2018 Complete Curricular Redesign 2018-2019 – New curriculum, LCME Visit, Run two curricula simultaneously in new building.

55. 2015-2016 CI Upload Portal opens August 1; closes September 30. Staging open now Goals: US medical schools: 100 percent participation (148) Canadian medical schools: 50 percent participation (9) Osteopathic School Pilot: 30 percent participation (10)

56. Questions?

57. Wednesday, July 13, 1 pm ET (Second Wednesday of each month, 1 pm ET) Registration Links posted in Training and Resources section of www.aamc.org/cir www.aamc.org/cir Please send agenda items to tcameron@aamc.orgtcameron@aamc.org Next meeting: