1. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Using Data Visualization to Engage Faculty and Improve Curricula  Susan Albright, Director, Technology for Learning in the Health Sciences Tufts University School of Medicine  Terri Cameron, MA, Director, Curriculum Programs, Association of American Medical Colleges  Brian Reid, Associate Director of IT - Curricular Applications Dartmouth Geisel School of Medicine  Christopher Vaughan, Director, Evaluation and Curriculum Management, Boston University School of Medicine

2. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Disclosures  None of the session faculty have anything to disclose…

3. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Learning Objectives:  By the end of this workshop, participants will be able to:  Discuss the value of visualization in presenting data  Describe how visualization of curriculum data can improve faculty engagement in documentation  Provide examples of how data visualization can support curriculum improvement  Explain how knowledge and experience with data visualization can improve curriculum documentation

4. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Format of Activities:  Introduction of Panel Discussion Issues by speakers (35 minutes)  Focus group discussions for session questions (35 minutes)  All groups answer all questions (9 mins per questions / 2 min wrap-up):  What are the best practices for using data visualization? What is currently being done?  How can data visualization improve faculty engagement?  What data visualizations best lend themselves to curriculum improvement?  Report back to the whole group (15 minutes) (Groups)  Questions/Wrap-up/summary (5 minutes)

5. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Data Visualization: Why?  To maximize the investment medical schools are making to document curriculum content.  To organize data and processes and engage faculty to ensure the highest possible quality of the data.  To reward faculty engagement by taking the data back to them in formats that make the best use of their time and shows the best options for curriculum improvement.

6. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Data Visualization: Why?  Large datasets are best understood through visualization.  Visualization pares information to its simplest form, stripping away noise to increase the efficiency with which a decision maker can understand it.  Visualization quickly shows coverage areas and helps faculty focus on areas for curriculum improvement.  There are two strong arguments for the power of data visualization:  vision is by far our dominant sense, and humans have evolved to perform many data sensing and processing tasks visually; and  visualization enhances pattern matching, showing data in patterns and pattern violations: trends, gaps, and outliers.

7. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Data Visualization: Challenges  Not all visualizations are actually that helpful.  Bar graphs, or line graphs made with software defaults and couched in a slideshow presentation or lengthy document can be at best confusing, and at worst misleading, but visualization done well can cause a revelation.  The best data visualizations expose something new about the underlying patterns and relationships contained within the data. Understanding those relationships — and being able to observe them — is key to good decision making.

8. © 2015 AAMC. May not be reproduced without permission. National Data Visualization Lecture Lab CAI Preceptor SP/OSCE TBL CBL PBL

9. Curriculum Management at BUSM Christopher Vaughan Director, Evaluation and Curriculum Management Office of Medical Education Office of Academic Affairs

10. Curriculum Management at BUSM Fundamentally we are looking at two things in this process: Our Institutional Learning Objectives Are they appropriate? Are we meeting our goals? Our Core Content Is it appropriate? Are there gaps or unplanned redundancies?

11. Curriculum Assessment as a Quality Improvement (QI) Based on: Using the Results of Assessment: Lessons from the Wabash Study Charles Blaich and Kathy Wise, Wabash Center for Inquiry Boston University Office of the Provost Program Learning Outcomes Assessment Workshop October 2015

12. Curriculum Assessment Should: Lead to experiments Data  Evidence  Experiment Data = values (quantitative, qualitative variables) Evidence = when a group of faculty have made sense and come to some agreement about what the data are telling us Experiment = when you try something new (can be small), see what happens, and talk about what you learned with your community

13. 4GeriatricsHome Visits20 Demonstrate effective and empathic interviewing and examination of elderly patients 3 B,C 1,2,3,6,8,9,10,11,12,13, 14,15,16,17,19 Clinical Experience Ambulatory, Preceptorship Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; EMR Professionalism; patient safety; transitions of care; medical finance; home health care; geriatrics/aging/elder care, communication skills, cultural competencies, patient & student safety, primary, prevention, geriatrics, bias, ethics, skill assess, reason assess, professionalism 4GeriatricsHome Visits Develop management plans that considers patient functional status and preferences for common geriatric issues 3 B,U 1,2,3,6,8,9,10,11,12,13, 14,15,16,17,19 Clinical Experience Ambulatory, Preceptorship Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; EMR Professionalism; patient safety; transitions of care; medical finance; home health care; geriatrics/aging/elder care, communication skills, cultural competencies, patient & student safety, primary, prevention, geriatrics, bias, ethics, skill assess, reason assess, professionalism 4GeriatricsNursing Home Visits12 Describe the roles of the multi disciplinary team in the nursing home. 3 S 1,2,3,5,8,9,10,11,12, 13,14,15,16,17,19 Clinical Experience Ambulatory, Preceptorship, patient presentation--Learner Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; Rehabilitation/care of the disabled interprofessional communication; team work; geriatrics/aging/elder care 4GeriatricsNursing Home Visits Compare the level of care available in the nursing home compared to home care or the hospital 3 S 1,2,3,5,8,9,10,11,12, 13,14,15,16,17,20 Clinical Experience Ambulatory, Preceptorship, patient presentation--Learner Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; Rehabilitation/care of the disabled interprofessional communication; team work; geriatrics/aging/elder care 4GeriatricsNursing Home Visits Develop a management plan for a patient that considers patient functional status and goals of care 3 U,S 1,2,3,5,8,9,10,11,12, 13,14,15,16,17,19 Clinical Experience Ambulatory, Preceptorship, patient presentation--Learner Clinical Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; Rehabilitation/care of the disabled interprofessional communication; team work; geriatrics/aging/elder care 4GeriatricsClinic Visits5 Demonstrate effective and empathic interviewing and examination of elderly patients 3 B,U,C 1,2,3,8,9,10,11,12, 13,14,15,16,19 Clinical Experience Ambulatory, Preceptorship, patient presentation--Learner Initial Documentation review, Clinical Performance Checklist, Participation, self- assessment, narrative assessment, oral patient presentation Real patient; EMR Clinical Problem- solving/Decision-making; geriatrics/aging/elder care YearModule/Course Session TitleContact HoursLearning ObjectivePCRS LO Type BU CARES Match to Course LOs Instructional Methods Assessment Methods ResourcesKeywords

15. Clerkships Course/ModuleFamily MedicineGeriatricsMedicine 1Medicine 2NeurologyOb/GynPediatricsPsychiatryRadiologySurgeryEOTYA Summative Assessment Method(s) Clinical Documentation Review (Pt Write-Ups) X X X Clinical Performance Rating/Checklist (CSEF, Passport, EOYA etc.) XXXXXXXXXXX Exam Inst Dev Clinical PerformanceX X X Exam Inst Dev Written Computer Based X X Exam Inst Dev Oral Exam - Licensure - Clinical (USMLE) Exam - Licensure - Written (USMLE) Exam - Nationally Normed - ShelfX X XXXX XX Multisource Assessment (365 Evaluation) X X XXX XX Narrative Assessment XX X Oral Patient PresentationX XX Participation (Level) X Peer Assessment X Practical (Lab/BioLucida) Research or Project AssessmentXX XXX X Self-Assessment (For Self Directed Learning) X Data Source: 15-16 OME Annual Survey Completed by Course Director Assessment Method Source: AAMC MedBiquitous Curriculum Inventory Standards

16. Identify the assessment items within each tool provided that can provide evidence of student attainment of your ILO "letter". Assessment MethodILO - E - PreclerkshipILO - E - Clerkship Clinical Documentation Review (Pt Write-Ups) xx Clinical Performance Rating/Checklist (CSEF, Passport, EOYA, Etc.) xx Exam - Inst Dev, Clinical Performance (OSCE, EOYA) xx Exam - Licensure, Clinical Performance (USMLE) x Exam - Nationally Normed/Standardized (Shelf) x Multisource Assessment (360 Evaluation) x Narrative Assessment xx Oral Pt Presentation xx Participation (Level) x Peer Assessment x Research or Project Assessment x Self-Assessment (For SDL) xx Where is your letter assessed? What course or clerkship? When in the curriculum? Course/Clerkship/ModuleILO - E- PreclerkshipILO - E- Clerkship IP 1 x ICM 1 A&B x IP 2 x ICM 2 x Family Medicine x Geri x Med 1 x Med 2 x Neurology x Ob/Gyn x Surgery x EOTYA x

19. C - PrelerkshipWhat are we doing well?What needs improvement? EOFYA, EOSYA ?Are our rubrics adequate to assess communication skills? EG evaluating write-ups - are the rubrics clear Some written exams (e.g. is this an open ended questions, are you using interviewing principles - in ?EPH, HBM Quantity of assessment - NOT adequate for this letter EG PCRS 3.8, 7.3, 4.1 IP Oral Presentations Need Peer Assessment: is this a good method> Need to be explicit about how we do this if we do. Portfolio based assessment C - ClerkshipWhat are we doing well?What needs improvement? Some reasonable assessments going on in a number of formats: Oral presentations, EOTYA, USMLE, OSCE For the rest (not on list above) – not a lot of validation and need more Difference for more rigorous summative evaluation (as opposed to formative), moving towards entrustability Need more formative assessment as well Needs to be more systematic Are teaching a lot of these areas, but are not evaluating many of them Need more evaluations on: more workplace evaluations, simulations, OSCEs Use 4 th year, make it count, take them to the EPA level we would like them to have for internship We are teaching lots of areas, but we are not evaluating them (e.g. death/dying, bad diagnoses, etc.) Each clerkship could take a piece, but then need to connect them and evaluate CSEF weakness: Doesn’t assess interaction with hospital staff (e.g. nursing), other than that it measures a lot of communication skills More direct faculty observation of students with patients is VERY needed

20. Curricular Data Visualization Brian Reid, Geisel Instructional Technology

21. Visualizing Data in the Curriculum Inventory Program/School Level – Objectives mapped to AAMC PCRS objectives Course Level – Course Objectives – Events/Sessions – When offered Event/Session Level – Primary method of instruction – Session objectives – Duration (hours) – When offered

22. School Objectives Mapped to AAMC PCRS Objectives

23. Bar Chart

24. School Objectives Mapped to AAMC PCRS Objectives Bar Chart

25. School Competency Domains Mapped to AAMC PCRS Domains Bar Chart

26. School Objectives Mapped to AAMC PCRS Competency Domains Polar Chart

27. Curricular Hours Mapped to Competency Domains All Academic Levels Year 1

28. Curricular Hours Mapped to Competency Domains Chord Diagram Competency Domains Academic Levels

29. Curricular Hours Mapped to Competency Domains Chord Diagram Competency Domains Academic Levels

30. Curricular Hours Mapped to Competency Domains Separated Chord Diagram Academic Levels Competency Domains

31. Year 2 Courses Mapped to Competency Domains Separated Chord Diagram Year 2 Courses Competency Domains

32. Year 3 Clerkships Mapped to Competency Domains Separated Chord Diagram Year 3 Clerkships Competency Domains

33. Competencies in Weekly Curricular Hours Stacked Bar Chart

34. Data Visualization across the Health Sciences Curriculum management, Faculty Support, Student Learning Susan Albright, Director Technology for Learning in the Health Sciences Tufts Technology Services

35. Where to use visualization? Any place data visualization moves understanding forward Competency relationships Curricular content Student achievement showing areas of difficulty and success Student Dashboard of progress toward competency achievement

36. Clinical Competency Dashboards For Administrators, course directors and students to view progress Link clinical assessment questions to school/course competencies to build a dashboard to compare competency achievement by competency across the 18 rotations in the 3 rd /4 th year of Vet school Show competency achievement and grades across the 10 Dental School Rotations for 3 rd and 4 th year Dental students Soon – link Examsoft questions to build a similar dashboard

37. 4 th year vet student Competencies dashboard by rotation by time in rotation Early rotationsLater rotations

38. 4 th year vet student Competencies dashboard by rotation by time in rotation

39. Challenges Normalizing the data 6 Rotations have scoring 1-20 12 Rotation have scoring 1-10 Differing rating scales i.e. 1-3, 1-4, 1-5 Vet school has taken this on as a project to make all scoring the same across the 18 assessments

40. 3 rd &4 th year Clinical Dental students by rotation

42. Competency Checklist- Identify areas most often requiring repeat assessments

43. Competency Linking Tool: Functional but not Readable

44. Visualizing the Curriculum MAP in TUSK

45. National Reference Set: PCRS PCRS competencies in the Patient Care Domain

46. Drilling Down National  School Competencies Drilling down to School level links

47. Drilling to Course Competencies Course Links

48. The Event: lecture, small group, assessmen t etc. Event level links

49. Proposed new competency visualization

51. Visualizations from Curriculum Inventory Data Academic Level 2 Courses/hours

52. Tools and Data Sources TUSK D3 - https://d3js.org/ Tableau - https://community.tableau.com/community/developers/javascript- api https://community.tableau.com/community/developers/javascript- api

53. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Discussion Questions: 35 Minutes: 9 Mins per Question; 2 Min Wrap-Up  What are the best practices for using data visualization? What is currently being done?  How can data visualization improve faculty engagement?  What data visualizations best lend themselves to curriculum improvement?

54. NEGEA 2016: Across the Medical Education Continuum: Learning, Sharing, Innovating Session Wrap-Up  What are some ‘take-home’ messages from this session?  What action items will you take back to your institution?  What resources do you need to assist you with this process?  What colleagues have you just met who could work with you to implement new strategies or help you with resources?