Use of Online Marketing Technology to Track Resident Engagement in a FOAM-Supplemented Curriculum Ian Justl Ellis, M.D. and Daniel Egan, M.D.| Department of Emergency Medicine at Mount Sinai St. Luke’s-Roosevelt Hospital Deliver exceptional, modern, and individualized content in an engaging fashion. Allow for continuous, real-time, individualized, and “hassle-free” monitoring of learner engagement, allowing for data-driven curriculum adjustment and targeted intervention in disengaged or struggling learners. Survey-based methods of monitoring are fraught with recall bias, and require both effort to conduct as well as honesty in self-reporting. We believe an ideal EM curriculum should: We now deliver weekly, senior-resident procured and curated “FOAM” content, alongside traditional resources (i.e., textbook chapters and journal articles), to our resident body via a free, online, -based marketing software (Mailchimp™). The decision to include FOAM resources in our curriculum this year stemmed from a growing sense that contemporary residents may be disengaging from textbook readings. This software provides automatic, comprehensive, individualized, and easy-to-use data analysis on resident engagement in our curriculum, facilitating continuous improvement, improving resident accountability, and providing a rich source of data for study. Educational innovation:

Use of Online Marketing Technology to Track Resident Engagement in a FOAM-Supplemented Curriculum Each week our Resident Director of Enhanced Medical Education (a leadership role assigned to a senior resident) generates a , using a simple drag-and-drop interface, which provides links to textbook chapters, primary literature, podcasts, blog posts, and other curated materials related to the chosen weekly topic (Figure 1). Once the is delivered, MailChimp automatically registers when, and if, a resident opens the or clicks on its links. The program continuously calculates “opened,” “not opened,” and “clicked” rates for the distribution list, subgroups, and individual residents both for specific s and the year as a whole. This allows for nearly effortless, 100% capture of resident engagement in the prescribed curriculum. Figure 1. Excerpt from a weekly created with Mailchimp™. How does it all work? Ian Justl Ellis, M.D. and Daniel Egan, M.D.| Department of Emergency Medicine at Mount Sinai St. Luke’s-Roosevelt Hospital

Use of Online Marketing Technology to Track Resident Engagement in a FOAM-Supplemented Curriculum Figure 2. Online dashboard showing overall data for individual campaigns. Data from each or “campaign” sent are presented to the program faculty in a simple and easy to interpret online dashboard (Figure 2). Viewing the report for a particular campaign displays each resident’s engagement or lack thereof with each and assigned resource. Reports can be easily generated tracking each resident’s engagement throughout the academic year. Faculty can intervene in real time as needed, either via sending a reminder to learners who have not clicked on a particular assignment, by meeting with individual learners who are not engaging with the prescribed material, or by adjusting the types of content sent to residents. How does it all work? Ian Justl Ellis, M.D. and Daniel Egan, M.D.| Department of Emergency Medicine at Mount Sinai St. Luke’s-Roosevelt Hospital

Use of Online Marketing Technology to Track Resident Engagement in a FOAM-Supplemented Curriculum Preliminary analysis of the data provided by Mailchimp has shown wide differences both between and within post-graduate years in preferred methods of learning. Formal research is ongoing; however, particularly notable in early data is apparent decreasing engagement with textbook-based materials with increasing level of training (Table 1). Overall, wide variation in utilization of traditional and non-traditional resources between residents demonstrates the need to continue to provide multiple types of media in order to engage the most learners. Total Textbook Chapters Provided 39Total FOAM Items Provided 110 PGY-1 Mean Chapters Read19Mean FOAM Items Read18 Median Chapters Read17Median FOAM Items Read16 Standard Deviation12Standard Deviation14 Range2 - 39Range PGY-2 Mean Chapters Read12Mean FOAM Items Read10 Median Chapters Read10Median FOAM Items Read4 Standard Deviation10Standard Deviation12 Range0 - 39Range PGY-3 Mean Chapters Read5Mean FOAM Items Read15 Median Chapters Read3Median FOAM Items Read9 Standard Deviation5 20 Range0 - 14Range Table 1. Consumption of Textbook Chapters and “FOAM” Content by Post- Graduate Year (Sample Size: 16 Weeks; 14 Residents Per Post-Graduate Year) What have we discovered? Future directions: We continue to accumulate a robust collection of data related to resident engagement within our curriculum, which will result in high- quality scholarship as well as contribute to continuing improvements in the quality of our curriculum. Disclosures: Neither Dr. Justl Ellis, nor Dr. Egan, nor the Emergency Medicine Residency Program at Mount Sinai St. Luke’s-Roosevelt have any financial interest in Mailchimp, and were not solicited by Mailchimp to utilize their marketing software. Ian Justl Ellis, M.D. and Daniel Egan, M.D.| Department of Emergency Medicine at Mount Sinai St. Luke’s-Roosevelt Hospital