Leading a Horse to Water: Using Automated Reminders to Increase Use of Online Decision Support James J. Cimino and Dmitriy Borovtsov NIH Clinical Center and Department of Biomedical Informatics Columbia University College of Physicians and Surgeons

Knowledge Resources are Underused Clinicians (physicians) with questions in clinical practice: –sought answers only 45% of the time –found answers only 34% of the time –used online resources 18% of the time * Ely JW, Osheroff JA, Maviglia SM, Rosenbaum ME. Patient-care questions that physicians are unable to answer. J Am Med Inform Assoc Jul- Aug;14(4):

Health Resources Available from main menu Static list of resources No assistance with retrieval

Infobuttons Links inserted next to clinical information Resources selected based on likely information need Links are customized to assist with retrieval

Log File of HR vs IB Health Resources Infobuttons Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Monthly UsageMonthly Usage

Log File of HR vs IB 2006, Health Resources Infobuttons Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Monthly UsageMonthly Usage

Educational Interventions Orientation for new housestaff Orientation for new medical students Infobutton exercises for new medical students

Educational Interventions 2006 Medical Students Housestaff Jun Jul Aug Sep Oct Nov Dec Monthly UsageMonthly Usage

Educational Interventions Medical Students Housestaff Jun Jul Aug Sep Oct Nov Dec Monthly UsageMonthly Usage

Usage of IB vs HR in Inpatient Meds Health Resources Infobuttons Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Monthly UsageMonthly Usage

Usage of IB in Outpatient Meds Health Resources Infobuttons Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Monthly UsageMonthly Usage

Hypothesis Users of the Health Resources page in a particular context will be especially receptive to a suggestion to use Infobuttons in the same context.

Send Reminder No Stop Yes Reminder Sent in past 2 months? No Stop Yes IB Used in past 2 months? Intervention HR used in OM HR = Health Resources Page OM = Outpatient Medications

Date: Fri, 2 Mar :00: (EST) From: James Cimino To: Subject: Getting Drug Information while Using WebCIS Dear WebCIS User: I am writing to let you know about a handy feature in WebCIS called the "infobutton". You will see it in places like the Outpatient Medication list - it is a little purple circle with a white letter "i". If you click on it, it will give you a list of topics that you can select to get more information about the drug next to the infobutton. I hope you'll find it useful. --Jim Cimino, Infobutton Project Manager

Reminders Sent 522 messages 371 users –80 attending physicians –189 housestaff –29 nurses –24 students –49 other/unknown 173 users received two messages 1 user received three messages Average 89.9 days between messages

Usage of Infobuttons by Reminder Recipients 371 recipients of 552 reminders 111 (20.1%) eventually used IBs in OM 52 (9.4%) “early responders” –26 housestaff –18 attending physicians –8 other/unknown 201 used either IB or HR within one month 25.9% “early responders with opportunity” 9 early responders used IBs 8 to 97 times in subsequent months

Intervention versus Control 552 Intervention 525 Control Days to First Use of Infobuttons after First Use of Health Resources Page P<0.05 by Day 4

Effect of delay

Adverse Effects 522 reminders 12 replies –All were positive –Most were thanking us for infobuttons

Discussion Traditional training method appears to have limited effect Simple, spam-like reminder appears to have an effect on a substantial fraction (26%) of users Sustained effect in some (17%) of the early responders

Limitations Impersonal message Timing of reminder less-than ideal However: Logfile+ approach is application independent Quasi-experimental design Convenience sample of controls However: Strong temporal association between intervention and outcome

Conclusions Automated, context-specific reminders are technically feasible, requiring few resources Reminders to use online information resources appear to be effective Timing of message is critical to success

Acknowledgments This work was supported in part by: National Library of Medicine Grant R01LM07593 NIH Clinical Center intramural research funds National Library of Medicine intramural research funds The authors also thank: Dr. Jianhua Li for technical support Dr. Rick Gallagher for log files Dr. Krystl Haerian for statistical support