1. Leading a Horse to Water: Using Automated Reminders to Increase Use of Online Decision Support James J. Cimino and Dmitriy Borovtsov NIH Clinical Center and Department of Biomedical Informatics Columbia University College of Physicians and Surgeons

2. Knowledge Resources are Underused Clinicians (physicians) with questions in clinical practice: –sought answers only 45% of the time –found answers only 34% of the time –used online resources 18% of the time * Ely JW, Osheroff JA, Maviglia SM, Rosenbaum ME. Patient-care questions that physicians are unable to answer. J Am Med Inform Assoc. 2007 Jul- Aug;14(4):407-14.

3. Health Resources Available from main menu Static list of resources No assistance with retrieval

6. Infobuttons Links inserted next to clinical information Resources selected based on likely information need Links are customized to assist with retrieval

8. Log File of HR vs IB 2006 2006 Health Resources Infobuttons Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 7000 6000 5000 4000 3000 2000 1000 0 Monthly UsageMonthly Usage

9. Log File of HR vs IB 2006, 2007 2006 2007 Health Resources Infobuttons Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 7000 6000 5000 4000 3000 2000 1000 0 Monthly UsageMonthly Usage

10. Educational Interventions Orientation for new housestaff Orientation for new medical students Infobutton exercises for new medical students

11. Educational Interventions 2006 Medical Students Housestaff Jun Jul Aug Sep Oct Nov Dec Monthly UsageMonthly Usage 450 400 350 300 250 200 150 100 50 0

12. Educational Interventions 2006 2007 Medical Students Housestaff Jun Jul Aug Sep Oct Nov Dec 450 400 350 300 250 200 150 100 50 0 Monthly UsageMonthly Usage

15. Usage of IB vs HR in Inpatient Meds Health Resources Infobuttons Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 1400 1200 1000 800 600 400 200 0 Monthly UsageMonthly Usage

16. Usage of IB in Outpatient Meds Health Resources Infobuttons Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 800 700 600 500 400 300 200 100 0 Monthly UsageMonthly Usage

17. Hypothesis Users of the Health Resources page in a particular context will be especially receptive to a suggestion to use Infobuttons in the same context.

18. Send Reminder No Stop Yes Reminder Sent in past 2 months? No Stop Yes IB Used in past 2 months? Intervention HR used in OM HR = Health Resources Page OM = Outpatient Medications

19. E-Mail Date: Fri, 2 Mar 2007 06:00:11 -0500 (EST) From: James Cimino To: xxx@columbia.edu Subject: Getting Drug Information while Using WebCIS Dear WebCIS User: I am writing to let you know about a handy feature in WebCIS called the "infobutton". You will see it in places like the Outpatient Medication list - it is a little purple circle with a white letter "i". If you click on it, it will give you a list of topics that you can select to get more information about the drug next to the infobutton. I hope you'll find it useful. --Jim Cimino, Infobutton Project Manager

20. Reminders Sent 522 messages 371 users –80 attending physicians –189 housestaff –29 nurses –24 students –49 other/unknown 173 users received two messages 1 user received three messages Average 89.9 days between messages

21. Usage of Infobuttons by Reminder Recipients 371 recipients of 552 reminders 111 (20.1%) eventually used IBs in OM 52 (9.4%) “early responders” –26 housestaff –18 attending physicians –8 other/unknown 201 used either IB or HR within one month 25.9% “early responders with opportunity” 9 early responders used IBs 8 to 97 times in subsequent months

22. Intervention versus Control 552 Intervention 525 Control Days to First Use of Infobuttons after First Use of Health Resources Page P<0.05 by Day 4

23. Effect of delay

24. Adverse Effects 522 reminders 12 replies –All were positive –Most were thanking us for infobuttons

25. Discussion Traditional training method appears to have limited effect Simple, spam-like reminder appears to have an effect on a substantial fraction (26%) of users Sustained effect in some (17%) of the early responders

26. Limitations Impersonal message Timing of reminder less-than ideal However: Logfile+e-mail approach is application independent Quasi-experimental design Convenience sample of controls However: Strong temporal association between intervention and outcome

27. Conclusions Automated, context-specific reminders are technically feasible, requiring few resources Reminders to use online information resources appear to be effective Timing of message is critical to success

28. Acknowledgments This work was supported in part by: National Library of Medicine Grant R01LM07593 NIH Clinical Center intramural research funds National Library of Medicine intramural research funds The authors also thank: Dr. Jianhua Li for technical support Dr. Rick Gallagher for log files Dr. Krystl Haerian for statistical support