1. Lessons Learned in Clinical Decision Support: Over-alerting Tejal K. Gandhi, MD MPH Director of Patient Safety Brigham and Women’s Hospital Assistant Professor in Medicine Harvard Medical School

2. Overriding of Alerts Studies have shown that MDs override clinical decision support alerts a large percent of the time Studies have shown that MDs override clinical decision support alerts a large percent of the time  88% of inpatient DDI alerts overridden (Payne et al. Proc AMIA 2002)  83% of inpatient drug-allergy alerts (Abookire et al. Proc AMIA 2000)  89% of outpatient high severity DDI alerts and 91% of outpatient drug-allergy alerts (Weingart et al. Arch Intern Med 2003)

3. Are Overrides Appropriate? 7,761 drug-allergy alerts in BWH inpatients, Aug-Oct 2002 7,761 drug-allergy alerts in BWH inpatients, Aug-Oct 2002 Alerts were overridden 80% of the time Alerts were overridden 80% of the time In chart review of 300 overrides, all were clinically justified In chart review of 300 overrides, all were clinically justified  Evidence that we are over-alerting Only 6% of alerts were triggered by an exact match between drug ordered and drug in allergy list (lots of potential noise) Only 6% of alerts were triggered by an exact match between drug ordered and drug in allergy list (lots of potential noise) Hsieh et al. JAMIA 2004

4. Alerts Triggered by Exact Drug-Allergy Matches are Rare Ordered drug Hydromorphone ( DILAUDID ® ) (N=18) Drug in allergy list  Codeine (39%)  Oxycodone (22%)  Meperidine (22%)  Morphine (17%)  Hydromorphone (0%)

5. Alerts Triggered by Exact Drug-Allergy Matches are Rare Ordered drug Furosemide ( LASIX ® ) (N=18) Hydrochlorothiazide (N=4) Drug in allergy list  ‘Sulfa’ (95%)  Furosemide (5%)  ‘Sulfa’ (100%)  Hydrochlorothiazide (0%)

6. Allergy Alerting Recommendations Need better specificity of alerts to avoid false positives Need better specificity of alerts to avoid false positives  E.g. requiring exact matches for certain classes  Good news: FDB recently removed lasix/HCTZ – sulfa interaction

7. Overall Alerting Issues Need more studies to maximize effectiveness of alerts/ minimize over-alerting Need more studies to maximize effectiveness of alerts/ minimize over-alerting Issue of how best to display the messages Issue of how best to display the messages  Need to learn from other industries (industrial engineering)

8. Drug-Pregnancy Level 1

10. Potential Strategies to Improve Alerting Creation of streamlined knowledge bases Creation of streamlined knowledge bases  Only essential content  Balance between sensitivity and specificity Tiering of alerts is also a possibility Tiering of alerts is also a possibility  Hard stop  Interruptive  Non-interruptive Minimizing interruptions Minimizing interruptions

11. Impact of Reduced Alerting on Override Rates Study in the ambulatory setting Study in the ambulatory setting Decision support included Decision support included  Duplicate drug  Drug-disease  Drug-drug  Drug-lab  Drug-pregnancy Shah et al. JAMIA 2006

12. Knowledge base streamlining Expert panel Expert panel  Physicians, pharmacists, informaticians Reviewed sources Reviewed sources  Vendor knowledge-bases, pre-existing locally created KBs, literature  Removed certain alerts and tiered the rest

13. Alert tiers Level 1 – Potentially life-threatening Level 1 – Potentially life-threatening  E.g., erythromycin - diltiazem -> V-fib  “Hard stop” – couldn’t proceed Level 2 – Potential for serious injury Level 2 – Potential for serious injury  Rizatriptan - linezolid -> serotonin syndrome  Interruptive, required a reason Level 3 – Use w/ caution Level 3 – Use w/ caution  Warfarin – levofloxacin -> increased PT  Noninterruptive

16. Actions w/ level 2 alerts Cancel Cancel  Do not proceed with order Modify, examples include Modify, examples include  Alter the problem list  Discontinue the pre-existing drug  Hold the other medication “Accept” defined as “cancel” or “modify” “Accept” defined as “cancel” or “modify” “Override” defined as not accept “Override” defined as not accept

17. Results Final knowledge base Final knowledge base  2% level 1 63% level 2, 35% level 3 18,115 alerts 18,115 alerts  12,933 non-interruptive (71%)  5,182 interruptive (29%) Of 5,182 interruptive alerts Of 5,182 interruptive alerts  3475 (67%) accepted

18. Interruptive alerts AlertNAcceptedOverridden Duplicate class 3,875 2,695 (77%) 910 (23%) Drug-drug1078 451 (42%) 627 (58%) Drug-disease19 10 (53%) 9 (47%) Drug-lab92 37 (40%) 55 (60%) Drug-pregnancy118 12 (10%) 106 (90%) Total5182 3,475 (67%) 1,707 (33%)

19. Summary of Reduced Alerting Study Can reduce alert burden by streamlining and tiering the knowledge base Can reduce alert burden by streamlining and tiering the knowledge base Concept of a “non-interruptive” alert may be helpful Concept of a “non-interruptive” alert may be helpful Still need more research on what is optimal level of alerting Still need more research on what is optimal level of alerting  “Are we missing things” is always the worry

20. Non-interruptive alerts Alert N (% of all alerts in that category) Duplicate drug 0 (0%) Drug-drug 3547 (77%) Drug-disease 24 (56%) Drug-lab 4,444 (98%) Drug-pregnancy 4,918 (98%) Total 12,933 (71%)

22. Impact of Non-Interruptive Alerts Drilled down into the drug-lab alerts Drilled down into the drug-lab alerts No difference in lab ordering between control and intervention groups No difference in lab ordering between control and intervention groups Is it a user interface issue or are non-interruptive alerts just not effective? Is it a user interface issue or are non-interruptive alerts just not effective? Cost- benefit issue Cost- benefit issue  Minimize interruptions so higher risk alerts are accepted more often… yet little benefit seen with non-interruptive alerts

23. Impact of Tiering on Inpatient DDI Alerts Two academic medical centers Two academic medical centers Same knowledge base Same knowledge base Site A used 3 tiers Site A used 3 tiers Site B had all of the alerts as interruptive (Level 2) Site B had all of the alerts as interruptive (Level 2) Overall alert acceptance higher at tiered site (29% vs 10%, p<.001) Overall alert acceptance higher at tiered site (29% vs 10%, p<.001) Paterno, et al. Unpublished data.

24. Tiered Inpatient DDI Acceptance Rates Level 1 Acceptance rates Level 1 Acceptance rates  100% (hard stop) vs 34% (not a hard stop) Level 2 Acceptance rates Level 2 Acceptance rates  29% vs 11%  Likely higher at tiered site since less alert fatigue because fewer interruptive alerts

25. Conclusions Streamlined knowledge bases and tiered alerting have higher acceptance rates Streamlined knowledge bases and tiered alerting have higher acceptance rates  Especially for very high risk alerts Non-interruptive alerts may have little value Non-interruptive alerts may have little value What is our ideal acceptance rate?? Sensitivity/specificity? Best way to display? What is our ideal acceptance rate?? Sensitivity/specificity? Best way to display? More work needs to be done to maximize the clinical benefits More work needs to be done to maximize the clinical benefits Sharing of streamlined knowledge should be widespread Sharing of streamlined knowledge should be widespread  No need to reinvent the wheel