A “bottom up” approach to curating electronic order sets S22: Identifying Medication Risk Ron Li, MD Clinical Informatics Fellow Stanford University School of Medicine ronl@stanford.edu

Disclosure I and my spouse/partner have no relevant relationships with commercial interests to disclose. AMIA 2017 | amia.org

Understanding how clinicians deliver care is an important piece of the Learning Health Care System Institute of Medicine. Best Care at Lower Cost. (2012)

Automatically generated ethnographic data

Digital trace data in healthcare

Adverse drug events are responsible for up to 770,000 inpatient injuries and deaths annually, most of which are from prescribing errors. These adverse events affect nearly 5% hospitalized patients in the United States. AHRQ Patient Safety Network. Computerized provider order entry. AHRQ PSNet Patient Safety Primer 2015 (2015). Available at: https://psnet.ahrq.gov/primers/primer/6/.

Clinical Decision Support Clinical decision making Unintended consequences of clinical decision support (CDS) on ordering medications Clinical Decision Support How is medication ordering affected? Clinical decision making Workflow

Errors mediated by automation bias Users tend to over-accept and over-rely on computer output “as a heuristic replacement of vigilant information seeking and processing, which can lead to both errors of omission and commission. Error of commission: a CDS generates an incorrect recommendation, prompting clinician to order a medication he otherwise would not have ordered Error of omission: a CDS fails to alert a clinician of a patient’s risk for severe sepsis. The clinician, who is accustomed to relying on the CDS alert, does not do as careful of an assessment and misses the diagnosis. Goddard, K., Roudsari, A. & Wyatt, J. C. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J. Am. Med. Informatics Assoc. 19, 121–127 (2012).

How does automation bias affect the use of the electronic order sets? Electronic order sets are present in more than 80% of inpatient EHRs and are important for standardizing/increasing efficiency of workflow and clinical decision making. Does automation bias cause order sets to also promote incorrect mediation ordering?

Order sets are currently curated based on the ad hoc insights of a small group of people: expert opinion, anecdotal experiences, guidelines, incident reports. This “top down” approach may result in a inaccurate and biased understanding of how order sets are affecting clinician ordering behavior, resulting in sub-optimally curated order sets. Aim: to develop a “bottom up” approach using EHR audit trail data to systematically capture ordering errors from order sets

Methods All inpatient medication orders from order sets at Stanford Hospital were identified from 2008-2017. Frequency of medication prescribing errors (in the form of near misses) estimated by frequency of orders discontinued within 30 minutes of ordering (“dw30m”)1. Medication prescribing error rate calculated for each order 1) when part of any order set and 2) when part of a specific order-order set pair. Medication error rates are compared to the error rates for when the medication was ordered not from an order set (relative risk). Koppel, R. et al. Identifying and Quantifying Medication Errors : Evaluation of Rapidly Discontinued Medication Orders Submitted to a Computerized Physician Order Entry System. J. Am. Med. Informatics Assoc. 15, 461–465 (2008).

Inpatient medication orders at Stanford Hospital (2008-2017) All data from 2008-2017 With at least one dw30m order # patient encounters 195,588 89,354 # orders 10,578,855 335,084 # times order set was used 745,621 41,329 # orders from order sets 3,395,541 67,560

Medication error rates per medication when ordered from order set After excluding medications with the bottom 25th percentile ordering frequencies, there were 519 unique medications with at least one order discontinued within 30 minutes. Medication # times dw30m from orderset Total # times ordered from orderset Dw30m rate Ondansetron 4mg Inj 3480 251,825 1.4% Fentanyl 50 mcg Inj 1960 120,156 1.6% Enoxaparin 40mg SC 937 28,606 3.3% Cefazolin 2g IV PGBK 315 7,616 4.1% Morphine 2mg IV Inj 288 12,653 2.3%

Min 0.2% Median 2.6% Max 29% IQR 3%

Error rates tend to be lower with order sets, but higher for a subset of medications Relative Risk < 1 Relative Risk >1 Min 0.06 Median 0.81 Max 23.1 IQR 0.77

Total # times pair was ordered Identifying the medication-order set pairs most prone to prescribing errors After excluding order sets and medication-order set pairs with the bottom 25th percentile ordering frequencies, there were 2678 unique medication-order set pairs with at least one order discontinued within 30 minutes Medication Order set # times dw30m Total # times pair was ordered Dw30m rate Ondansetron 4mg Inj Anesthesia PACU 1,066 71,471 1.5% Fentanyl 50 mcg Inj Neurosurgery ICU 95 7494 1.3% Enoxaparin 40mg SC Medicine General Admit 512 11,967 4.3% Cefazolin 2g IV PGBK Pre-admission/Pre-op 245 9,026 2.7% Morphine 2mg IV Inj 96 5924 1.6%

Min 0.05% Median 2.0% Max 42.5% IQR 2%

Relative Risk < 1 Relative Risk >1 Min 0.1 Median 0.67 Max 46.9 IQR 0.8

Data driven order set curation Medication Order set # times ordered # times dw30m Dw30m rate Hydromorphone 2 mg/ml inj Bone marrow and aspirate 40 17 43% Amiodarone infusion periph ICU cardiac surg postop 34 14 42% Cefoxitin 1 gram IV Surg general admit 43 40% Cefazolin 1 gram inj GU general admit 188 73 39% Calcium chloride 8 g/1000 ml NS IV infusion Continuous renal replacement therapy(CRRT) 1162 425 37%

Data-driven order set curation Medication Order set # times ordered # times dw30m Relative risk of dw30m Hydromorphone 2 mg/ml inj Bone marrow and aspirate 40 17 46.96 Ondansetron 4 mg/2 ml inj Ane ECT PACU orders 679 121 9.22 Influenza vaccine Med alcohol withdrawal 248 28 8.93 Aztreonam 2 gram/100ml IVB Med cystic fibrosis admit 57 4 8.70 Hydrocodone-acetaminophen 5-325 mg PO Nursing triage pain management protocol 12 8.66

Discussion Medication prescribing errors from order sets are relatively uncommon, but are high (as high as 29%) for a subset of medications. Most medications have lower prescribing error rates when ordered from an order set, but a subset of medications are more likely to be prescribed in error when from order sets. These data can be used to target medications and order sets for further investigation during the order set review process. Limitations: results are unadjusted for differences between order set/non-order set medication orders and are from a single institution

Acknowledgements Jonathan Chen, MD PhD Christopher Sharp, MD Jason Wang Connie Taylor, RN Stanford Clinical Informatics Fellowship Stanford Department of Medicine Stanford Division of Biomedical Informatics Research Stanford Research IT

AMIA is the professional home for more than 5,400 informatics professionals, representing frontline clinicians, researchers, public health experts and educators who bring meaning to data, manage information and generate new knowledge across the research and healthcare enterprise. AMIA 2017 | amia.org

Email me at: ronl@stanford.edu Thank you! Email me at: ronl@stanford.edu