1. A “bottom up” approach to curating electronic order sets
S22: Identifying Medication Risk
Ron Li, MD
Clinical Informatics Fellow
Stanford University School of Medicine

2. Disclosure
I and my spouse/partner have no relevant relationships with commercial interests to disclose.
AMIA | amia.org

3. Understanding how clinicians deliver care is an important piece of the Learning Health Care System
Institute of Medicine. Best Care at Lower Cost. (2012)

4. Automatically generated ethnographic data

5. Digital trace data in healthcare

6. Adverse drug events are responsible for up to 770,000 inpatient injuries and deaths annually, most of which are from prescribing errors.
These adverse events affect nearly 5% hospitalized patients in the United States.
AHRQ Patient Safety Network. Computerized provider order entry. AHRQ PSNet Patient Safety Primer 2015 (2015). Available at:

7. Clinical Decision Support Clinical decision making
Unintended consequences of clinical decision support (CDS) on ordering medications
Clinical Decision Support
How is medication ordering affected?
Clinical decision making
Workflow

8. Errors mediated by automation bias
Users tend to over-accept and over-rely on computer output “as a heuristic replacement of vigilant information seeking and processing, which can lead to both errors of omission and commission.
Error of commission: a CDS generates an incorrect recommendation, prompting clinician to order a medication he otherwise would not have ordered
Error of omission: a CDS fails to alert a clinician of a patient’s risk for severe sepsis. The clinician, who is accustomed to relying on the CDS alert, does not do as careful of an assessment and misses the diagnosis.
Goddard, K., Roudsari, A. & Wyatt, J. C. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J. Am. Med. Informatics Assoc. 19, 121–127 (2012).

10. How does automation bias affect the use of the electronic order sets?
Electronic order sets are present in more than 80% of inpatient EHRs and are important for standardizing/increasing efficiency of workflow and clinical decision making. Does automation bias cause order sets to also promote incorrect mediation ordering?

11. Order sets are currently curated based on the ad hoc insights of a small group of people: expert opinion, anecdotal experiences, guidelines, incident reports.
This “top down” approach may result in a inaccurate and biased understanding of how order sets are affecting clinician ordering behavior, resulting in sub-optimally curated order sets.
Aim: to develop a “bottom up” approach using EHR audit trail data to systematically capture ordering errors from order sets

12. Methods
All inpatient medication orders from order sets at Stanford Hospital were identified from Frequency of medication prescribing errors (in the form of near misses) estimated by frequency of orders discontinued within 30 minutes of ordering (“dw30m”)1. Medication prescribing error rate calculated for each order 1) when part of any order set and 2) when part of a specific order-order set pair. Medication error rates are compared to the error rates for when the medication was ordered not from an order set (relative risk).
Koppel, R. et al. Identifying and Quantifying Medication Errors : Evaluation of Rapidly Discontinued Medication Orders Submitted to a Computerized Physician Order Entry System. J. Am. Med. Informatics Assoc. 15, 461–465 (2008).

13. Inpatient medication orders at Stanford Hospital (2008-2017)
All data from
With at least one dw30m order
# patient encounters
195,588
89,354
# orders
10,578,855
335,084
# times order set was used
745,621
41,329
# orders from order sets
3,395,541
67,560

14. Medication error rates per medication when ordered from order set
After excluding medications with the bottom 25th percentile ordering frequencies, there were 519 unique medications with at least one order discontinued within 30 minutes.
Medication
# times dw30m from orderset
Total # times ordered from orderset
Dw30m rate
Ondansetron 4mg Inj
3480
251,825
1.4%
Fentanyl 50 mcg Inj
1960
120,156
1.6%
Enoxaparin 40mg SC
937
28,606
3.3%
Cefazolin 2g IV PGBK
315
7,616
4.1%
Morphine 2mg IV Inj
288
12,653
2.3%

15. Min
0.2%
Median
2.6%
Max
29%
IQR 3%

16. Error rates tend to be lower with order sets, but higher for a subset of medications
Relative Risk < 1
Relative Risk >1
Min
0.06
Median
0.81
Max
23.1
IQR
0.77

17. Total # times pair was ordered
Identifying the medication-order set pairs most prone to prescribing errors
After excluding order sets and medication-order set pairs with the bottom 25th percentile ordering frequencies, there were 2678 unique medication-order set pairs with at least one order discontinued within 30 minutes
Medication
Order set
# times dw30m
Total # times pair was ordered
Dw30m rate
Ondansetron 4mg Inj
Anesthesia PACU
1,066
71,471
1.5%
Fentanyl 50 mcg Inj
Neurosurgery ICU 95
7494
1.3%
Enoxaparin 40mg SC
Medicine General Admit
512
11,967
4.3%
Cefazolin 2g IV PGBK
Pre-admission/Pre-op
245
9,026
2.7%
Morphine 2mg IV Inj 96
5924
1.6%

18. Min
0.05%
Median
2.0%
Max
42.5%
IQR 2%

19. Relative Risk < 1 Relative Risk >1 Min 0.1 Median 0.67 Max 46.9
IQR
0.8

20. Data driven order set curation
Medication
Order set
# times ordered
# times dw30m
Dw30m rate
Hydromorphone 2 mg/ml inj
Bone marrow and aspirate 40 17
43%
Amiodarone infusion periph
ICU cardiac surg postop 34 14
42%
Cefoxitin 1 gram IV
Surg general admit 43
40%
Cefazolin 1 gram inj
GU general admit
188 73
39%
Calcium chloride 8 g/1000 ml NS IV infusion
Continuous renal replacement therapy(CRRT)
1162
425
37%

21. Data-driven order set curation
Medication
Order set
# times ordered
# times dw30m
Relative risk of dw30m
Hydromorphone 2 mg/ml inj
Bone marrow and aspirate 40 17
46.96
Ondansetron 4 mg/2 ml inj
Ane ECT PACU orders
679
121
9.22
Influenza vaccine
Med alcohol withdrawal
248 28
8.93
Aztreonam 2 gram/100ml IVB
Med cystic fibrosis admit 57 4
8.70
Hydrocodone-acetaminophen mg PO
Nursing triage pain management protocol 12
8.66

22. Discussion
Medication prescribing errors from order sets are relatively uncommon, but are high (as high as 29%) for a subset of medications.
Most medications have lower prescribing error rates when ordered from an order set, but a subset of medications are more likely to be prescribed in error when from order sets.
These data can be used to target medications and order sets for further investigation during the order set review process.
Limitations: results are unadjusted for differences between order set/non-order set medication orders and are from a single institution

23. Acknowledgements
Jonathan Chen, MD PhD Christopher Sharp, MD Jason Wang Connie Taylor, RN Stanford Clinical Informatics Fellowship Stanford Department of Medicine Stanford Division of Biomedical Informatics Research Stanford Research IT

24. AMIA is the professional home for more than 5,400 informatics professionals, representing frontline clinicians, researchers, public health experts and educators who bring meaning to data, manage information and generate new knowledge across the research and healthcare enterprise.
AMIA | amia.org

25. Email me at: ronl@stanford.edu
Thank you!
me at: