Emergency Physician Risk Estimates and Admission Decisions for Chest Pain: A Web- Based Scenario Study  David L. Schriger, MD, MPH, Michael Menchine, MD, MPH, Warren Wiechmann, MD, MBA, Guy Carmelli, MD  Annals of Emergency Medicine  Volume 72, Issue 5, Pages 511-522 (November 2018) DOI: 10.1016/j.annemergmed.2018.03.003 Copyright © 2018 American College of Emergency Physicians Terms and Conditions

Figure 1 Histogram and box plot (on a log scale) of the distribution of ratings for the probability of serious outcome (red) and acute coronary syndrome (blue) for 208 raters rating 5 scenarios. The distributions are similar, with the mode and median being 1 in 100 for each. Circles in the box plots represent the geometric mean for the distributions. The ends of the bars are the upper and lower adjacent values. Despite all 5 scenarios’ involving patients with negative troponin results and with no sign of ST-segment elevation myocardial infarction, risk estimates span 5 orders of magnitude, from 1 in 100,000 to 1 in 1. The 2 lines represent the cumulative percentage of patients whom the raters admitted at the corresponding risk level (to avoid instability, the curves begin from the left once an N of 20 is reached). For example, approximately 25% of patients whose probability of event was rated at 1 in 1,000 or less were admitted regardless of whether the endpoint was serious outcome or ACS. At 1 in 100, there was an approximately 25% increase in admission rate. Overall, 75% of scenario patients were admitted. The 2 lines follow a similar pattern, with, as expected, a higher admission rate at any given value by raters judging “probability of a serious outcome” than those judging “probability of ACS.” The difference, however, is not as large as one would expect, given that the literature suggests that a serious outcome occurs in far less than 10% of patients with non–ST-segment elevation myocardial infarction ACS. Annals of Emergency Medicine 2018 72, 511-522DOI: (10.1016/j.annemergmed.2018.03.003) Copyright © 2018 American College of Emergency Physicians Terms and Conditions

Figure 2 Waterfall plot showing 1 vertical line per rater. The endpoints of each line depict the probability estimate for the discharged patient with the highest probability (anchored on the wider gray line and ordered left to right from lowest to highest) and the probability estimate for the admitted patient with the lowest probability. Somewhere along this vertical line lies the physician’s admission threshold, the probability at which the disposition switches from discharge to admit. Ten of the 208 raters (lines go down and have magenta endpoints) were seemingly illogical; they discharged a patient who they thought was at higher probability for the outcome than a patient they admitted. Black dots (24 total) indicate raters for whom the 2 probability estimates were the same even though one patient was discharged and the other admitted. Red dots (28 total) indicate raters who discharged all 5 patients; the dot indicates the risk estimate for the highest-risk patient. Conversely, the blue dot represents the lowest rating of the one rater who admitted all 5 patients. The admission boundaries are positioned either between 2 labeled y axis values (eg, between 1 in 100,000 and 1 in 10,000) or on a specific value. They represent the cumulative percentage of raters who had that threshold. For example, in the left-hand panel (risk of serious outcome) between 2% and 15% of subjects had a threshold below 1 in 10,000, and between 3% and 20% of subjects had a threshold at or below 1 in 10,000. According to this plot, (1) 95% of the physician raters were logical in their ratings and dispositions; (2) at least 72% of physicians had admission thresholds of 1 in 100 (1%) chance of ACS or lower; and (3) although admission rates for a given probability of serious outcome were higher than for the same probability of acute coronary syndrome, the difference was smaller than expected according to the frequency of serious outcomes in patients with ACS. L, Lower boundary; U, upper boundary; admission rate boundaries, minimum and maximum percentage of physicians whose admission threshold was that value or lower. Annals of Emergency Medicine 2018 72, 511-522DOI: (10.1016/j.annemergmed.2018.03.003) Copyright © 2018 American College of Emergency Physicians Terms and Conditions

Figure 3 Distribution of qualitative risk ratings for each scenario, stratified by whether the rater indicated that he or she would discharge or admit the patient. N is the number of ratings represented in each stack. The figure demonstrates that it is rare for a patient to be discharged if he or she is deemed to be at anything more than low risk. Only 7% of patients deemed to be at moderate risk were discharged. Justification for combining ratings of “risk of ACS” and “risk of serious outcome” in this graph is provided in Figure E2 (available online at http://www.annemergmed.com), which shows that the patterns are virtually identical for the 2 outcomes rated. Dispo, Disposition. Annals of Emergency Medicine 2018 72, 511-522DOI: (10.1016/j.annemergmed.2018.03.003) Copyright © 2018 American College of Emergency Physicians Terms and Conditions

Figure 4 Histogram and box plot estimates of the probability of the outcome for patients with the indicated qualitative risk rating. The number of patients with that qualitative risk rating is shown in parentheses. The red marker within the box plot is the geometric mean of the probability estimates. With rare exception, raters used “very low risk” and “low risk” only when the quantitative risk was below 1 in 100 (the red dashed line). When deemed “moderate,” 23% of probability estimates were below 1 in 100, 45% were above, and 32% were at that value. Thirty-one percent of “high risk” ratings were rated at 1 in 100 or less. Annals of Emergency Medicine 2018 72, 511-522DOI: (10.1016/j.annemergmed.2018.03.003) Copyright © 2018 American College of Emergency Physicians Terms and Conditions

Figure 5 Each panel represents data for a single scenario, from lowest risk (1) to highest risk (5). Each dot represents one rating of a scenario comparing the rater’s score on the fear-of-malpractice scale versus the probability estimate. The lines were created from a locally weighted scatter plot smoothing regression, which makes no assumptions about the shape of the relationship. There was no important relationship between these variables for any scenario. We graphed the same variables, but using each raters’ geometric mean risk rating, in Figure E3 (available online at http://www.annemergmed.com). Annals of Emergency Medicine 2018 72, 511-522DOI: (10.1016/j.annemergmed.2018.03.003) Copyright © 2018 American College of Emergency Physicians Terms and Conditions