Duality of error Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany State University of New York Public Administration and Policy PAD634 Judgment and Decision Making Behavior Copyright © Thomas R. Stewart

duality-of-error.ppt2 Example of a False Positive (New York Times)

duality-of-error.ppt3 Base rate = 20/100 =.20 Decision table terminology: Data for an imperfect categorical forecast over 100 days (uncertainty)

duality-of-error.ppt4 Decision A Threshold model Even if a decision or forecast is categorical, it is based on a degree of belief or judgment Judgment LowHigh Threshold 1 Decision B Threshold 2 Decision C

duality-of-error.ppt5 Uncertainty, Judgment, Decision, Error Taylor-Russell diagram – Decision cutoff – Criterion cutoff (linked to base rate) – Correlation (uncertainty) – Errors False positives (false alarms) False negatives (misses)

r =.50 Decision threshold Act Don’t Act

r =.50 Criterion threshold Action is appropriate Action is inappropriate

True negatives True positives r =.50 Decision threshold Criterion threshold False positives False negatives Taylor-Russell diagram

True negatives True positives r =.50 Decision threshold Criterion threshold False positives False negatives Taylor-Russell diagram

True negatives True positives r =.50 Decision threshold Criterion threshold False positives False negatives Taylor-Russell diagram

True negatives True positives r =.95 Decision threshold Criterion threshold False negatives False positives Taylor-Russell diagram

duality-of-error.ppt12

duality-of-error.ppt13 Tradeoff between false positives and false negatives

duality-of-error.ppt14 Uncertainty, Judgment, Decision, Error Another view: ROC analysis – Decision cutoff – False positive proportion – True positive proportion – A z measures forecast quality

duality-of-error.ppt15 ROC Curve

duality-of-error.ppt16 Problem: Optimal decision cutoff Given that it is not possible to eliminate both false positives and false negatives, what decision cutoff gives the best compromise? – Depends on values – Depends on uncertainty – Depends on base rate Decision analysis is one optimization method.

duality-of-error.ppt17 Example: Weather forecaster’s decision to warn the public about an approaching storm

duality-of-error.ppt18 Decision tree (for tornado warning example)

duality-of-error.ppt19 Expected value Expected Value = P(O 1 )V(O 1 ) + P(O 2 )V(O 2 ) + P(O 3 )V(O 3 ) + P(O 4 )V(O 4 ) V(O i ) is the value of outcome i P(O i ) is the probability of outcome i

duality-of-error.ppt20 Expected value One of many possible decision making rules Used here for illustration because it’s the basis for decision analysis Intended to illustrate principles

duality-of-error.ppt21 Expected value Assign each point a number representing its value. The expected value is the average (mean) of those values. Assume all points in a quadrant have the same value.

duality-of-error.ppt22 Expected value True positives False positives False negatives True negatives Expected value = 690/14 = 49.3 Don’t ActAct Action is not appropriate Action is appropriate

duality-of-error.ppt23 Expected value True positives False positives False negatives True negatives Expected value = 680/14 = 48.6 (.7 less than previously) Action is not appropriate Action is appropriate Don’t ActAct

duality-of-error.ppt24 Where do the values come from?

duality-of-error.ppt25 Values depend on many factors Event – Hail – Tornado Time – Season – Day of week – Time of day Location – Population – Highways

duality-of-error.ppt26 Descriptions of outcomes True positive (hit--a warning is issued and the storm occurs as predicted) – Damage occurs, but people have a chance to prepare. Some property and lives are saved, but probably not all. False positive (false alarm--a warning is issued but no storm occurs) – No damage or lives lost, but people are concerned and prepare unnecessarily, incurring psychological and economic costs. Furthermore, they may not respond to the next warning.

duality-of-error.ppt27 Descriptions of outcomes (cont.) False negative (miss--no warning is issued, but the storm occurs) – People do not have time to prepare and property and lives are lost. NWS is blamed. True negative (no warning is issued and storm occurs) – No damage or lives lost. No unnecessary concern about the storm.

duality-of-error.ppt28 Values depend on your perspective Forecaster Emergency manager Public official Property owner Business owner Many others...

duality-of-error.ppt29 Which is the best outcome?  True positive?  False positive?  False negative?  True negative? Give the best outcome a value of 100.

duality-of-error.ppt30 Which is the worst outcome?  True positive?  False positive?  False negative?  True negative? Give the worst outcome a value of 0.

duality-of-error.ppt31 Rate the remaining two outcomes  True positive?  False positive?  False negative?  True negative? Rate them relative to the worst (0) and the best (100)

duality-of-error.ppt32 Interpreting values Compare pairs where the weather is the same but the forecast is different. Weather is not severe True negative - False positive = penalty for false alarm Weather is severe True positive - False negative = benefit of correct forecast

duality-of-error.ppt33 Interpreting values Decision Don’t warnWarn Event Tornado No tornado False NegativeTrue positive True negativeFalse positive 100 0TP FP FP = penalty for false alarm TP - 0 = benefit of warning

duality-of-error.ppt34 Values reflect different perspectives True positive? False positive? False negative? True negative? Perspective Measuring values

duality-of-error.ppt35 Expected value Expected Value = P(O 1 )V(O 1 ) + P(O 2 )V(O 2 ) + P(O 3 )V(O 3 ) + P(O 4 )V(O 4 ) V(O i ) is the value of outcome i P(O i ) is the probability of outcome i

duality-of-error.ppt36 Expected value depends on the decision threshold

duality-of-error.ppt37 Expected value depends on the value perspective

duality-of-error.ppt38 Value differences can lead to disagreement and conflict Best threshold for person A Best threshold for person B

duality-of-error.ppt39 Whose values? Forecasting weather is a technical problem. Issuing a warning to the public is a social act. Any warning policy has an implicit set of values. Should those values be made explicit and subject to public scrutiny?

duality-of-error.ppt40 Conclusion Choosing the cutoff – Value tradeoffs are unavoidable. – Decisions are based on values that should be critically examined.

duality-of-error.ppt41 Injustice (welfare problem, see Hammond, p. 56) The deserving The undeserving RejectedRewarded Injustice to individuals Injustice to society

Uncertainty Judgment Error Injustice Conflict

duality-of-error.ppt43 Implications and predictions Implications: – Tradeoff between different kinds of errors is inevitable. – Science can’t dictate the correct decision. – Explicit consideration of values is important, but is rarely done. – Solving one kind of problem creates another. – Detecting rare events means many false alarms. – Small gains in predictability can have large benefits. Predictions: – For rare events, reaction to high visibility false (negative/positive) is likely to create a large number of less visible false (positives/negatives). – Decision thresholds are likely to cycle.

duality-of-error.ppt44 Major points to remember Uncertainty creates the need for judgment Separate judgmental accuracy from the decision cutoff Uncertainty leads to two kinds of error – False positive/false negative tradeoff Error creates injustice Under uncertainty, value differences lead to conflict Reducing uncertainty reduces conflict

duality-of-error.ppt45 Disposition Decisions in Psychiatric Emergency Rooms False negatives: Inappropriate releases – Occasionally lead to violence against others – Increase the risk of suicide – Increase the risk of injury or death due to accidents – Place stress and extra burdens on community support systems – Aggravate psychiatric symptoms – Patient does not obtain proper treatment

duality-of-error.ppt46 Disposition Decisions in Psychiatric Emergency Rooms False positives: Inappropriate admissions – Can be disruptive and stigmatizing – May lead to the loss of jobs, housing, and child custody – Average inpatient admission costs nearly $10,000

duality-of-error.ppt47 Disposition Decisions in Psychiatric Emergency Rooms Taylor-Russell analysis – Base rate – Selection rate – Judgmental accuracy – Costs and benefits of outcomes

duality-of-error.ppt48 Disposition Decisions in Psychiatric Emergency Rooms No policies regarding psychiatric emergency room admissions can be meaningfully evaluated without simultaneously considering all four factors. Unfortunately, few public policy discussions discuss all four factors. This means that implicit assumptions about omitted factors have been made. These buried assumptions may give rise to debates and disputes that will be difficult to resolve, unless they are brought to the surface and explicated.

duality-of-error.ppt49 Base rate What percentage of persons who present at psychiatric ERs would benefit from in-patient treatment and, thus, "ought" to be admitted? – Difficult to determine – No “gold standard” – Initial assumption: 50% – Requires sensitivity analysis Psychiatric ERs

duality-of-error.ppt50 Selection rate Varies substantially across sites Initial assumption: 50% Approximates the average rate found in research to date Psychiatric ERs

duality-of-error.ppt51 Judgmental accuracy No data due to absence of a “gold standard” Study by Bruce Way found that the correlation among psychiatrists recommended dispositions was.34. If this is an estimate of reliability, then accuracy can be no higher than the square root of.34 =.58 Psychiatric ERs

duality-of-error.ppt52 Cost and benefits of outcomes Rather than trying to develop monetary estimates, the present analysis relies on a decision analytic approach, in which each possible outcome is assigned a score from 0 to 100, reflecting its relative desirability. Psychiatric ERs

duality-of-error.ppt53 Which is the best outcome?  True positive?  False positive?  False negative?  True negative? Give the best outcome a value of 100. Psychiatric ERs

duality-of-error.ppt54 Which is the worst outcome?  True positive?  False positive?  False negative?  True negative? Give the worst outcome a value of 0. Psychiatric ERs

duality-of-error.ppt55 Rate the remaining two outcomes  True positive?  False positive?  False negative?  True negative? Rate them relative to the worst (0) and the best (100) Psychiatric ERs

duality-of-error.ppt56 Value perspectives Psychiatric ERs True positive False positive True negative False negative Perspective

duality-of-error.ppt57 Taylor-Russell analysis If the assumptions regarding the underlying base rate, the payoff function, and the degree of predictive accuracy are approximately correct, is the admission rate of 50% optimal, in terms of maximizing total value? In light of the substantial variation across institutions in observed admission rates – from less than 10% to more than 90% – this is an extremely pertinent question, with substantial potential policy implications. Left to their own devices, different institutions have come up with quite different answers about what percentage of potential patients is appropriate to admit. Psychiatric ERs

duality-of-error.ppt58 Taylor-Russell analysis Injustice – To individuals – To society Cycles of differential injustice? Optimal cutoff and admission rate Sensitivity to base rate Improving judgmental accuracy Psychiatric ERs

duality-of-error.ppt59 Rationing or quotas What happens if there are only a limited number of beds to be filled? The cutoff is determined by the number of beds available. Resource constraints dictate the value tradeoffs Psychiatric ERs

duality-of-error.ppt60 Left out of Taylor-Russell Creating new alternatives that may eliminate some of the tough tradeoffs. Design and planning Dynamic properties of decision or environments The potential effects of testing and cutoffs and standards on the points in the graphs (e.g., measures designed to increase airline security have a deterrent effect. Also, potential terrorists develop countermeasures) Implementation issues Cost of decision processes Amount of information -- how much is enough? Cue intercorrelations, causal texture Outcomes in the same quadrant may have different values Multidimensional nature of outcomes