Doing better: Some practical advice

How to see past our biases and get at the truth? Data-driven, quantitative risk modeling and decision-making

Ten barriers to making good unaided decisions Plunging in Frame blindness Lack of frame control Overconfidence in your judgment Shortsighted shortcuts Shooting from the hip Group failure Fooling yourself about feedback Not keeping track Failure to audit your decision process

Plunging in Symptoms Cures Solving the wrong problem Not solving the right one Not solving anything; analysis-paralysis Omitting key objectives, alternatives, consequences Solving it badly (missing a clearly better solution) Cures Before starting to make decisions, identify: What is the crux of the problem? Step back: Answer, How should decisions like this one be made? (“meta-decision”) Control framing deliberately and well. Choose frames to improve decisions. What to focus on, what to ignore?

Plunging in Symptoms Cures Solving wrong problem Not solving right one Not solving anything; analysis-paralysis Omitting key objectives, alternatives, consequences Solving it badly (missing a clearly better solution) Cures Before starting to make decisions, identify: What is the crux of the problem? Step back: Answer, How should decisions like this one be made? (“meta-decision”) Framing = decision boundaries reference points (“success”) measures guiding metaphors Examples: sunk-cost fallacy gain vs. loss frames endowment effect house money effect

Examples of frame blindness Surgery vs. radiation therapy Comparing outcomes to poor reference groups Walk 2 blocks to save $30 on: (a) A new $70 watch; or (b) A new $800 computer

Improving decision-making Select frames deliberately and well Frames drive decisions! Collect and use information effectively What information should you seek? Ask disconfirming questions. Experiment! Fault trees, scenarios, prospective hindsight How should you express and assess beliefs? Use confidence intervals, calibration Good group decisions need productive conflict Use objective models to improve decisions Learn effectively from experience Avoid: fast but stupid decision rules (e.g., screening & ranking) substituting your confident judgments for factual data confirmation bias information overload, pursuit of useless information uncritical trust in traditional rules of thumb recency, saliency, anchoring & availability biases

Collect and use information effectively What information should you seek? What data are relevant?  Wason selection Ask disconfirming questions. Experiment! Fault trees Scenarios prospective hindsight How should you express and assess beliefs? Use confidence intervals, calibration Good group decisions need productive conflict

Use objective (data-driven) quantitative models to improve decisions Creating models Validating models Critiquing models: How good is performance? Face validity Descriptive validity Predictive validity Implementing models Refining and updating

Why bother to use objective models? They work! + Framing effects distort our perceptions and choices Allais paradox (1953) Repeated gambles Use of relative instead of absolute changes Excessive loss aversion Reference dependence Irrational disappointment, regret

How to do it? Creating models Validating models Critiquing models Learning from data & knowledge Predicting, assessing, adapting Refining and updating Validating models Critiquing models Implementing models Using models effectively

Learning more effectively Avoid biases that impair learning confirmation bias hindsight bias. (We did not “know it all along”!) illusion of control self-serving attribution biases Objectively attribute effects to their causes Value information correctly Be willing to explore and experiment Know when to stop investigating, and decide! Flexible, adaptive learning

Obstacles to learning: Rationalizaton biases Misremember what really happened, to make ourselves look better Blame failures on others, or unforeseeable events Self-serving biases and explanations “Success is due to skill, failure to bad luck” Example: Predictable cost overruns Re-interpret original predictions to make them sound better (in light of the facts) than they were Change preferences to make failure ok “Sour grapes” Cures Specify goals, criteria, extent of control, in advance Debrief after project Review successes and failures periodically. For repeated decisions, use statistical analysis to quantify contributions of chance and choice

Improving feedback Close gaps in feedback loops Be aware of treatment/placebo effects Correct for random effects (“noise”) Keep track of relevant information Use information: Don’t ignore useful feedback! Design experiments to learn how to do better

Ten barriers to making good unaided decisions Plunging in Frame blindness Lack of frame control Overconfidence in your judgment Shortsighted shortcuts Shooting from the hip Group failure Fooling yourself about feedback Not keeping track Failure to audit your decision process