1. Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany State University of New York T.STEWART@ALBANY.EDU Public Administration and Policy PAD634 Judgment and Decision Making Behavior

2. components-of-skill.ppt2 Fundamental tenet of correspondence research "Human competence in making judgments and decisions under uncertainty is impressive. Sometimes performance is not. Why? Because sometimes task conditions degrade the accuracy of judgment." Hammond, K. R. (1996). Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. New York, Oxford University Press (p. 282).

3. components-of-skill.ppt3 Brunswik's lens model Distal variable Judgment X Cues Y e Y s

4. components-of-skill.ppt4 True Descriptors Subjective Judgment Cues Expanded lens model Distal variable

5. Environmental predictability Fidelity of the information system Match between environment and judge Reliability of information acquisition Reliability of information processing True Descriptors Subjective Judgment Cues components-of-skill.ppt5 Components of skill and the lens model Distal variable

6. components-of-skill.ppt6  Area under ROC curve  Correlation (r YO ) O is the observation, or “gold standard” Y is the judgment  Mean square error:  Skill score: Alternative measures of performance (correspondence approach)

7. components-of-skill.ppt7 Decomposing accuracy--the skill score Murphy (1988): Skill = Correlation - Conditional bias - Unconditional bias score (regression bias) (base rate bias)

8. components-of-skill.ppt8 Decomposing accuracy: The Lens Model Equation (Tucker 1964)

10. components-of-skill.ppt10 Decomposition of skill score Skill score =

11. components-of-skill.ppt11

12. components-of-skill.ppt12 1. Environmental predictability Environmental predictability is conditional on current knowledge and information. It can be improved through research that results in improved information and improved understanding of environmental processes. Environmental predictability determines an upper bound on performance and therefore indicates how much improvement is possible through attention to other components. Components of skill

13. components-of-skill.ppt13 Environmental predictability limits accuracy of judgment

14. components-of-skill.ppt14 2. Fidelity of information system Judgmental skill may be degraded if the information system that brings data to the judge does not accurately represent actual conditions, i.e., if the cues do not accurately measure the true descriptors. Fidelity of the information system refers to the quality, not the quantity, of information about the cues that are currently being used. Fidelity is improved by developing better measures, e.g., though improved instrumentation or increased density in space or time. Components of skill

15. components-of-skill.ppt15 3. Match between environment and judge The match between the model of the judge and the environmental model is an estimate of the potential skill that the judge's current strategy could achieve if the environment were perfectly predictable (given the cues) and the judgments were unbiased and perfectly reliable. This component might be called “knowledge.” It is addressed by training and experience. If the judge learns to rely on the most relevant information and ignore irrelevant information, this component will generally be good. Components of skill

16. components-of-skill.ppt16 Reliability Reliability is high if identical conditions produce identical judgments. Humans are rarely perfectly reliable. There are two sources of unreliability: – Reliability of information acquisition – Reliability of information processing Components of skill

17. components-of-skill.ppt17 Reliability Reliability decreases as amount of information increases. Components of skill Theoretical relation between amount of information and accuracy of judgment

18. components-of-skill.ppt18 Reliability decreases as environmental predictability decreases Components of skill 18

19. components-of-skill.ppt19 4. Reliability of information acquisition Reliability of information acquisition is the extent to which the judge can reliably interpret the objective cues. It is improved by organizing and presenting information in a form that clearly emphasizes relevant information. Components of skill

20. components-of-skill.ppt20 Visual display of information Components of skill Books by Edward R. Tufte The Visual Display of Quantitative Information. Envisioning Information Visual Explanations Beautiful Evidence The Cognitive Style of Powerpoint: Pitching Out Corrupts Within Cheshire, Connecticut: Graphics Press

21. components-of-skill.ppt21 Visual display of information Components of skill Image from the videotape “Study of a Numerically modeled severe storm,” National Center for Supercomputing applications, University of Illinois (Tufte, Visual Explanations, p. 20) Tufte’s questions: How big is the cloud? What direction is it moving? What are the dimensions of the grid?

22. components-of-skill.ppt22 Visual display of information Components of skill Redesign by Tufte and Bushell (Tufte, Visual Explanations, p. 21) Restored quantitative information Directional arrows Grid size given (and reduce the dominance of the grid pattern) Time scale, with small clouds depicting storm history

23. components-of-skill.ppt23 Visual display of information Components of skill Chart prepared by Morton Thiokol for Challenger commission. Tufte, Visual Explanations, p. 47 Visual display problems  Disappearing legend.  The O-ring damage legend was on another slide.  Chartjunk  Obscures cause and effect  Temperatures turned sideways  O-ring anomalies depicted by scattered little marks  Wrong order  Date of launch rather than temperature

24. components-of-skill.ppt24 Visual display of information: Scatterplot of temperature vs. O-ring damage Components of skill Tufte, Visual Explanations, p. 45

25. components-of-skill.ppt25 Visual display of information influences mode of thought Components of skill  Use of images tends to induce intuition  Use of numbers tends to induce analysis

26. components-of-skill.ppt26 5. Reliability of information processing  Decreases with increasing information and with increasing environmental uncertainty  Methods for improving reliability of information processing:  Limit the amount of information used in making judgments. Use a small number of very important cues.  Use mechanical methods to process information.  Combine several judgments.  Require justification of judgments. Components of skill

27. components-of-skill.ppt27 Theoretical relation between amount of information and accuracy

28. components-of-skill.ppt28 The relation between information and accuracy depends on environmental uncertainty - - - - - Theoretical limit of accuracy ——— Actual accuracy - - - - - Theoretical limit of accuracy ——— Actual accuracy

29. components-of-skill.ppt29 6 and 7. Bias -- Conditional (regression bias) and unconditional (base rate bias)  Together, the two bias terms measure judgment "calibration.”  Reducing bias:  Experience  Statistical training  Feedback about nature of biases in judgment  Search for discrepant information  Statistical correction for bias Components of skill

30. components-of-skill.ppt30 Calibration of judgments depends on the task Calibration data for precipitation forecasts (Murphy and Winkler, 1974) Heideman (1989)

31. components-of-skill.ppt31 Conclusion Problem: Improving judgmental accuracy – Understanding and improving judgment requires understanding the task and the environment. – Decomposing skill can aid in identifying the factors that limit judgmental accuracy.