Coherence and correspondence Public Administration and Policy PAD634 Judgment and Decision Making Behavior Coherence and correspondence Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany State University of New York T.STEWART@ALBANY.EDU Copyright © Thomas R. Stewart

Coherence research Coherence research measures the quality of judgment against the standards of logic, mathematics, and probability theory. Coherence theory argues that decisions under uncertainty should be coherent, with respect to the principles of probability theory. coherence-correspondence.ppt

Representativeness problem Coherence Representativeness problem There are two programs in a high school. Boys are a majority (65%) in program A, and a minority (45%) in Program B. There is an equal number of classes in each of the two programs. You enter a class at random, and observe that 55% of the students are boys. What is your best guess--does the class belong to program A or program B? (Kahneman and Tversky, 1972) coherence-correspondence.ppt

Representativeness problem Coherence Representativeness problem Most subjects (67 out of 89) picked program A because the class had more boys, and so did program A. This is an example of an important heuristic: “representativeness.” We judge an event to be a member of a set of events if its characteristics are similar to our belief about members of that set. This is a good rule of thumb in a lot of situations, but not this one. coherence-correspondence.ppt

Representativeness problem Coherence Representativeness problem Some people might feel that the answer could be “either program” because 55% is midway between 45% and 65%. They would have noticed a quantitative relationship, and made a reasonable guess. But they would be wrong. coherence-correspondence.ppt

Representativeness problem Coherence Representativeness problem The correct answer is Program B. The variance of a binomial will be larger for p = .45 than for p = .65. Therefore, there will be more variability in the number of boys in Program B’s classes, and it is more likely that a class in Program B will have as many as 55% boys. coherence-correspondence.ppt

Representativeness problem Coherence Representativeness problem Why does it matter that students (in this case Israeli high school students) fail such a difficult question? Unfortunately, they also fail easier questions. Furthermore, its not only students who make these mistakes. These kinds of errors have been found in studies of scientists, statisticians, physicians, and other professionals. coherence-correspondence.ppt

Mammography illustration Coherence Mammography illustration David Eddy (1982) asked us to imagine that a physician has a patient with a breast mass that he thinks is probably benign. Lets interpret “probably” as a probability of .99. The physician is 99% sure, but not completely sure, that the mass is benign. This is his “prior probability.” The physician orders a mammogram and receives a report that the radiologist thinks the mass is malignant. What should the physician do? coherence-correspondence.ppt

Mammography illustration Coherence Mammography illustration If he read the literature available in 1982, he would find statements like: “The accuracy of mammography is approximately 90 percent” “A positive report of carcinoma is highly accurate.” “The accuracy of mammography in correctly diagnosing malignant lesions averages 80 to 85 percent.” coherence-correspondence.ppt

Mammography illustration Coherence Mammography illustration Note that this is NOT a decision table. If he wanted more specific data, he could find these. Mammograpy outcomes (from Eddy, 1982) Results of X ray If malignant lesion (cancer) If benign lesion (no cancer) P(Positive test) = .792 .096 P(Negative test) = .208 .904 Sensitivity = .792 Specificity = .904 coherence-correspondence.ppt

Mammography illustration Coherence Mammography illustration The physician’s subsequent action for this patient might be influenced by his judgment about the likelihood that the woman has a malignant lesion. What would be your judgment about that likelihood based on these data? coherence-correspondence.ppt

Mammography illustration Coherence Mammography illustration Eddy sampled 100 physicians and found that most of them thought the probability would be around 75 percent. We can calculate that likelihood using the data given and Bayes theorem: coherence-correspondence.ppt

Bayes Theorem Coherence ca = malignant lesion (cancer) benign = benign lesion (no cancer) pos = positive test coherence-correspondence.ppt

Mammography illustration Coherence Mammography illustration So, the actual probability is about .08. The physicians’ judgments were off by almost a factor of 10! Does this matter? It only matters if the previous statement is true, that is, “the physician’s subsequent action might be influenced by his judgment about the likelihood that the woman has a malignant lesion.” In fact, this is a basic tenet of coherence research. coherence-correspondence.ppt

Fundamental tenet of coherence research "Probabilistic thinking is important if people are to understand and cope successfully with real-world uncertainty." Example: "Accurate estimations of probability in diagnosis and prognosis are important for physicians, because these probabilities influence their diagnostic and therapeutic decisions. Objective probabilities, based on epidemiological research, are often not available. In these instances, physicians must rely on their own judgment and apply their probability estimates to individual patients. Research has shown that the accuracy of probability estimates of both experts, such as physicians and of lay people in general is often inadequate...." Timmermans et al. (1996, pp. 107-108) coherence-correspondence.ppt

Fundamental tenet of coherence research "Probabilistic thinking is important if people are to understand and cope successfully with real-world uncertainty." Example: "Accurate estimations of probability in diagnosis and prognosis are important for physicians, because these probabilities influence their diagnostic and therapeutic decisions. Objective probabilities, based on epidemiological research, are often not available. In these instances, physicians must rely on their own judgment and apply their probability estimates to individual patients. Research has shown that the accuracy of probability estimates of both experts, such as physicians and of lay people in general is often inadequate...." Timmermans et al. (1996, pp. 107-108) Two elements of this quote: Probability judgments are important because they influence decisions made under uncertainty. People are not very good at making probability judgments. coherence-correspondence.ppt

Correspondence research Correspondence research measures the quality of judgment against the standards of empirical accuracy. Correspondence theory argues that decisions under uncertainty should result in the least number of errors possible, within the limits imposed by irreducible uncertainty. coherence-correspondence.ppt

Fundamental tenet of correspondence research "Human competence in making judgments and decisions under uncertainty is impressive. Sometimes performance is not. Why? Because sometimes task conditions degrade the accuracy of judgment." Hammond, K. R. (1996). Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. New York, Oxford University Press (p. 282). coherence-correspondence.ppt

Coherence and correspondence theories of competence Coherence theory of competence Uncertainty irrationality error Correspondence theory of competence Uncertainty inaccuracy error What is the relation between coherence and correspondence? coherence-correspondence.ppt

Comparison of Coherence and Correspondence Theories Sources of error Sources of disagreement Methods for improving judgment Value judgments Non-repetitive decisions Sources of competence coherence-correspondence.ppt

Sources of error Coherence Correspondence Comparison Sources of error Coherence "Irrationality" with respect to a normative system Correspondence Multiple fallible indicators: Unreliability in information acquisition Unreliability in information processing Match between environment and judge Fidelity of information system Environmental uncertainty Bias coherence-correspondence.ppt

Sources of disagreement Comparison Sources of disagreement Coherence Use of different heuristics Correspondence Same as sources of error: Unreliability in information acquisition Unreliability in information processing Match between environment and judge Fidelity of information system Environmental uncertainty Bias coherence-correspondence.ppt

Methods for improving judgment Comparison Methods for improving judgment Coherence Decomposition and mechanical recomposition Training in probability and statistics Formal elicitation of subjective probabilities Correspondence Environmental changes, training, and cognitive aids to address: Unreliability in information acquisition Unreliability in information processing Match between environment and judge Bias coherence-correspondence.ppt

Value judgments Coherence Correspondence Comparison Value judgments Coherence Decision theory assumes values can be elicited. Correspondence Has little to say. Correspondence not relevant. coherence-correspondence.ppt

Non-repetitive decisions Comparison Non-repetitive decisions Coherence Decision theory provides prescriptive method. Correspondence Not clear how to apply correspondence theory when decisions are not repetitive. coherence-correspondence.ppt

Sources of competence Coherence Correspondence Comparison Sources of competence Coherence Must be taught, usually in a formal setting, like a school. Correspondence Use of multiple fallible indicators is innate. coherence-correspondence.ppt

Integration of coherence and correspondence theories Learn/teach identification of tasks demanding coherence. Learn/teach identification of tasks demanding correspondence. coherence-correspondence.ppt