1. Coherence and correspondence
Public Administration and Policy
PAD634 Judgment and Decision Making Behavior
Coherence and correspondence
Thomas R. Stewart, Ph.D.
Center for Policy Research
Rockefeller College of Public Affairs and Policy
University at Albany
State University of New York
Copyright © Thomas R. Stewart

2. Coherence research
Coherence research measures the quality of judgment against the standards of logic, mathematics, and probability theory. Coherence theory argues that decisions under uncertainty should be coherent, with respect to the principles of probability theory.
coherence-correspondence.ppt

3. Representativeness problem
Coherence
Representativeness problem
There are two programs in a high school. Boys are a majority (65%) in program A, and a minority (45%) in Program B. There is an equal number of classes in each of the two programs. You enter a class at random, and observe that 55% of the students are boys. What is your best guess--does the class belong to program A or program B? (Kahneman and Tversky, 1972)
coherence-correspondence.ppt

4. Representativeness problem
Coherence
Representativeness problem
Most subjects (67 out of 89) picked program A because the class had more boys, and so did program A. This is an example of an important heuristic: “representativeness.” We judge an event to be a member of a set of events if its characteristics are similar to our belief about members of that set. This is a good rule of thumb in a lot of situations, but not this one.
coherence-correspondence.ppt

5. Representativeness problem
Coherence
Representativeness problem
Some people might feel that the answer could be “either program” because 55% is midway between 45% and 65%.
They would have noticed a quantitative relationship, and made a reasonable guess.
But they would be wrong.
coherence-correspondence.ppt

6. Representativeness problem
Coherence
Representativeness problem
The correct answer is Program B.
The variance of a binomial will be larger for p = .45 than for p = Therefore, there will be more variability in the number of boys in Program B’s classes, and it is more likely that a class in Program B will have as many as 55% boys.
coherence-correspondence.ppt

7. Representativeness problem
Coherence
Representativeness problem
Why does it matter that students (in this case Israeli high school students) fail such a difficult question? Unfortunately, they also fail easier questions. Furthermore, its not only students who make these mistakes. These kinds of errors have been found in studies of scientists, statisticians, physicians, and other professionals.
coherence-correspondence.ppt

8. Mammography illustration
Coherence
Mammography illustration
David Eddy (1982) asked us to imagine that a physician has a patient with a breast mass that he thinks is probably benign. Lets interpret “probably” as a probability of The physician is 99% sure, but not completely sure, that the mass is benign. This is his “prior probability.” The physician orders a mammogram and receives a report that the radiologist thinks the mass is malignant. What should the physician do?
coherence-correspondence.ppt

9. Mammography illustration
Coherence
Mammography illustration
If he read the literature available in 1982, he would find statements like:
“The accuracy of mammography is approximately 90 percent”
“A positive report of carcinoma is highly accurate.”
“The accuracy of mammography in correctly diagnosing malignant lesions averages 80 to 85 percent.”
coherence-correspondence.ppt

10. Mammography illustration
Coherence
Mammography illustration
Note that this is NOT a decision table.
If he wanted more specific data, he could find these.
Mammograpy outcomes (from Eddy, 1982)
Results of X ray
If malignant lesion (cancer)
If benign lesion (no cancer)
P(Positive test) =
.792
.096
P(Negative test) =
.208
.904
Sensitivity = Specificity = .904
coherence-correspondence.ppt

11. Mammography illustration
Coherence
Mammography illustration
The physician’s subsequent action for this patient might be influenced by his judgment about the likelihood that the woman has a malignant lesion. What would be your judgment about that likelihood based on these data?
coherence-correspondence.ppt

12. Mammography illustration
Coherence
Mammography illustration
Eddy sampled 100 physicians and found that most of them thought the probability would be around 75 percent.
We can calculate that likelihood using the data given and Bayes theorem:
coherence-correspondence.ppt

13. Bayes Theorem Coherence
ca = malignant lesion (cancer) benign = benign lesion (no cancer) pos = positive test
coherence-correspondence.ppt

14. Mammography illustration
Coherence
Mammography illustration
So, the actual probability is about The physicians’ judgments were off by almost a factor of 10!
Does this matter?
It only matters if the previous statement is true, that is, “the physician’s subsequent action might be influenced by his judgment about the likelihood that the woman has a malignant lesion.” In fact, this is a basic tenet of coherence research.
coherence-correspondence.ppt

15. Fundamental tenet of coherence research
"Probabilistic thinking is important if people are to understand and cope successfully with real-world uncertainty."
Example:
"Accurate estimations of probability in diagnosis and prognosis are important for physicians, because these probabilities influence their diagnostic and therapeutic decisions. Objective probabilities, based on epidemiological research, are often not available. In these instances, physicians must rely on their own judgment and apply their probability estimates to individual patients. Research has shown that the accuracy of probability estimates of both experts, such as physicians and of lay people in general is often inadequate...." Timmermans et al. (1996, pp )
coherence-correspondence.ppt

16. Fundamental tenet of coherence research
"Probabilistic thinking is important if people are to understand and cope successfully with real-world uncertainty."
Example:
"Accurate estimations of probability in diagnosis and prognosis are important for physicians, because these probabilities influence their diagnostic and therapeutic decisions. Objective probabilities, based on epidemiological research, are often not available. In these instances, physicians must rely on their own judgment and apply their probability estimates to individual patients. Research has shown that the accuracy of probability estimates of both experts, such as physicians and of lay people in general is often inadequate...." Timmermans et al. (1996, pp )
Two elements of this quote:
Probability judgments are important because they influence decisions made under uncertainty.
People are not very good at making probability judgments.
coherence-correspondence.ppt

17. Correspondence research
Correspondence research measures the quality of judgment against the standards of empirical accuracy. Correspondence theory argues that decisions under uncertainty should result in the least number of errors possible, within the limits imposed by irreducible uncertainty.
coherence-correspondence.ppt

18. Fundamental tenet of correspondence research
"Human competence in making judgments and decisions under uncertainty is impressive. Sometimes performance is not. Why? Because sometimes task conditions degrade the accuracy of judgment."
Hammond, K. R. (1996). Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. New York, Oxford University Press (p. 282).
coherence-correspondence.ppt

19. Coherence and correspondence theories of competence
Coherence theory of competence
Uncertainty irrationality error
Correspondence theory of competence
Uncertainty inaccuracy error
What is the relation between coherence and correspondence?
coherence-correspondence.ppt

20. Comparison of Coherence and Correspondence Theories
Sources of error
Sources of disagreement
Methods for improving judgment
Value judgments
Non-repetitive decisions
Sources of competence
coherence-correspondence.ppt

21. Sources of error Coherence Correspondence
Comparison
Sources of error
Coherence
"Irrationality" with respect to a normative system
Correspondence
Multiple fallible indicators:
Unreliability in information acquisition
Unreliability in information processing
Match between environment and judge
Fidelity of information system
Environmental uncertainty
Bias
coherence-correspondence.ppt

22. Sources of disagreement
Comparison
Sources of disagreement
Coherence
Use of different heuristics
Correspondence
Same as sources of error:
Unreliability in information acquisition
Unreliability in information processing
Match between environment and judge
Fidelity of information system
Environmental uncertainty
Bias
coherence-correspondence.ppt

23. Methods for improving judgment
Comparison
Methods for improving judgment
Coherence
Decomposition and mechanical recomposition
Training in probability and statistics
Formal elicitation of subjective probabilities
Correspondence
Environmental changes, training, and cognitive aids to address:
Unreliability in information acquisition
Unreliability in information processing
Match between environment and judge
Bias
coherence-correspondence.ppt

24. Value judgments Coherence Correspondence
Comparison
Value judgments
Coherence
Decision theory assumes values can be elicited.
Correspondence
Has little to say. Correspondence not relevant.
coherence-correspondence.ppt

25. Non-repetitive decisions
Comparison
Non-repetitive decisions
Coherence
Decision theory provides prescriptive method.
Correspondence
Not clear how to apply correspondence theory when decisions are not repetitive.
coherence-correspondence.ppt

26. Sources of competence Coherence Correspondence
Comparison
Sources of competence
Coherence
Must be taught, usually in a formal setting, like a school.
Correspondence
Use of multiple fallible indicators is innate.
coherence-correspondence.ppt

27. Integration of coherence and correspondence theories
Learn/teach identification of tasks demanding coherence.
Learn/teach identification of tasks demanding correspondence.
coherence-correspondence.ppt