1. Visiting human errors in IR systems from decision making perspective
Yan Zhang, School of Information and Library Science, University of North Carolina, Chapel Hill
Introduction
Purposes
Research method
Errors are failures of some planned sequence of mental or physical activities to achieve its intended outcome.
People’s interaction with IR systems is a constant decision making process. Knowledge of biases and heuristics inherent in human reasoning as suggested by decision making theories provides us a useful tool to explore the underlying cognitive processes of human errors in using IR systems. In this pilot study, the potential of decision-making theories in explaining human IR errors is explored.
Explore the potential of decision making theories in explaining human IR errors.
Propose a tentative knowledge structure for classifying human IR errors.
Inform the design of IR systems that are able to predict human errors and to prevent the errors proactively.
Collect human errors in using IR systems, particularly OPACs, commercial databases, and the web, reported in the published literature.
Inspect each error to see whether it can be explained by certain decision making theories.
Preliminary results
Decision making theories and IR errors
Knowledge structure
Dual-process theory: human reasoning system consists of two sub-systems. System 1 encompasses primarily the ecological reasoning process. System 2 is a controlled analytical process.
Adding word(s) or using AND to resolve 0-hit.
In IR context, adding more words and using AND have different meaning than they are in daily environment.
Attentional bias: a failure to look for evidence against an initial possibility, or a failure to consider alternative possibilities.
Users often cannot recover quickly from errors; rather they keep trying the strategies that failed them in the first place.
Simple heuristics vs. Multi-attribute decisions:
people rarely use multiple cues in decision making.
The majority of searches were simple, specifying only one field or data type to be searched.
The advanced search features were rarely used.
Errors that are caused mainly by interface design.
Example: a confusing label of a function.
Comparatively easy to solve and should be addressed with a high priority.
Errors that are caused by inherent human biases.
Example: Using AND to solve 0-hit.
Comparatively hard to address, given the persistency of such biases; but easy to predict.
Errors caused by mismatch between people’s mental models of the system with the system’s features and functions.
Hard to predict, however, frequently occurred mismatches are easy to identify.
Decision making theory
Examples
Type of IR errors
Underlying mechanisms
Solvability
Usability problems
Perceptual errors
Conjunction fallacy: p(A&B) <= p(A), p(B)
Using AND to resolve 0-hit.
Random errors
Errors that do not occur systematically; slips.
Easy to address, but hard to predict.
Gaps in mental models
References
Borgman, C.L. (1987). The study of user behavior on information retrieval systems. ACM SIGCUE Outlook, 19(3-4),
Reason, J. (1990). Human Error. Cambridge: Cambridge University Press.
Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment. Psychological Review, 90,