1. Modeling Human Reasoning About Meta-Information Presented By: Scott Langevin Jingsong Wang

2. Introduction Human decision-making in real-time, dynamic environments is becoming more complex Decision-makers must manage large amounts of incoming information and integrate it with previous knowledge to develop a “situational awareness” Relies on domain-knowledge but also on the qualifiers (meta-information) describing the information Problem: To replicate human reasoning or behavior, need to model both information and meta-information Most approaches have focused on representing the information, but little discussion of the meta- information

3. What is Meta-Information? Definitions –Data is output from a system that may or may not be useful to decision-making (radar reports storm is coming) –Information is recognized inputs that are useable to decision-making (storm is coming that may affect UAVs) –Meta-data is qualifiers of data that may or may not be useful to decision-making (radar can locate aircraft with error of +/-1.5m) –Meta-information is qualifiers of information that affect decision- making, reasoning, or behavior Information processing Situation awareness Decision-making Definitions serve to explicitly identify the critical role of meta-information in human decision-making

4. Human Behavioral Models Attempt to replicate human cognitive processes Attempt to model human behaviors must capture the impact of meta-information HBM have wide variety of applications –Developing and testing theories of human cognition –Representing realistic human behavior in training –Expert and Decision Support Systems Modelers typically do not address meta-information because of challenges acquiring, aggregating and integrating Focus of this research is on modeling meta-information in Bayesian Belief Networks (BBNs)

5. Uncertainty and Human Decision-Making Human decision-making under uncertainty deviates from logical decision-making and largely based on experience-based heuristic methods Often the heuristics represent how experts reason about the meta- information Uncertainty of information is one type of meta-information Different methods of classifying uncertainty: –Executional uncertainty –Goal uncertainty –Environment uncertainty –Lack of information, etc While these classifications of uncertainty and an understanding of their impacts on decision-making have been useful, they may not generalize to other types of meta-information not based on uncertainty (recency, reliability, trust)

6. Computational Approaches to Uncertainty Probability Measures Dempster–Shafer belief functions Extensions to first-order logic (e.g., defeasible reasoning, argumentation) Ranking functions ‘plausibility” measures Fuzzy set theory Causal network methods (e.g., Bayesian belief networks, similarity networks, influence diagrams)

7. Types and Sources of Meta-Information Identified the main types of meta-information that impact the decision-making process Research from over 30 domain experts, and over 500h of interviews, observations and evaluations From this developed a list of sources and types of meta-information that was consistently encountered across application domains Believe this approach developed an understanding of expert reasoning and behavior sufficient to understand the impact of meta-information at a level that supports modeling

8. Types and Sources of Meta-Information

9. Modeling Human Reasoning and Behavior Computational Representation of human reasoning and behavior Model based on recognition-primed decision-making –Experts do not do significant amounts of reasoning and problem solving, but rather have been trained to recognize critical elements of a situation and act accordingly –Domain independent, modeling situation awareness-centered decision- making in high-stress, time-critical environments SAMPLE is a general use HBM –Defined modules: Information Processing, Situation Assessment, Decision Making –Inputs processed by information processing module –Processed data (detected events) passed to situation assessment module –Assessed situation is passed to decision-making module Rules, or lookup table of actions after situational assessment performed

10. SAMPLE Model

11. Bayesian Modeling about and with Meta- Information Difficult aspect of modeling human cognition and behavioral processes is the need to reflect the known impacts of meta- information on those processes Identified five features of reasoning that need representation within human behavior models: –Should succeed or fail to recognize relevant meta-information based on attentional and cognitive demands –Should support the representation of successful or unsuccessful human strategies to process information according to meta-information –Should represent the aggregation of meta-information –Should capture how effectively meta-information is understood relative to any prior understanding or knowledge –Should succeed and fail at incorporating meta-information-mediated situation assessments into behavior or decisions

12. Methods for Representing Human Reasoning Bayesian belief networks Fuzzy set theory Rule-based production systems Case-based reasoning BBNs address multiple types of modeling requirements Two types of meta-information reasoning –Deductive reasoning –Abductive reasoning BBNs support both types of reasoning

13. Two Types of Reasoning

14. Modeling the Recognition and Aggregation of Meta-Information In many cases, human decision-makers will have to compute meta-information from multiple factors Data and meta-data can map to meta-information in the following ways: –One-to-one mappings –Many-to-one mappings –One-to-many mappings –Many-to-many mappings Once meta-information is calculated, it can influence the information gathering, situation assessment, and decision-making process

15. Applying BBNs to Model Congnitive Computation of Meta Information

16. Sensor Type as Node in Network: Sensor Type 3

17. Sensor Type as Node in Network: Sensor Type 1

18. Aggregating Meta-Information to Compute Overall Confidence

19. Modeling the Recognition and Aggregation of Meta-Information Knowing the best means to aggregate meta- information is challenging –Observation and study of human decision-making amongst subject matter experts may provide some justification, but will often unavoidably result in inclusion of biases –Using engineering data about sources may not adequately represent how a human would reason about meta-information, resulting in less reflective human behavior models

20. Modeling the Impact of Meta-Information on Situation Assessment Three Approaches –Simply filter or prioritize information based on meta- information –Include meta-information within BBN models of information gathering, situation assessment, and decision-making processes –Use the meta-information in a specific parameter

21. Incorporating Meta-Information Explicitly into a BBN: No Confidence

22. Incorporating Meta-Information Explicitly into a BBN: Low Confidence

23. Examples of Computing the Probability of a Discrete Value for a BBN Node

24. Conclusion We described the application of meta-information and BBNs in modeling each of the following types of cognitive tasks: –Recognition of relevant meta-information based on aggregation of available data, meta-data, information, and meta-information into types of meta-information. –Filtering and prioritization of information based on meta-information. –Aggregation of different types of meta-information to acquire their combined impact. –Understanding of the impact of meta-information on existing knowledge –Incorporation of meta-information into mediation of situation assessment and decision-making.

25. Questions?