1. Methods and Metrics for Analysis of Sensemaking Dr Karen Carr& Mr Barry McGuinness BAE SYSTEMS Advanced Technology Centre

2. Our objectives For this meeting: –Contribute what makes sense to us, in our given context and with our goals In our work: –Develop the ability to supply C3I ‘capabilities’ (in partnership) –Systems Engineering of socio-technical systems –Driven by need to deliver usable and demonstrable results –Science as well as engineering and domain expertise NB We want to ensure that human issues drive the developments - but we don’t want to forget that we have to inform technology (as well as organization, process)

3. What we mean by sensemaking Why we want to use this concept to try and answer the questions we need to answer Our question: “How can we develop technology, design and manage systems which support/ enhance the human roles in defence operations?” –Significant human role is ability to adapt, respond to unexpected, creativity, play mind games, etc. Need to preserve & enhance that - not interfere. –Support is needed for dealing with the unexpected, the unknown, as well as recognisable situations –Include broad System of Systems issues, developers, rapid change Sensemaking (& Situational Awareness) is a working concept to enable us to start manipulating, analysing, and measuring context, goals and human performance

4. Why we want methods and metrics for studying sensemaking Need to attribute effects - predict - in order to provide support. Move from concepts to metrics to analysis to (testable) models. Reduce subjective bias (influence of our own sensemaking, interpretation) No existing clear metrics we can use - no absolutes 1. Understand how human performs, and what conditions facilitate ‘good’ performance (what hinders) 2. Identify the properties of organisation, process, technology, training, etc which are important for success 3.Develop design and management methods and tools to enable implementation NB not necessarily numbers - could be properties

5. Range of methods observation (non intrusive) subjective investigation (e.g. ethnographic, knowledge elicitation) storytelling/anecdotes (knowledge building) metaphor (pattern matching) scientific method (empirical hypothesis testing) mathematical analysis (baseline)

6. Methods and Metrics Concepts Metrics Some analyses Implications for sensemaking

7. Concepts Orientation –complex, uncertain situations –SA determines capacity to decide and act –sensemaking determines SA –cognitive processes are intrinsically goal-directed –people form nested hierarchy of processes & outcomes Objectives –Understand SA and sensemaking –Feed into design & development of information systems and human systems –Applied research -- theory into practice

8. What is a Situation? A situation is a pattern in state space, especially one which appears to deviate from a “normal” intended or expected pattern. Example:- aircraft fuel x time into flight Aircraft fuel level Time into Flight Unexpected rate-- we have a situation! Normal takeoff Normal cruise

9. Unrecognized Patterns An unrecognized pattern demands attention. Attention! Perceived pattern ??? Known patterns? Unknown pattern? perceptioncomprehension

10. Definitions Knowledge: = capacity for “action” Situational Awareness: = dynamic “situated” knowledge, i.e. capacity to act effectively here & now in a given specific situation Sensemaking: = process of creating effective SA in situations of uncertainty “Knowing what’s going on so you can figure out what to do.” doing saying thinking

11. Situational awareness Dynamic mental representation of the current and future state of one’s domain of action –includes awareness of environment entities events processes actions others’ perceptions & intentions –insofar as these are relevant to performing an action, or choosing a course of action Through a continuous process of situation assessment

12. Situational awareness SA is based on... prior KNOWLEDGE –SCHEMAS: generalized patterns representing typical situations –based on experience, training, culture recent INFORMATION –direct perception of the environment –perception of instruments and displays –received communications SA INFORMATION KNOWLEDGE PHYSICAL DOMAIN instrumentscommunications

13. Central role of SA SA Decision- making Action performance PHYSICAL DOMAIN Information acquisition Sense- making SA both informs and is informed by sense-making decision-making COGNITIVE DOMAIN

14. Inside SA: Cutting up the cake Concrete (situation-specific) Abstract (generalized patterns) ObservedImplied Information specific propositions e.g., “rate of fuel loss is high” Intentions selected actions afforded by situation e.g., “Contact ATC and inform” Models situational schemas e.g., “Fuel leak?” “Faulty sensor?” Projections mental simulations e.g., “Risk of not reaching destination”

15. Processes involved in SA PERCEPTIONAcquisition of information about the given situation COMPREHENSION Diagnostic interpretation of the given situation PROJECTION Prognostic simulation of future situations and their possible outcomes RESOLUTION Selection of actions to direct the given situation towards the desired outcome … All serving to support dynamically effective action information models intentions projections

16. Sensemaking and SA Information Models Intentions Projections PERCEPTION COMPREHENSION PROJECTION RESOLUTION Decision making Sense making Sensing Acting Sense-making: when comprehension is uncertain Decision-making: when resolution is uncertain

17. Metacognition Defined as: –“Thinking about thinking” or “knowledge about knowledge” –i.e. “Awareness of your own SA” noticing uncertainties, gaps, conflicts in your mental reps identifying information needs employing strategies for sensemaking & decision-making SA “It’s like looking over your own shoulder.” ? Gives a subjective sense of SA

18. SA and metacognition Four possible states: Appropriate Confidence (ideal state) Inappropriate Confidence (danger state) Inappropriate Sensemaking Appropriate Sensemaking True SAFalse SA Confident in SA Not confident in SA Actual awareness: Need for sensemaking Subjective attitude

19. Team SA and shared SA Not the same thing Team SA = sum of current knowledge held across a team, irrespective of who has it Shared SA = those parts of the team SA that are common between team members Shared SA Team SA Personal SA

20. What to share, with whom? The nature of SA in groups is dictated by goals Goals are hierarchic Top-level goals are shared by all members –therefore need shared SA with respect to that objective Lower-level goals are specific to individuals –therefore need personal SA with respect to own task Sharing one’s SA is necessary only to the extent that the knowledge has bearing on the goals of others

21. Team SA and shared SA Shared SA elements can be differentially allocated perception comprehension projection resolution perception resolution perception resolution perception resolution projection comprehension

22. M u Distributed SA in the C2 HQ m Ops m m m Signal Commander Intel m Intentions (RESOLUTION) Information (PERCEPTION) Models (COMPREHENSION) m Metacognition ? z Projections (PROJECTION)

23. So... Explicit sensemaking processes are needed when comprehension cannot easily occur Sensemaking requires metacognitive awareness of own knowledge -- uncertainties, gaps Metacognitive assessments can be wrong and lead to inappropriate subjective attitude -- and inappropriate behaviour

24. Measuring SA COGNITIVE approach –queries about the situation –Reveals mental reps Multiple choice (SAGAT) True/False (QUASA) Sit Reps SUBJECTIVE approach –self-ratings of SA –Reveals metacognitive state Unidimensional (SARS) Multidimensional (SART) Multidimensional and intelligible! (CARS) OBJECTIVE approach – behavioural & physiological correlates – Reveals changes in metacognitive state EEG, fMRI Eye pointing As a rule, take both cognitive & subjective measures together.

25. CARS Crew Awareness Rating Scale a subjective tool to elicit operator’s subjective sense of SA multi-dimensional generic, adaptable, easy to use

26. Dimensions Knowledge Processing Perception Comprehension Projection Resolution

27. Eight CARS questions 1. the most recent information 2. what is really going on here 3. what could happen 4. what actions should be taken 1. monitor the flow of information 2. understand the big picture 3. predict how it is likely to evolve 4. decide what actions to take Would you say you have a good sense of … Would you say it is easy for you to … knowledge processing

28. Six possible responses YES NO Certain Uncertain Definitely not Think not Think so For sure? Don’t know Don’t need it Do I ?

29. CARS results CONTENT Perception|||||||| Comprehension|||||||| Projection|||||||| Resolution|||||||| PROCESSING Perception|||||||| Comprehension|||||||| Projection|||||||| Resolution|||||||| Def Prob Prob not Def notDKNA

30. CARS results 0 40 80 20 60 % of ratings 100 Run 1 Run 2 Run 3 Run 4 1. Definitely 2. Probably 3. Probably NOT 4. Definitely NOT 1. Definitely 2. Probably 3. Probably NOT 4. Definitely NOT Comprehension knowledge over time

31. QUASA Quantitative Assessment of Situational Awareness a probe tool to elicit operator’s actual SA mathematical : based on SDT still under development, but promising

32. QUASA Signal Detection Theory YES! perceptiondiscrimination Square ? Targets vs non-targets Hits, False Alarms, Good Misses, False Rejections Also applies to internal (mental) representations

33. QUASA “Is this item true?” –Confidence in perceived truth value of items varies Confidence in truth value of items Number of items Max SENSITIVITY = ideal SA Weak Strong TRUE items FALSE items

34. QUASA Confidence in truth of items Number of items No sensitivity, poor SA Weak Strong Literally can’t tell the difference between true & false items… They have similar-strength levels of confidence

35. QUASA Confidence in truth value of items Number of items Max NEGATIVE sensitivity = worst case SA Weak Strong FALSE items TRUE items Deception

36. QUASA Confidence in truth value of items Number of items SA’ IB’’ Weak Strong Some positive sensitivity Low positive bias (acceptance threshold) Bad acceptancesBad rejections Good rejections Good acceptances

37. QUASA Example probe: “ The tanks adjacent to bridge are enemy ” Response: YES (accept as true) or NO (reject as false) Evaluate: Sensitivity ( discrimination of true/false situations ) = SA’ Bias ( probability of item acceptance/rejection ) = IB’’

38. QUASA Maximum positive sensitivity: ideal SA Maximum negative sensitivity: the wrong situation! Maximum positive bias: too rash Maximum negative bias: too cautious TYPICAL +100 0 -100

39. QUASA Perception: information Comprehension: model of situation Projection: Future developments Resolution: CoA intention +100 0 -100

41. QUASA Mathematical assessment of SA Needs the truth! SA, bias, components, temporal ? Team & shared SA

42. Behavioural correlate of SA Tracking eye-point-of-gaze (EPOG) Do EPOG patterns correlate with SA?

43. EPOG research ‘Entropy’ = known loss of SA

44. SA and flightdeck automation Collision avoidance system +01 Heathrow control this is Speedbird five five, descending now to flight level one four. Speedbird five five, Heathrow control, roger, descending to flight level one four. Heathrown control this is Delta four zero four, flight level two zero, request descent clearance. Delta four zero four, Heathrow control, what is your present altitude?... Radio “party line”

45. SA and flightdeck automation Traffic Situation Reporting Detectable aircraft Non-Detectable aircraft 50% 25% 0% With automation Conventional Reported aircraft

46. SA and C2 digitization - ISTAR Own force positions Enemy positions BGHQ crewstation Common Operational Picture

47. SA and C2 digitization - ISTAR Battlespace digitization demonstrator Synthetic environment

48. SA and C2 digitization - ISTAR 2-hr ISTAR recce operation Performed with voice AND digital C2 systems Measures taken of mental workload & situational awareness

49. SA and C2 digitization - ISTAR rating voice digital aspects of SA (knowledge of enemy) PERCEPTION DEF PROB NOT DEF NOT PROB PROJECTIONRESOLUTIONCOMPREHENSION

50. Some implications Both actual SA and subjective sense of SA affect decision-making & performance Technology can affect SA for better or worse Analyses with metrics provide specific insights

51. Other work DS1 trials BattleLab trials Cognitive modelling –COGNET in C2 environment –Ideal Decision Maker ? Can be used to predict dips in SA and sensemaking needs Building Industry-MoD partnership

52. Implications for sensemaking Thinking about thinking Concepts : sensemaking as processes supporting SA Role of metacognition : group context Metrics of SA : can be used to evaluate sensemaking solutions Data can feed development of predictive models Knowing what’s going on so we can figure out what to do!