1. National Research Lab. Theoretical Bases for Assessment of Situation Awareness with Eye Movement and EEG Data May, 31 2005 Jun Su Ha Dept. of Nuclear and Quantum Engineering Korea Advanced Institute of Science and Technology

2. Contents 1. Necessity of Study 2. Scope of Study 3. Tasks in MCR of NPPs 4. Human Information Processing 5. Visual Perception Process 6. Model of SA in Dynamic Decision-making 7. Cognitive Processing in Human Brain 8. Experiment Design 9. Summary and Further Study

3. Korea Advanced Institute of Science and Technology 1. Necessity of Study (1) What is Situation Awareness (SA)? : Endsley defined that “situation awareness (SA) is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future”, which has been one of the most well-known definitions [Endsey (1995)]. What is the difference between SA and Situation Assessment? : Some researchers defined Situation Assessment as SA, but Endsley summarized that SA is a state of kowledge whereas Situation Assessment is the processes to achieve that state [Endsey (1995)]. Most techniques for the assessment of SA in literatures were based on the “questioning” :e.g., performance-based tech, subjective rating tech, and direct query tech.

4. Korea Advanced Institute of Science and Technology 1. Necessity of Study (2) In some cases such as the integrated system validation in nuclear industry, required are continuous and objective measures of SA that is not based on the “questioning”: → because on-exercise questioning tech. (performance-based tech or direct query tech.) is intrusive to human performance (e.g., workload) and post-exercise questioning tech. (subjective rating tech.) is not objective and continuous. Even though Visual Indicator of Situation Awareness (VISA) of OECD HRP seems to be a possible solution, it is based on basic assumptions as follows [ A. Droivoldsmo et al (1998)] : 1. The operators’ visual information gathering from process formats, alarm system, trend diagrams, etc., in the control room, is the main source for perception of relevant elements in the environment (Endsley’s level-1 SA). 2. Spending time on visual examination of key areas in the user interface, can be seen as an indication of situation understanding (Endsley’s level-2 SA).

5. Korea Advanced Institute of Science and Technology 1. Necessity of Study (2) Who can assure that eye fixation on the relevant elements and time spent during the fixation can be interpreted into SA? → very questionable!!! Hence, an improved technique satisfying the attributes (continuous and objective) and having reasonable bases needs to be developed. Eye fixation in predefined areas and time spent on the eye fixation can be considered as the clues of level-1 and 2 SA. But the eye fixation and the time spent can not provide whether cognitive processing for achieving SA occurs or not. The cognitive processing can be detected by EEG measurement. The pattern of eye movement and EEG data are analyzed at the moment when the clues occur. A SA measure based on pattern matching technique will be developed; ANN can be used as a tool for the pattern matching.

6. Korea Advanced Institute of Science and Technology 2. Scope of Study 1. Stage A (literature survey) : Constructing theoretical bases for the assessment of SA with eye movement and EEG data. The eye fixation and the time spent can be considered as valid clues for achieving SA? The cognitive processing interpreted by EEG data can be considered as SA? 2. Stage B (experiment) : Examining and finding out SA patterns with eye movement and EEG data. 3. Stage C (literature survey, development of a method, and experiment) : Developing a method for the assessment of SA based on pattern matching tech.

7. Korea Advanced Institute of Science and Technology 3. Tasks in MCR of NPPs Human operator’s tasks in an MCR of an NPP are a sequence of cognitive works such as: Monitoring the environment, Detecting data or information, Understanding and assessing the situation, Diagnosing the symptoms, Decision-making, Planning responses, And implementing the responses. Similar to the sequence of human information processing Closely related to SA

8. Korea Advanced Institute of Science and Technology 4. Human Information Processing (1) Human information-processing system is represented by different stages [Durso (1999)] : 1. Perception of information about the environment 2. Central processing or transforming that information 3. Responding to that information The first and second stages are highlighted as cognition processes, as shown in a model of information processing [Wichens (2000)] The senses gather information, which is then perceived, providing a meaningful interpretation of what is sensed as aided by prior knowledge stored in long-term memory, through the top-down processing. As we “think about” or manipulate perceived information in working memory, an action is delayed or not executed at all. Information from long-term memory is retrieved every time we perceive familiar information Experience Stimulus World Knowledge (Expectancies and Desires) The Senses Top-Down Processing Bottom-Up Processing Perception

9. Korea Advanced Institute of Science and Technology 4. Human Information Processing (2) The stages of information processing depend on mental or cognitive resources, a sort of pool of attention or mental effort that is of limited availability and can be allocated to processes as required. There are two aspects of attention; selecting sensory channels for further information processing and dividing attention between tasks. Selective attention does not guarantee perception, but it is usually considered necessary to achieve it. The selection of channels to attend is typically driven by four factors: Salience - stimulus (e.g., alarms or alerts) Expectancy - the world where we expect to find information Value - the frequency of looking at or attending to channels is also modified by how valuable it is to look at Effort - selective attention may be inhibited if it is effortful Perception proceeds by 3 simultaneous processes: Bottom-up feature analysis Unitization Top-down processing Experience Stimulus World Knowledge (Expectancies and Desires) The Senses Top-Down Processing Bottom-Up Processing Perception

10. Korea Advanced Institute of Science and Technology 5. Visual Perception Process (1) SearchFixation Selection Structuralizing Simplification Grouping Thinking Analysis Synthesis Comparison Decision-Making Problem Solving Combining Separating Putting Eye MovementVisual ThinkingCognitive Thinking Visual perception includes visual thinking and essentially means not only eye movement but also selective and active perception to solve the relevant problem [Arnheim (1969), Gibson (1966), Irvin Rock (1975), and Y.M. Park (2000)]. The selective and active perception are achieved by iterating searching and fixation. After the searching and fixation process, visual perception is achieved through more complex process. The fixation can be interpreted as selection and understanding the core of a object.

11. Korea Advanced Institute of Science and Technology 5. Visual Perception Process (2) SearchFixation Selection Structuralizing Simplification Grouping Thinking Analysis Synthesis Comparison Decision-Making Problem Solving Combining Separating Putting Eye MovementVisual ThinkingCognitive Thinking The understanding can be accomplished by simplifying, structuralizing, and grouping the object. In the thinking process the object is analyzed, synthesized and then compared with the knowledge stored in memory. The knowledge obtained during the thinking process is utilized for decision-making and problem solving. Finally, the knowledge is combined, separated, and then putted into the memory.

12. Korea Advanced Institute of Science and Technology 6. Model of SA in Dynamic Decision-making (1) State Of The Environment Situation Awareness Perception Of Elements in Current Situation Level 1 Comprehension Of Current Situation Level 2 Projection Of Future Status Level 3 Decision - Goals & Objectives - Preconceptions (Expectations) - Information Processing Mechanisms - Long Term Memory Stores - Automaticity - Abilities - Experience - Training - System Capability - Interface Design - Stress & Workload - Complexity - Automation Performance Of Actions

13. Korea Advanced Institute of Science and Technology 6. Model of SA in Dynamic Decision-making (2) Preattentive processing According to most research on information processing [Wickens (2000)], environmental features are initially processed in parallel through preattentive sensory stores in which certain properties are detected, such as spatial proximity, color, shape, or movement providing cues for further focalized attention. Those objects that are most salient, based on preattentively registered characteristics, will be further processed using focalized attention to achieve perception Cue salience, therefore, will have a large impact on which portions of the environment are initially attended to, and these elements will form the basis for the level-1 SA. Attention Direct attention is needed for not only perceiving and processing the cues attended to but also the later stages of decision making and response execution.

14. Korea Advanced Institute of Science and Technology 6. Model of SA in Dynamic Decision-making (3) Perception In addition to affecting the selection of elements for perception, the way in which information is perceived is directed by the contents of both working memory and long-term memory. The information will be processed faster if it is in agreement with expectations. Long-term memory stores also play a significant role in classifying perceived information into known categories or mental representations as an almost immediate act in the perception process [Hinsley et al. (1977)] The classification made in the perception stage is a function of the knowledge available for making such classifications and will produce the elements of level-1 SA. Working memory In the absence of other mechanisms such as relevant long-term memory stores, most of a person’s active processing of information must occur in working memory. New information must be combined with existing knowledge and a composite picture of the situation developed (level-2 SA). Wickens (2000) has stated that prediction of future states (the culmination of good SA) imposes a heavy load on working memory by requiring the maintenance of present conditions, future conditions, rules used to generate the latter from the former, and actions that are appropriate to the future conditions. Fracker (1987) hypothesized that working memory constitutes the main bottleneck for SA. (this is most likely the case for novices or those dealing with novel situations).

15. Korea Advanced Institute of Science and Technology 6. Model of SA in Dynamic Decision-making (4) Long-term memory In practice long-term memory structures can be used to circumvent the limitations of working memory. Rouse and Morris (1985) defined mental models as “mechanisms whereby humans are able to generate descriptions of system purpose and form, explanations of system functioning and observed system states, and predictions of future states” Knowledge in long-term memory is organized by mental model A well-developed mental model provides (a) knowledge of the relevant elements of the system that can be used in directing attention and classifying information in the perception process, (b) a means of integrating the elements to form an understanding of their meaning (level-2 SA), and (c ) a mechanism for projecting future states of the system based on its current state and an understanding of its dynamics (level-3 SA). The key to using these models to achieve SA rests on the ability of the individual to recognize key features in the environment-critical cues- that will map to key features in the model. The model can then provide for much of the higher levels of SA (comprehension and projection) without loading working memory. Mental model is developed as a function of training and experience in a given environment.

16. Korea Advanced Institute of Science and Technology 6. Model of SA in Dynamic Decision-making (5) To summarize, The eye fixation can be considered as a valid clue for achieving not only level 1-SA (perception) but also level-2 SA (understanding). The selective and active perception are achieved by iterating searching and fixation. The selection of channels to attend is typically driven by four factors: salience, expectancy, value, and effort. The fixation can be interpreted as selection and understanding the core of a object (or situation). Selective attention does not guarantee SA, but it is usually considered necessary to achieve it. Those objects that are most salient, based on preattentively registered characteristics, will be further processed using focalized attention to achieve perception. Cue salience, therefore, will have a large impact on which portions of the environment are initially attended to, and these elements will form the basis for the level-1 SA. The time spent can be considered as a valid clue for achieving level 2-SA (understanding). As we “think about” or manipulate perceived information in working memory, an action is delayed or not executed at all. In the thinking process the situation is analyzed, synthesized and then compared with the knowledge stored in memory. Wickens (2000) has stated that prediction of future states (the culmination of good SA) imposes a heavy load on working memory by requiring the maintenance of present conditions, future conditions, rules used to generate the latter from the former, and actions that are appropriate to the future conditions.

17. Korea Advanced Institute of Science and Technology 7. Cognitive Processing in Human Brain (1) 1. EEG? : Electroencephalogram (EEG) is a electrical signal representing brain activities. 2. Relationship between the waves and the cognitive activities δ-wave (0.2~3.99 Hz) : sleeping θ-wave (4~7.99 Hz) : resting or before sleeping α-wave (8~12.99 Hz) : being comfortable β-wave (13~30 Hz) : conscious state (e.g.,being awake or speaking,…) γ-wave (30~50 Hz) : being impatient or performing highly cognitive activity (e.g., reasoning or decision-making) 3. Event Relate Potential (ERP) analysis : wave patterns regarding latencies and amplitudes of each peak are analyzed after providing specific stimulations. The ERP is recognized at various peaks such as N100, N200, P300, N400, P600, P800, and so on. Each of the peaks has the relevant meaning about information processing. A previous study reported that P300 reflected attention, perception, or memory searching. The ERP is not applicable to the study on complex cognitive activities such as SA in NPP, → Because event evoking the ERP should be simple and iterated quite many time [http://www.laxtha.com/SiteView.asp?x=7&y=32&z=33&infid=199, Donchin & Coles, R. Johnson Jr]. The ERP is not applicable to the study on complex cognitive activities such as SA in NPP, → Because event evoking the ERP should be simple and iterated quite many time [http://www.laxtha.com/SiteView.asp?x=7&y=32&z=33&infid=199, Donchin & Coles, R. Johnson Jr].

18. Korea Advanced Institute of Science and Technology 7. Cognitive Processing in Human Brain (2) 1. α-power, θ-power, and the variation of pulse width increase during the period when the mental workload is expected to increase during the turbine operation in NPP [B.S. Moon et al. (2002)]. 2. W. Lang observed that α-power decreases and θ-power increases while the subjects were asked to transform letters into Morse code [Lang et al. (1988)]. 3. Mecklinger et al. also showed that θ-power increases during memory search work [Mecklinger et al. (1992)]. 4. Widespread gamma activation of cortical EEG can easily be demonstrated during mental activity [Fitzgibbon et al. (2004)]. All of the cognitive tasks augmented gamma power relative to a control condition (eyes open watching a blank computer screen). The Expectancy, Learning, Reading and Subtraction tasks expressed the most impressive gamma response, up to 5 fold above the control condition and there was some task-related specificity of the distribution of increased gamma power, especially in posterior cortex with visual tasks.

19. Korea Advanced Institute of Science and Technology 7. Cognitive Processing in Human Brain (3) Cognitive ActivitiesDescription of the Cognitive Activitiesγ-powerβ-powerα-powerθ-power During turbine operation [Moon] Diagnose and mitigate an abnormal or an emergency situation N/A Increase (workload ↑) Long-term and Working memory (???) [Lang] Transform letters into Morse code N/A DecreaseIncrease Memory search [Mecklinger et al] Perform a semantic memory search task N/A Increase Story reading [Fitzgibbon et al. & Stipacek et al.] Read silently for a period of 28 s Much increase Small increase near γ-band No significant increase N/A Subtraction task [Fitzgibbon et al. & Stipacek et al.] Subtract 7 from 1000 Increase Small increase near γ-band No significant increase N/A Expectancy [Fitzgibbon et al.] Expect the lateralized visual target following a visual direction cue Much increase Small increase near γ-band No significant increase N/A Learning [Fitzgibbon et al. & Stipacek et al.] Memorize a set of 10 words Much increase Small increase near γ-band No significant increase N/A To summarize, Cognitive activities affecting SA (such as processing in working memory, retrieving form long-term or short-term memory, expecting, and learning) are closely related to the γ-power and somewhat related to θ-power. If cognitive activities ↑, the γ-power ↑. To summarize, Cognitive activities affecting SA (such as processing in working memory, retrieving form long-term or short-term memory, expecting, and learning) are closely related to the γ-power and somewhat related to θ-power. If cognitive activities ↑, the γ-power ↑.

20. Korea Advanced Institute of Science and Technology 8. Experiment Design Experiment-1 Experiment-2 Development of a SA measure SA pattern? Yes No Experiment-1: pattern analyses (1) Identify the clue of SA of a subject with eye movement data (fixation & time spent) (2) Confirm whether the subject is really aware of the situation using a direct query technique. (3) Analyze the SA pattern with eye movement and EEG data at the moment when the clue occurs. (4) Iterate (1), (2), and (3) with other subjects Experiment-2: validation of the proposed measure (1) Training the ANN model for the SA developed with the data obtained experiment-1. (2) Validate the SA of subjects using the direct query technique. (3) Iterate (1) and (2) with other subjects Another approach Development of a continuous and objective measure for SA based on pattern matching of eye movement and EEG data Experiment environment: NPP simulator (FISA or CNS) Subject: 20 subjects having nuclear knowledge Experiment time: less than 30 minutes/ a experiment Validation tool: a direct query tech. Data analysis: statistical analyses Experiment environment: NPP simulator (FISA or CNS) Subject: 20 subjects having nuclear knowledge Experiment time: less than 30 minutes/ a experiment Validation tool: a direct query tech. Data analysis: statistical analyses

21. Korea Advanced Institute of Science and Technology 9. Summary and Further Study Constructed are theoretical bases for the assessment of SA with eye movement and EEG data with regard to the following questions. The eye fixation and the time spent can be considered as valid clues for achieving SA? The cognitive processing interpreted by EEG data can be considered as SA? The experiment is going to be designed in more detail and through the experiment SA patterns will be examined and found out with eye movement and EEG data. A method for the assessment of SA based on pattern matching tech. will be developed.

22. Korea Advanced Institute of Science and Technology Reference M. R. Endsley, 1995,“Toward a Theory of Situation Awareness in Dynmic System”, Human Factors Vol.37 (1), pp. 32-64. Asgeir Droivoldsmo et al., 1998, “Continuous measures of situation awareness and workload”, OECD Halden Reactor Project, HPR-539. Christopher D. Wickens and Justin G. Hollands, 2000, “Engineering Psychology and Human Performance”, 3rd edition, Prentice Hall, pp10-14. D. Hinsley, J. R. Hayes, and H. A. Simon, 1977, “From words to equations”, In P. Carpenter and M. Just (Eds.), Cognitive processed in comprehension, Hillsdale, NJ: Erlbaum. M. L. Fracker, 1987, “Situation Awareness: A Decision Model”, Unpublished manuscript. Daytom, OH. W. B. Rouse and N. M. Morris, 1985, “On looking into the black box: Prospects and limits in the search for mental models (DTIC AD-A159080)”, Atlanta, GA: Georgia Institute of Tech., Center for Man- Machine Systems Research. F. Durso, 1999, “Handbook of Applied Cognition”, NewYork, Wiley. Rudolf Arnheim, 1969, “Visual Thinking”, Califonia Press, pp. 12-23. Gibson, 1966, “The Senses Considered as Perceptual System”, Houghton Mifflin co., pp. 9. Irvin Rock, 1975, “An Introduction to Perception”, Mavmillan, pp. 49-52. Y.M. Park, 2000, “A Study on the Evaluation of Visual Perception on the Web Graphic User Interface – with Emphasis on the Eye Movement Analysis –”, KAIST, Master’s Thesis, pp. 17-26.

23. Korea Advanced Institute of Science and Technology Reference Jung Hwan Kim and Min Jeong Woo, “EEG measurements and analysis”, http://www.laxtha.com.http://www.laxtha.com E. Donchin and M. G. H. Coles, 1988, “Is the P300 component a manifestation of cognitive updating?”, The Behavioral and Brain Science, Vol. 11, pp. 357-427. R. Johnson Jr, 1988, “The amplitude of the P300 component of the event-related potential: Review and synthesis”, Advances in Psychophysiology, Vol. 3, pp. 69-137. Christopher D. Wickens et al, 2004, “An Introduction to Human Factors Engineering”, 2nd edition, Prentice Hall, pp120-155. S.P. Fitzgibbon, K.J. Popec, L. Mackenzie, C.R. Clark, and J.O. Willoughby, 2004, “Cognitive tasks augment gamma EEG power”, Clinical Neurophysiology, Vol. 115, pp.1804-1807. B.S. Moon et al., 2002, “Fuzzy systems to process ECG and EEG signals for quantification of the ment al workload”, Information Sciences, Vol. 142, pp. 34. W. Lang et al., 1988, “Event-related EEG-spectra in a concept formation task”, Human Neurobiology, Vol. 6,pp. 295-301. A. Mecklinger et al., 1992, “Event related potentials and EEG components in a semantic memory search task”, Psychophysiology, Vol. 29, pp.104-119. A. Stipacek, R.H. Grabner, C. Neuper, A. Fink, A.C. Neubauer, 2003, “Sensitivity of human EEG alpha band desynchronization to different working memory components and increasing levels of memory load”, Neuroscience Letters, Vol. 353, pp.193-196.