National Research Lab. An Approach for Assessment of Situation Awareness with Eye Movement and EEG data March, Jun Su Ha Dept. of Nuclear and Quantum Engineering Korea Advanced Institute of Science and Technology

Contents 1. Necessity of Study 2. Background of Approach 3. Use of Eye Movement Data 4. Introduction to Electroencephalogram (EEG) 5. Use of Electroencephalogram (EEG) 6. Method of Approach 7. Experiment Design 8. Contribution to HUPESS Project 9. Summary and Further Study

Korea Advanced Institute of Science and Technology 1. Necessity of Study (1) Endsley defined that “situation awareness (SA) is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future”, which has been one of the most well-known definitions Techniques for the assessment of SA in literatures can generally be categorized into four categories as follows: Performance-based tech. Subjective rating tech. Direct query tech. In some cases such as the integrated system validation in nuclear industry, continuous and objective measures of SA are required.

Korea Advanced Institute of Science and Technology 1. Necessity of Study (2) Even though Visual Indicator of Situation Awareness (VISA) of OECD HRP seems to satisfy the attributes, it is based on basic assumptions as follows: 1. The operators’ visual information gathering from process formats, alarm system, trend diagrams, etc., in the control room, is the main source for perception of relevant elements in the environment (Endsley’s level-1 SA). 2. Spending time on visual examination of key areas in the user interface, can be seen as an indication of situation understanding (Endsley’s level-2 SA). Who can assure that eye fixation on the relevant elements and time spent during the fixation can be interpreted into SA? → very questionable!!! Hence, an improved technique satisfying the attributes and having reasonable bases needs to be developed.

Korea Advanced Institute of Science and Technology 2. Background of Approach 1. The eye fixation on the relevant elements and time spent during the fixation can provide the clues of SA but can not provide whether cognitive processing of SA occurs or not. 2. The cognitive processing in human-being can be detected by another measures such as EEG. 3. In addition, when a human-being is aware of the situation, he (or she) maybe has specific responses such as specific fixation, blink, or EEG characteristics. 4. SA can be measured by analyzing the pattern of the specific responses: artificial neural networks (ANN) can be used as a useful tool!

Korea Advanced Institute of Science and Technology 3. Use of Eye Movement Data In the majority of cases the primary means of information input to the human-being are through the visual channel. There are three main parameters associated with the eye movement behaviors: 1. Eye blinking 2. Eye fixation 3. Pupillary response Among those, fixation-related data will be utilized as not only measures but also clues for the SA measurement: Eye fixation in predefined areas during critical time periods (level-1 SA: perception) Time spent on the visual examination (level-2 SA: comprehension) However level-3 SA (projection) are not applicable yet. Eye tracking system: faceLAB.

Korea Advanced Institute of Science and Technology 4. Introduction to Electroencephalogram (EEG) (1) 1. EEG? Electroencephalogram (EEG) is a electrical signal representing brain activities. 2. Mechanism of EEG occurrence Brain electrical activities depend on the activities of neurons, glia cells, and blood-brain barrier; mainly the activities of neurons. EEG is mainly generated by the difference of potential of synapses which connects the neurons. AP (Action Potential) EPSP (Excitatory Post Synaptic Potential) IPSP (Inhibitory Post Synaptic Potential)

Korea Advanced Institute of Science and Technology 4. Introduction to Electroencephalogram (EEG) (2) 3. Classification of EEG δ-wave: 0.2~3.99 Hz θ-wave: 4~7.99 Hz α-wave: 8~12.99 Hz β-wave: 13~30 Hz γ-wave: 30~50 Hz 4. Relationship between the waves and the cognitive activities δ-wave: sleeping θ-wave: resting or before sleeping α-wave: being comfortable β-wave: conscious state (e.g.,being awake or speaking,…) γ-wave: being impatient or performing highly cognitive activity (e.g., reasoning or decision- making)

Korea Advanced Institute of Science and Technology 5. Use of Electroencephalogram (EEG) Evoked Potential (EP) or Event Relate Potential (ERP) analysis : wave patterns regarding latencies and amplitudes of each peak are analyzed after providing specific stimulations. The EP (or ERP) consists of various peaks such as N100, N200, P300, N400, P600, P800, and so on. Each of the peaks has the relevant meaning about information processing. A previous study reported that P300 reflected attention, perception, or memory searching. The study on EP (or ERP) is expected to provide helpful information for the evaluation of SA with EEG.

Korea Advanced Institute of Science and Technology 6. Method of Approach 1. Eye fixation in predefined areas and time spent on the eye fixation can be considered as not only the measures but also the clue of level-1 and 2 SA. 2. The pattern of other eye movement and EEG data are analyzed at the moment when the clue occurs. 3. An experiment will be performed to identify the SA pattern. 4. A SA measure based on pattern matching technique will be developed; ANN will be used as a tool for the pattern matching. 5. Another experiment will be followed to validate the SA measure to be developed

Korea Advanced Institute of Science and Technology 7. Experiment Design (1) Experiment-1 Experiment-2 Development of a SA measure SA pattern? Yes No Experiment-1: pattern analyses (1) Identify the clue of SA of a subject with eye movement data (fixation & time spent) (2) Confirm whether the subject is really aware of the situation using a direct query technique. (3) Analyze the SA pattern with eye movement and EEG data at the moment when the clue occurs. (4) Iterate (1), (2), and (3) with other subjects Experiment-2: validation of the proposed measure (1) Training the ANN model for the SA developed with the data obtained experiment-1. (2) Validate the SA of subjects using the direct query technique. (3) Iterate (1) and (2) with other subjects Another approach Development of a continuous and objective measure for SA based on pattern matching of eye movement and EEG data

Korea Advanced Institute of Science and Technology 7. Experiment Design (2) Experiment environment: NPP simulator (FISA or CNS) Subject: 20 subjects having nuclear knowledge Experiment time: less than 30 minutes/ a experiment Validation tool: a direct query tech. (e.g., SAGAT or SACRI) Data analysis: statistical analyses

Korea Advanced Institute of Science and Technology 8. Contribution to HUPESS Project HUPESS project will be the reference R&D for the human performance evaluation in HFE V&V. HUPESS is to adopt a continuous and objective measure which is similar to the VISA. → the pattern analysis tech. to be developed in this study can be used to improve the SA measure of HUPESS. The EEG study can be considered as a further R&D for the HUPESS upgrade later → if non-intrusive EEG measurement system is developed, the results of the EEG study can be used to upgrade the HUPESS for the improvement of the evaluation of SA.

Korea Advanced Institute of Science and Technology 9. Summary and Further Study 1. In some cases, continuous and objective measures of SA are required. 2. Eye movement data were also used as the measures but the base of the cognitive processing is questionable. 3. The measures are expected to be improved by blending EEG analysis into the existing study. 4. A method of approach is shown and the relevant experiments are designed. 5. Through additional literature survey, the relationship among the eye movement, EEG, and SA will be studied. 6. Experiments will be followed to complement the literature survey and finally to validate measures to be developed in this study.

Korea Advanced Institute of Science and Technology Reference [1] Endsley, M. R., “Design and Evaluation for Situation Awareness Enhancement”, Proceedings of the Human Factors Society 32nd Annual Meeting (pp ), Santa Monica, CA: Human Factors and Ergonomics Society, [2] Nureg/CR-6393, “ Integrated System Validation Methodology & Review Criteria” [3]Asgeir Droivoldsmo et al., “Continuous measures of situation awareness and workload”, OECD Halden Reactor Project, HPR-539, [4] Jung Hwan Kim and Min Jeong Woo, “EEG measurements and analysis”,

Korea Advanced Institute of Science and Technology Appendix: A Similar R&D in Europe Name : MOSES: More Operational flight-Safety by Enhanced Situation awareness Duration : Jul Dec 2004 Objectives : Within the scope of MOSES are following scientific goals: A measurable definition of situation awareness starting from three different levels (perception, comprehension, projection) Development of safety-critical scenarios from accidents/incidents with loss of situation awareness Proposition of a blend of measurements indicating situation awareness (pilot's eye-point-of-regard measurements, pilot behaviour, physiological and input data, flight and display parameters, and pilot self- assessment of situation awareness) Application of the measurements in safety-critical scenarios Identification of key elements for visual display design and procedures which supports situation awareness Approach: Pilots will fly different realistic scenarios. Their situation awareness will be assessed by different methods: physiological data (eye tracking and EEG for example), psychological assessments (e.g. cognitive styles) and performance.