1. Observational Task-Oriented Assessment of Situation Awareness Lab Seminar Aug. 2 nd, 2010 Seung Ki Shin

2. Korea Advanced Institute of Science and Technology Contents Introduction Background Observational Task-Oriented Assessment of SA Experimental Study Design of Experiment Results Discussion Conclusions 1/17

3. Korea Advanced Institute of Science and Technology References J. Patrick, N. James, A. Ahmed, P. Halliday, “Observational assessment of situation awareness, team differences and training implications”, Ergonomics, 2006. H. Artman, “Team situation assessment and information distribution”, Ergonomics, 2000. M. R. Endsley, “A comparative analysis of SAGAT and SART for evaluations of situation awareness”, Proceedings of Annual Meeting of the Human Factors & Ergonomics Society, 1998. 2/17

4. Korea Advanced Institute of Science and Technology Introduction Situation Awareness (SA) originates in aviation during the First World War, where it was noted that a few pilots were responsible for shooting down a large number of enemy aircraft and this was attributed to heightened SA. Many studies have aimed to identify and enhance SA not only in the aviation context, but also in other complex, dynamic performance situations including main control room in NPPs. There are still divergent views concerning how to define SA, its nature and how it should be measured. Team SA is a topic that has received increased attention in more recent literature, although currently few detailed empirical studies exist. 3/17

5. Korea Advanced Institute of Science and Technology Background Situation awareness Task hierarchy of supervisory control ‘Achieving SA’ means that a person or team has the necessary and sufficient knowledge to carry out their work activities in any given situation. This knowledge need to be distinguished from the information process including attention, perception, comprehension, etc., which are responsible for generating it that Endsley (1995) label situation assessment. 4/17

6. Korea Advanced Institute of Science and Technology Background Methodologies for measuring situation awareness Self-rating techniques (Subjective) SART (SA Rating Technique) – Taylor (1990) Questionnaires that identify a person or team’s knowledge (Objective) SACRI (SA Control Room Inventory) – Hogg (1995) SAGAT (SA Global Assessment Technique) – Endsley (1990) Freeze the simulation at random points in time. Query the subject with a question (correct or incorrect) chosen randomly from a pre- defined bank of task-relevant questions. 5/17

7. Korea Advanced Institute of Science and Technology Observational Task-Oriented Assessment of SA Problems of the existing assessment methods Existing methods have difficulties in measuring SA in detailed and specific manner because they break the link between the varying situations and a person or team’s awareness. SAGAT With a global measure of SA, the tight coupling between a situation and a person’s awareness of it has to be broken. Repeated use of any SA measure during freezes of a simulated scenario is likely to affect an operator’s monitoring and attention allocation strategies. During dynamic supervisory control situations, it is necessary to be able to measure SA with regard to specific events that occur, rather than using a method that assesses an aggregate form of SA. The measure of SA needs to be targeted at capturing a team or person’s awareness that is relevant to their specific work activities at a particular point in time. 6/17

8. Korea Advanced Institute of Science and Technology Observational Task-Oriented Assessment of SA Solution to these problems To design specific measures of SA and use observational assessment From pilot studies, potential problems are identified from operator self-report and expert observations. Fine-grained and situation-relevant SA measures can be taken. Interruptions to a simulated scenario can be avoided. To use a behavioral observation scale to measure SA If a specific form of SA that has to be achieved by a team at particular time can be identified and designed into a scenario so that it necessitates some immediate communication with colleagues and action, then SA can be assessed on the basis of this behavioral evidence. Observers are informed not only about the nature and timing of behavioral evidence but also how an individual or team can achieve the necessary SA. Unambiguous criterion-based measures of SA can be achieved. 7/17

9. Korea Advanced Institute of Science and Technology Experimental Study Objectives To investigate the practical feasibility and reliability of using an observational methodology to investigate SA with regard to specific events during complex dynamic scenarios To investigate team SA and aspects that may be problematic in the context of nuclear control room team There have been no empirical comparisons of team SA that adopt the detailed task-oriented approach together with an observational methodology. Design of experiment All five shift teams from on plant take part in three scenarios as part of their bi- annual simulator training using the full-scale simulator situated in the on-site training facility of a gas-cooled nuclear reactor. - Objectives, Design of Experiment 8/17

10. Korea Advanced Institute of Science and Technology Experimental Study All eight SA points are embedded in three scenarios. The SA points could not be circumvented whatever the earlier actions or decisions of the shift in the scenario. Each SA point constitutes a significant practical issue from the trainer’s perspective. Three scenarios Scenario 1: Shift handover with a leak on the main feed pump. SA point 1: Communication It concerns the omission of some written and verbal information during sift handover briefing that there had been a leak on the main feed pump, which was now not leaking. SA point 2: Attention Scenario 2: A start-up with a partially closed valve leading to low flow to a boiler. SA point 3, 4: Communication SA point 5: Perception Scenario 3: A bomb explosion and double reactor trip SA point 6, 7: Attention SA point 8: Initial knowledge and /or reasoning - Design of Experiment 9/17

11. Korea Advanced Institute of Science and Technology Experimental Study Procedure During the scenarios, three observers from the research team independently rate shifts’ performance using a 7-point scale. Observer gallery is situated above the simulator control room and soundproofed. Observers are given detailed information of the nature and timing of all programmed events. In the assessment of SA the behavioral criteria are carefully scripted and general judgments of technical competence are not required. Video recordings are scrutinized further in order to identify possible reasons for lack of SA. - Procedure 10/17

12. Korea Advanced Institute of Science and Technology Experimental Study Results - Situation awareness of shifts during scenarios - - Results 11/17

13. Korea Advanced Institute of Science and Technology Experimental Study Retrospective analysis of significant individual and team processes From the analysis of video recordings, total 28 incidents which are effective with regard to achieving SA are identified using CIT (Critical Incident Technique). CIT is a set of procedures used for collecting human behavior that makes a significant contribution – either positively or negatively – to an activity or phenomenon. 1)Determining and reviewing the incident 2)Fact-finding about the details of the incident from the participants 3)Identifying the issues 4)Developing the possible solutions 5)Evaluating whether the selected solution solves the root cause of the situation As a consequence of discussion, a final list of 20 categories are produced and classified into 6 high-level categories. Planning, Problem-solving, Team coordination, Attention, Communication, Knowledge A total of 102 behavioral elements (41 positive and 61 negative) are extracted from the identified incidents and categorized. - Results 12/17

14. Korea Advanced Institute of Science and Technology Experimental Study - Team and individual situation awareness processes - - Results 13/17

15. Korea Advanced Institute of Science and Technology Experimental Study Discussion Practicability of the observer methodology to measure SA ICC (Intraclass Correlation Coefficient) is calculated to assess the reliability of he mean SA rating of the three observers. ICC is one measure of inter-rater reliability. All ICCs indicated very high reliability between raters, ranging from 0.99 to 0.89. An increase in the number of raters will increase the reliability of these ratings. - Discussion 14/17

16. Korea Advanced Institute of Science and Technology Experimental Study Lack of consistency in team differences in SA In five of the SA points covering attention, communication and perception, statistically significant differences between shifts are found, but there was no discernible consistency to these differences. It gives support to the situation-specific perspective of SA that rejects the notion that there is some general SA ability or attribute. Prediction future SA is likely to be problematic for measures of SA that are based on the aggregation of scores across different situations, even within the same scenario. Application of the CIT method supports the identification of incidents and associated behavioral elements that contributed to both SA and a lack of it. The overall set of categories provide a good starting point for the development of training aimed at improving SA for control room teams. - Discussion 15/17

17. Korea Advanced Institute of Science and Technology Experimental Study Team coordination emerges as an important and the most frequent factor from the classification of the SA-related behaviors. Team is able to change the tempo of their work in an emergency situation. The majority of behavioral elements under team coordination relates to the shift leader. Planning, problem-solving and attention are also identified as important behavioral dimensions related to SA. Contingency planning involved anticipating future problems before specific operations constitute an aspect of team behavior that affect SA. There is a need for teams to be trained to consider alternative plausible hypotheses in parallel and to avoid confirmation bias. It is also important what control screens are monitored and how attention is divided between different tasks by different shift members. The pattern of communications within a shift team can also affect the extent to which SA is achieved. A high level of communication can have both positive and negative effects depending on the situation. - Discussion 16/17

18. Korea Advanced Institute of Science and Technology Conclusions SA should be measured in a relevant task-specific manner and not aggregated into a global index. S and A of SA should not be separated. SA analyses have to preserve the link between characteristics of a particular situation and a person or team’s awareness of it. A major advantage of the proposed method is that it obviates the need for successive ‘freezes’ of the scenario, and enables specific aspects of SA to be assessed. This study has implications for the training of control room teams. Training should be devised on the basis of specific individual and team behaviors because of the lack of consistency in SA differences among teams and the difficulty of making generalizations. SA is a very situation-specific phenomenon that requires equally specific training. 17/17