1. Effect Estimation of the Integrated HMI Seung Jun Lee Nuclear I&C and Information Engineering Laboratory Dept. of Nuclear and Quantum Engineering KAIST Dec 22, 2005 Lab seminar

2. Korea Advanced Institute of Science and Technology2 Table of Contents 1.Introduction 2.Quantitative Estimation of Support Systems 3.Effect Estimation of the Integrated HMI 4.Summary and Conclusions 5.Further Study

3. Korea Advanced Institute of Science and Technology3 1. Introduction For nuclear power plants, various kinds of intelligent support systems have been developed to support human operators. Since the HMI including these support systems of advanced MCRs takes more important role than conventional MCRs, there are lots of discussions about new HMI design. Operator support systems have advantages such as increase of operators’ performance and reduction of human errors, but they also have several disadvantages such as over-reliance problem. Therefore, performance evaluations for operator support systems need to be performed in order to validate the efficiency of operator support systems. Tasks

4. Korea Advanced Institute of Science and Technology4 1. Introduction How can we estimate the performance of operator support systems? There are several methods for evaluating operator support systems. The methods evaluate operator support systems with various ways. The purpose of this work is to estimate the effect of the integrated HMI which is proposed for advanced MCRs. In order to obtain more reliable results, not only theoretical evaluations but also experiments should be performed. Before practical experiments, the theoretical evaluation was performed. It will be further study to perform experiments and compare the result with that of the theoretical evaluations.

5. Korea Advanced Institute of Science and Technology5 2. Quantitative Estimation Experiment 1 for estimating effects of support systems. Four information aiding types : No aid (N), Alarm (A), Hypothesis (H), and Hypothesis + Expected symptoms of the Hypothesis (H+S) AlarmHypothesis with Certainty Factor Hypothesis with Certainty Factor + Expected Symptom Patterns

6. Korea Advanced Institute of Science and Technology6 2. Quantitative Estimation Hypothesis-and-Test Strategy Time # of Navigated Windows P = 0.000802 P = 0.000787 Time # of Navigated Windows P = 0.012128 P = 0.010907 Decision Table Strategy Time # of Navigated Windows P = 0.496956 P = 0.074549 Pattern Recognition Strategy Effects of support systems were measured by response time and # of navigated windows.

7. Korea Advanced Institute of Science and Technology7 2. Quantitative Estimation Experiment 2 for estimating diagnostic performances of operators in a ship Subjects First level (3 subjects): lower operator functions for the rank of seaman Second level (3 subjects): functions requiring the rank of petty officer Third level (2 subjects): functions requiring the rank of officer Scenarios 8 scenarios which can be solved using general technical principles and do not require knowledge specific to the particular type of ship. All of the 8 scenarios are about leakage, pump failure, sensor failure an so on. In order to get a test score which is a measure of the amount of system knowledge of the subjects, a short test involving 20 questions was given. 5 quality scores of diagnostic performance 1: no diagnosis or diagnosis incorrect 2: diagnosis incorrect with respect to the main problem: sub-problem correctly diagnosed but relation with main problem not recognized 3: problem correctly localized but wrong component or explanation given 4: correct diagnosis but not localized in sufficient detail 5: fully correct diagnosis

8. Korea Advanced Institute of Science and Technology8 2. Quantitative Estimation The results clearly show that performance in this diagnostic task in highly correlated with the system knowledge that the operator possesses, the knowledge concerning the functional relationship between system parts. Lack of system knowledge is a main cause for poor diagnostic performance. SubjectExperience (years)Function level Diagnostic Performance (# of problems solved correctly: Max. 8) Test score (Max. 39) 17.52421 231422 5111425 372528 84.53525 471632 611.52629 763627

9. Korea Advanced Institute of Science and Technology9 2. Quantitative Estimation The purpose of the second experiment is to see whether the addition of the help system increases the percentage of correct diagnoses. The help system provides the subjects with information concerning the possible causes for an alarm. Quality scoreHelp not availableHelp available Frequency (%) 19 (14)1 (2) 215 (24)6 (9) 315 (23)19 (30) 415 (23)18 (28) 510 (16)20 (31) BackgroundHelp not availableHelp available Electrical6694 Mechanical5984 % of problems correctly solved Quality of diagnostic performance Diagnostic performance The results showed significant increases of quality of diagnostic performance and percentage of correctly solved problems. Response time Without the help system, the mean time for solution was 228s. With the help system, there is a small but non- significant increase of 9% to 248s. Confidence There was a mean increase in confidence with the availability of help of 0.15 (max. 5), but this difference was not significant. Effects of support systems were measured by quality and accuracy of diagnostic performance

10. Korea Advanced Institute of Science and Technology10 2. Quantitative Estimation If there is no available human operator Monitoring/ Detection Monitoring/ Detection Situation Assessment Situation Assessment Response Planning Response Planning Response Implementation Response Implementation Display System Display System Instrumentation System Instrumentation System Control/Protection System Control/Protection System Plant Fault Diagnosis System Fault Diagnosis System Decision Support System Decision Support System Implementation System Implementation System a,bc,d e,f g,h i,j k,l mm x x x q rst uvw y z n(o,p) s Autonomous Operating System Autonomous Operating System s Z A,B I&C Systems HMI Human Operators Experiment 3 for estimating an effect of the autonomous operating system

11. Korea Advanced Institute of Science and Technology11 2. Quantitative Estimation Aw=0.0001 Au=0.0001 Aw=0.0005 Au=0.0001 Aw=0.001 Au=0.0001 Bw=0.0001, Bu=0.00010.000599820.000999630.00149938 Bw=0.0005, Bu=0.00010.000999630.001399590.00189954 Bw=0.001, Bu=0.00010.001499380.001899540.00239974 Bw=0.005, Bu=0.00010.005497380.005899140.00640133 Design Hardware Results (Recovery failure Probability) Operator : 0.00052 Operator + Autonomous operating system : 0.00032 Autonomous operating system : 0.00059 Effects of support systems were represented by recovery failure probability.

12. Korea Advanced Institute of Science and Technology12 3. Effect Estimation of Integrated HMI 3.1 Architecture Advanced Display System Advanced Display System Plant Fault Diagnosis System Fault Diagnosis System Computerized Procedure System Computerized Procedure System Implementation System Implementation System Instrumentation System Instrumentation System Control/Protection System Control/Protection System a1,a2a1,a2 q Monitoring/ Detection Monitoring/ Detection Situation Assessment Situation Assessment Response Planning Response Planning Response Implementation Response Implementation n(o,p) Operation Validation System Operation Validation System b1,b2b1,b2 c1,c2c1,c2 d1,d2d1,d2 e1,e2e1,e2 f 1, f 2 g1,g2g1,g2 mm k q q s s r u s vw t t x y z y I&C Systems HMI Human Operators Plant Human Operator Adequate operation suggestion Fault diagnosis system Alarm processing Operation validation Operation validation system Display system Operation execution Plant parameter display Fault diagnosis Detailed fault analysis Procedure navigation Computerized procedure system Procedure display Operation interruption Autonomous procedure display Step search Procedure search Candidate list generation Operation execution Procedure Database Procedure entry condition Database Plant parameter Database Modeled procedure Database Alarm system

13. Korea Advanced Institute of Science and Technology13 3. Effect Estimation of Integrated HMI 3.2 Components Design Hardware Correct (a 1C ) Wrong (a 1W ) Unavailable (a 1U ) Correct (a 2C ) Correct (a 1C a 2C ) Wrong (a 1W a 2C ) Unavailable (a 1U a 2C ) Wrong (a 2W ) Wrong (a 1C a 2W ) Wrong (a 1W a 2W ) Unavailable (a 1U a 2W ) Unavailable (a 2U ) Unavailable (a 1C a 2U ) Unavailable (a 1W a 2U ) Unavailable (a 1U a 2U ) Hardware Design Instrumentation System Instrumentation System Hardware Instrumentation System Instrumentation System Design Control/Protection System Control/Protection System Infor matio n C (correct) (q C ) W (wrong) (q W ) U (unavailable) (q U ) Design C (b 1C ) W (b 1W ) U (b 1U ) C (b 1C ) W (b 1W ) U (b 1U ) C (b 1C ) W (b 1W ) U (b 1U ) HW C (b 2C ) CWUWWUUUU W (b 2W ) WWUWWUUUU U (b 2U ) UUU U UUUUU Main Factors for Control/Protection System Main Factors for Instrumentation System

14. Korea Advanced Institute of Science and Technology14 3. Effect Estimation of Integrated HMI 3.2 Components Display Human Error Correct (r C ) Wrong (r W ) Unavailable (r U ) None (m C ) Correct (r C m C ) Wrong (r W m C ) Unavailable (r U m C ) Exist (m W ) Wrong (r C m W ) Wrong (r W m W ) Unavailable (r U m W ) No Action (m U ) Unavailable (r C m U ) Unavailable (r W m U ) Unavailable (r U m U ) Display System Display System Human Error Human Error Monitoring/ Detection Monitoring/ Detection Main Factors for Human Monitoring/Detection Main Factors for Situation Assessment Fault Diagnosis System Fault Diagnosis System Monitoring/ Detection Monitoring/ Detection Operators’ Ability Operators’ Ability Situation Assessment Situation Assessment Monitori ng C (correct) (u C ) W (wrong) (u W ) U (unavailable) (u U ) Fault Diagnosi s Sys. C (s C ) W (s W ) U (s U ) C (s C ) W (s W ) U (s U ) C (s C ) W (s W ) U (s U ) Operat ors’ Ability High (n H ) CCCCUUCUU Medium (n M ) CCCUWWUUU Low (n L ) CWUUWWUWU

15. Korea Advanced Institute of Science and Technology15 3. Effect Estimation of Integrated HMI 3.2 Components Operators’ Ability State Definition High Human Operators can establish correct situation model if either the transferred information or the fault diagnosis system is correct. They never form wrong situation model. Medium Human Operators can establish correct situation model if the transferred information from the monitoring/detection is correct. They have the ability to discard wrong information using fault diagnosis system. Low Human operators are highly dependant on the fault diagnosis system. Because of that, they sometimes fails to establish correct situation model even though the information for the monitoring/detection is correct. Definitions of the States of Operators’ Ability Expertise Stress (workload) High (o H ) Medium (o M ) Low (o L ) High (p H ) Medium (o H p H ) Low (o M p H ) Low (o L p H ) Medium (p M ) High (o H p M ) Medium (o M p M ) Low (o L p M ) Low (p L ) High (o H p L ) High (o M p L ) Medium (o Lp p L ) Situation Assessment Situation Assessment Response Planning Response Planning Operators’ Ability Operators’ Ability Operators’ Expertise Operators’ Expertise Operators’ Stress Operators’ Stress

16. Korea Advanced Institute of Science and Technology16 3. Effect Estimation of Integrated HMI 3.2 Components Situation Assessment C (correct) (v C ) W (wrong) (v W ) U (unavailable) (v U ) Computerized Procedure Sys. C (t C ) W (t W ) U (t U ) C (t C ) W (t W ) U (t U ) C (t C ) W (t W ) U (t U ) Operat ors’ Ability High (n H ) CCCWWWUUU Medium (n M ) CCCWWWUUU Low (n L ) CWUWWWUUU Main Factors for Response Planning Situation Assessment Situation Assessment Computerized Procedure System Computerized Procedure System Operators’ Ability Operators’ Ability Response Planning Response Planning Main Factors for Response Planning Response Planning Response Planning Human Error Human Error Response Implementation Response Implementation Operation Validation System Operation Validation System Human Error None (m C ) Exist (m W ) No Action (m U ) Operation Validation System C (t C ) W (t W ) U (t U ) C (t C ) W (t W ) U (t U ) C (t C ) W (t W ) U (t U ) Response Planning C (correct) (v C ) CCCCWWCUU W (wrong) (v W ) CWWCWWCUU U (unavailable) (v U ) CWUCWUCUU

17. Korea Advanced Institute of Science and Technology17 3. Effect Estimation of Integrated HMI 3.2 Components Main Factors for Implementation System Hardware Response Implementation Response Implementation Design Implementation System Implementation System Response Implementaion C (correct) (v C ) W (wrong) (v W ) U (unavailable) (v U ) Design C (t C ) W (t W ) U (t U ) C (t C ) W (t W ) U (t U ) C (t C ) W (t W ) U (t U ) HW C (t C ) CWUWWUUUU W (t W ) WWUWWUUUU U (t U ) UUUUUUUUU Human Response Control/ Protection Correct (z C ) Wrong (z W ) Unavailable (z U ) Correct (q C )Success (z C q C )Failure (z W q C )Success (z U q C ) Wrong (q W )Success (z C q W )Failure (z W q W )Failure (z U q W ) Unavailable (q U )Success (z C q U )Failure (z W q U )Failure (z U q U ) Veitch Chart with Probabilities for Final Recovery Probability

18. Korea Advanced Institute of Science and Technology18 3. Effect Estimation of Integrated HMI 3.2 Components Factor IFactor IIFactor IIIOutput Monitoring/Detection Advanced Display System (r C, r W, r U ) Human Error (m C, m W, m U ) u C, u W, u U Situation Assessment Monitoring/ Detection (u C, u W, u U ) Operators ’ Ability (n C, n W, n U ) Fault Diagnosis System (s C, s W, s U ) v C, v W, v U Response Planning Situation Assessment (v C, v W, v U ) Operators ’ Ability (n C, n W, n U ) Computerized Procedure System (t C, t W, t U ) w C, w W, w U Response Implementation Response Planning (w C, w W, w U ) Human Error (m C, m W, m U ) Operation Validation System (y C, y W, y U ) x C, x W, x U

19. Korea Advanced Institute of Science and Technology19 3. Effect Estimation of Integrated HMI 3.3 Fault Probabilities Assumptions For evaluating the integrated HMI, fault probabilities are assumed as follows: The probability that the hardware in various systems belongs to the unavailable state and the probability that the hardware or the design in various systems belongs to the wrong state are commonly assumed to be 10 -4. The probability that the design in various systems belongs to the unavailable state is assumed to be 10 -2, if the implemented algorithm would be complex, or 10 -4 if the implemented algorithm seems to be simple. The probability that the design in the intelligent support systems belongs to the wrong or unavailable state is assumed to be 10 -1 or 10 -2. The implementation system is considered to be extra simple, thus the probabilities of hardware and design to be in the wrong state and the unavailable state are assumed to be 10 -6. Factors related human operators are assumed as follows: Simple error  P[none]=m C =0.998, P[exist] = m W = 0.001, P[no action] = m U = 0.001 Expertise  P[high] = o H = 0.8, P[medium] = o M = 0.1 (g), P[low] = o L = 0.1 (d). Stress  P[high] = p H = 0.1 (d), P[medium] = p M = 0.1 (g), P[low] =p L = 0.8

20. Korea Advanced Institute of Science and Technology20 I&C Systems HMI Human Operators 3. Effect Estimation of Integrated HMI 3.4 Results Advanced Display System Advanced Display System Plant Fault Diagnosis System Fault Diagnosis System Computerized Procedure System Computerized Procedure System Implementation System Implementation System Instrumentation System Instrumentation System Control/Protection System Control/Protection System a 1, a 2 q Monitoring/ Detection Monitoring/ Detection Situation Assessment Situation Assessment Response Planning Response Planning Response Implementation Response Implementation n(o,p) Operation Validation System Operation Validation System b 1, b 2 c 1, c 2 d 1, d 2 e 1, e 2 f 1, f 2 g 1, g 2 mm k q q s s r u s vw t t x y z y 1. Operator2. Operator + FDS2. Operator + FDS + CPS 2. Operator + FDS + CPS + OVS

21. Korea Advanced Institute of Science and Technology21 3. Effect Estimation of Integrated HMI 3.4 Results Results Operator : 0.00253 Operator + FDS : 0.00166 Operator + FDS + CPS : 0.00151 Operator + FDS + CPS + OVS : 0.00077 Operation Validation System d 1w =0.01 d 1u =0.01 d 1w =0.1 d 1u =0.01 d 1w =0.01 d 1u =0.1 d 1w =0.1 d 1u =0.1 f 1w =0.01, f 1u =0.010.00029920.00196660.00064600.0033492 f 1w =0.1, f 1u =0.010.00096880.00310560.00178500.0048487 f 1w =0.01, f 1u =0.10.00043850.00244580.00077260.0033719 f 1w =0.1, f 1u =0.10.00110810.00358490.00191170.0052189 Fault Diagnosis System

22. Korea Advanced Institute of Science and Technology22 3. Effect Estimation of Integrated HMI 3.5 Relative Sensitivity Analysis The relative sensitivity can be used to evaluate the potential contribution of a parameter to the output while varying same “proportion” of each parameter. The relative sensitivity of parameter x to the function f(x,y,z, ¼) is defined as follows : Relative Sensitivity = Parameter Relative Sensitivity (x10 -6 ) Parameter Relative Sensitivity (x10 -6 ) Parameter Relative Sensitivity (x10 -6 ) Parameter Relative Sensitivity (x10 -6 ) a 1W -0.736727c 2W -0.157994f 1W -27.19252mWmW -100.046 a 1U -27.0624c 2U -0.099276f 1U -44.72124mUmU -2.14555 a 2W -0.729432d 1W -21.68131f 2W -1.121523oMoM -6.02563 a 2U -0.267943d 1U -35.01152f 2U -0.728742oLoL -6.16302 b 1W -0.120239d 2W -0.652973g 1W -0.998683pHpH -6.16302 b 1U -12.0239d 2U -0.181138g 1U -0.020362pMpM -6.02563 b 2W -0.119049e 1W -8.351342g 2W -0.998683 b 2U -0.119049e 1U -18.004087g 2U -0.020362 c 1W -0.159574e 2W -0.351342 c 1U -10.0269e 2U -0.193487

23. Korea Advanced Institute of Science and Technology23 3. Effect Estimation of Integrated HMI 3.6 Discussions Recovery Failure Probability Operator : 0.00253 Operator + FDS : 0.00166 Operator + FDS + CPS : 0.00151 Operator + FDS + CPS + OVS : 0.00077 Relative Sensitivity Design factors of the fault diagnosis system, the computerized procedure system, and the operation validation system have very high relative sensitivity than the average of other systems. Especially, the operation validation system has very high relative sensitivity values. Factors of the instrumentation system have high relative sensitivities. Almost all nodes are dependent on the instrumentation system. Factor that has the highest relative sensitivity is simple error. The factors related to human operators are also found to have high relative sensitivities.

24. Korea Advanced Institute of Science and Technology24 3. Effect Estimation of Integrated HMI 3.6 Discussions Operator support systems may affect ‘Operators’ ability’ and ‘Simple error’. Fault Diagnosis System and Computerized Procedure System may be able to compensate insufficient operators’ expertise or to reduce operators’ stress. The new interface of Advanced Display System may be able to reduce the probability of simple errors. These additional effects of support systems do not considered yet. If these kinds of effects are considered, it is expected to get better results. Monitoring/ Detection Monitoring/ Detection Situation Assessment Situation Assessment Response Planning Response Planning Response Implementation Response Implementation Simple Error Simple Error Operators’ Ability Operators’ Ability Simple Error Simple Error Operators’ Expertise Operators’ Expertise Operators’ Stress Operators’ Stress Fault Diagnosis System Fault Diagnosis System Computerized Procedure System Computerized Procedure System Advanced Display System Advanced Display System

25. Korea Advanced Institute of Science and Technology25 4. Summary and Conclusions For nuclear power plants, various kinds of intelligent support systems have been developed to support human operators. Performance evaluations for operator support systems need to be performed in order to validate the efficiency of operator support systems. The quantitative evaluation of the integrated HMI was performed. When all operator support systems are utilized, the lowest recovery failure probability was obtained. In case of just considering an operator, the worst recovery failure probability was obtained. Relative sensitivity analysis was performed. Design factors of the fault diagnosis system, the computerized procedure system, and the operation validation system have very high relative sensitivity than the average of other systems. Factors of the instrumentation system have high relative sensitivities. Factor that has the highest relative sensitivity is simple error. The factors related to human operators are also found to have high relative sensitivities.

26. Korea Advanced Institute of Science and Technology26 5. Further Study Experiments for estimating effects of supports systems will be performed. Experiments will be performed for 3 types of scenarios. ARP (Alarm Response Procedure) for simple alarm AOP (Abnormal Operating Procedure) for multiple alarms EOP (Emergency Operating Procedure) for reactor shutdown For operator support systems, simple systems will be used. Advanced display system: Display system including alarm system Fault diagnosis system: The results of the ADAS with their symptoms Computerized procedure system: Computerized procedures of the target scenarios Operation validation system: Warning system against operations not included in corresponding procedures. The experimental result will be compared with that of the theoretical evaluations. Leak of system knowledge is a main cause for poor diagnostic performance.