Review on Test-Based Approach of Software Reliability November 22 nd, 2010 Nuclear I&C and Information Engineering LabKAIST Bo Gyung Kim

KAIST Nuclear I&C and Information Engineering Lab Contents  Introduction  Software Failure Probability  Failure Probability in Digital System  Input Profile  Example Application  Conclusion  Further Work  References

KAIST Nuclear I&C and Information Engineering Lab Reference  KANG et al., An Overview of Risk Quantification Issues for Digitalized Nuclear Power Plants using a Static Fault Tree, Nuclear Engineering and Technology, Vol.41, No.6,  KANG et al., Input-profile-based software failure probability quantification for safety signal generation systems, Reliability Engineering and System Safety, 94,  White, R.M and Boettcher, D.B, Putting Sizewell B digital protection in context, Nuclear Engineering International, pp ,  Musa JD, The operational profile in software reliability engineering: an overview. In: Third International Symposium on Software Reliability Engineering, 1992.

KAIST Nuclear I&C and Information Engineering Lab Introduction  Background Risk Quantification Issues of Digital System Probabilistic Risk Assessment(PRA) Risk Quantification Issues of Digital System Probabilistic Risk Assessment(PRA) Hardware Issue Software Issue System Issue Safety Function Issue Software Failure mode Software Failure Probability Quantification in consideration of software testing and verification and validation (V&V)

KAIST Nuclear I&C and Information Engineering Lab Software Failure Probability  Software failure probability must be quantified based on testing results. The software reliability growth model(SRGM) is the most mature technique for software dependability assessment.  However this approach is known to be inappropriate for safety-critical systems.  Because the fixes cannot be assumed as effective and the last fix may have introduced new faults when applying the software reliability growth models to safety-critical software.  Applying the lower limit of a software failure probability estimated conservatively through testing can be an alternative to using conventional SRGM.  The number of observed failures of a highly reliable software program during a test is expected to be zero because the elucidated errors will be debugged in the corresponding code and the test will be repeated.  One of the important aspects of test-based software reliability assessment is that the test cases should represent the inputs which are encountered during actual use.

KAIST Nuclear I&C and Information Engineering Lab Software Failure Probability  The test inputs for safety-critical applications such as the reactor protection system (RPS) of a nuclear power plant are inputs which cause the activation of a protective action such as a reactor trip.  An appropriate input profile must be determined for effective software failure probability quantification.  The paper of “Input-profile-based software failure probability quantification for safety signal generation systems” proposes a simple but realistic method to perform the software failure probability quantification in consideration of the characteristics of a digital system and plant dynamics.  The proposed method will be especially useful for reduction of the required number of test cases for the digital processing equipment of plant parameters.  For quantifying the software failure probability using final test results, binomial distribution and Bayesian approach using beta distribution is useful statistical models.

KAIST Nuclear I&C and Information Engineering Lab Failure Probability in Digital System  A digital system treats inputs from instrumentation sensors in a discrete manner (binary digital values) by using an analog-to-digital converter (ADC).  Input space is not infinite.  The number of possible test inputs is n ≤ 2 r. (r: resolution of ADC)  Each input digital values means a partitioned sampling space.  Then the failure probability can be determined as  In order to quantify θ t, the input profile (p i ) must be determined. θ i : the software failure probability for input i p i : the input probability

KAIST Nuclear I&C and Information Engineering Lab Input Profile  If a deviation happens in a nuclear power plant, the process parameter values deviate from normal values and moves to the setpoints. If a parameter goes beyond the setpoint, the RPS activates the reactor trip signal. The input profile depends on the scan time and the plant dynamics. Fig. The scan time and the demand generation in consideration of the input domain

KAIST Nuclear I&C and Information Engineering Lab Input Profile D max means the maximum i given ADC resolution, scan time, and deviation. Fig. The scan time change and the demand point change  Scan Timing If the scan time is large, the physical parameter may go far beyond the set point and cause the late detection of a plant deviation. The demand point changes, when the scan timing changes.

KAIST Nuclear I&C and Information Engineering Lab Input Profile  The scan timing varies randomly. So, the portion of the scan time for each digital value of input i depends on the graph shape of the process parameter in deviation x.  Usually the scan time is very short and it can assume uniform distribution in this narrow region.  p 1 =p 2 =p 3 =…=p max and p i =0 (D max <i≤n)  p 1 +p 2 +p 3 +…+p max =fraction of deviation x frequency  p 1 =p 2 =p 3 =…=p max =F(deviation x)/D max (deviation x)  F(x) : the fraction of x over all possible deviations

KAIST Nuclear I&C and Information Engineering Lab Input Profile  Input profile (p i ) generation

KAIST Nuclear I&C and Information Engineering Lab Example Application  For simplicity, this paper only investigated the loss of coolant accident (LOCA) and the process parameter of the pressurizer pressure. Based on the USNRC, authors categorized the LOCA groups. IDHole diameter (m)Frequency (#/y)Fraction E E E E E E E E E E E E-06 ID Scan time (12-bit ADC) 30ms50ms100ms200ms ID Scan time (14-bit ADC) 30ms50ms100ms200ms Table 1. Categorization of the LOCAs and their frequencies Table 2. D max of the pressurizer pressure for various scan times (12-bit ADC) Table 3. D max of the pressurizer pressure for various scan times (14-bit ADC)

KAIST Nuclear I&C and Information Engineering Lab Example Application Fig. The developed input profile (scan time 100 ms, 12-bit ADC). Fig. Input profiles for various scan times (12-bit ADC).  Input profile ID Scan time (12-bit ADC) 30ms50ms100ms200ms Table 2. D max of the pressurizer pressure for various scan times (12-bit ADC)

KAIST Nuclear I&C and Information Engineering Lab Example Application Table 4. Software failure probability (scan time 100 ms, 12-bit ADC) Table 5. Software failure probabilities for various scan times (30, 50, 100, 200 ms)  Software Failure Probability

KAIST Nuclear I&C and Information Engineering Lab Conclusion  Software is one of the most important safety issues in digital system safety assessment. It can be treated in a probabilistic manner in consideration of characteristics of input sequences.  A study which mentioned above proposed a method for software failure probability estimation for the signal-processing system in consideration of the input profile, which can be produced based on process parameter analysis.  With the proposed method, a very high reliability of safety-critical software in signal-processing system can be proved with a small number of testing cases.  In this study, uncertainty was not considered. In order to develop a more accurate method, it is necessary to investigate modeling of input profile in consideration of uncertainty.

KAIST Nuclear I&C and Information Engineering Lab Further Work 2. CCF (Common Cause Failure) Risk Quantification Issues of Digital System Probabilistic Risk Assessment(PRA) Risk Quantification Issues of Digital System Probabilistic Risk Assessment(PRA) Hardware Issue Software Issue System Issue Safety Function Issue Software Failure mode Software Failure Probability Quantification 1.

Thank you for your attention