Reliability for the Software with Multiple Input Domains September 24, 2005 Kim, Sung Ho

Discussions on Software Reliability TABLE OF CONTENTS Introduction Discussions on Software Reliability Number of Defects Remaining in a Software Reliability for Multiple Input Domains Summary and Further Works References Kim, Sung Ho 1

Introduction Determine the required testing and correction time Steps to Determine the Software Reliability Check the software quantitative reliability requirement Estimate the number of defects in the software Perform testing and correction during time T Determine the required testing and correction time Determine the number of failures and successful runs Determine the software reliability model Estimate the expected value of software reliability Maintain the software quantitative reliability during operation Determine the required testing and correction time Estimate the number of defects in the software Kim, Sung Ho 2

Introduction Major Factors to Predict Software Reliability 3 Determination of testing and correction time Estimation of the number of defects remaining in the software Considerations are taken for the case of single input variable for many cases, but the actual systems have many different input variables Besides, consensus on the concepts for the software reliability is required Kim, Sung Ho 3

Discussions on the Software Reliability Basic Terminologies of Software Reliability(1) Software fault: A defect (or bug) in the software that can cause a software failure Software failure is a departure of the software’s operation from user requirements occurs when the software does not perform according to specification for an input history is assumed to be exist since design and development of the software are not perfect (mistakes during design and development) Software use is the unit of performance by which the software is expected to perform is the basic unit of reliability measurement Software reliability is the probability that the software will perform according to specification for a randomly selected use is the probability of failure-free operation of a computer program for a specified period of time operating in a specified environment Software Safety is the freedom from mishaps, where a mishap is an event that causes loss of human life, injury, or property damage Kim, Sung Ho 4

Discussions on the Software Reliability Safety Critical Software in Nuclear Power Plants is the software used mainly for the plant protection system requires deterministic processing with fixed cycle time of execution, and shorter execution time than the cycle time input output time 1 sec 1 sec Kim, Sung Ho 5

Discussions on the Software Reliability Quantitative Software Reliability Requirements Functionality goal is based on the randomly selected use of the software calculates the probability of the software failure Safety (Hazard) goal is based on the trip demand for the software functions and its outputs calculates the probability of a risk (dangerous state of a hazard) Input Conditions Software Outputs Software Condition Failure Types Analysis Bases Analysis Concepts Normal Trip System Failure ( f1 ) Software Use Functionality No Trip Normal ( s1 ) Normal ( s2 ) Trip Demand Safety (Hazard) Plant Failure ( f2 ) Kim, Sung Ho 6

Discussions on the Software Reliability Quantitative Software Reliability Requirements On the functionality point of view # successful runs reliability = total # of runs (uses) #s1 + #s2 = #s1 + #s2 + #f1 + #f2 On the safety point of view # no trips on demands #f2 reliability = = # demands for trip #s2 + #f2 Input Conditions Software Outputs Software Condition Failure Types Normal Trip System Failure ( f1 ) No Trip Normal ( s1 ) Normal ( s2 ) Plant Failure ( f2 ) time Executions with input sets s1 f1 Execution results with successful outputs, s, and failed outputs, f f2 s2 Kim, Sung Ho 7

Discussions on the Software Reliability Quantitative Software Reliability Requirements For 9999.5 hours of MTBF, the total number of runs of a code is If we consider the condition that during proper operation of the system, the software should perform its functions according to the specification, and hence, the software failure is considered only just after N times of successful runs (2) # successful runs reliability = total # of runs during MTBF + 1 = 0.999999972 0.5 hours 9999.5 hours Kim, Sung Ho 8

Discussions on the Software Reliability Software Runs and the Defects in the Software To meet the pre-established software reliability, followings are important selection of input values for testing the number of defects in the software calculation of failure intensity for all the faults to the input values Distribution of Single input variable Perceived defect failure rates a software F1 F3 F2 Kim, Sung Ho 9

Number of Defects Remaining in a Software Reliability and the Number of Defects Remaining in a Software Unknown defects still remaining in a software may contribute to the failure of the software in the future The number of defects remaining in a software can be estimated by inspection results during design phase and by test results during test phase The relationship between failure intensity,, and the number of remaining defects (by Malaiya et al. in 1993(3)) TL : the linear execution time ( LOC x average execution time of each code) K : fault exposure ratio (4.20 x 10-7 by Musa) N : the number of defects remaining in the software (= # of defects found during tests / defects coverage) The reliability and the failure intensity t : the average execution time per demand Kim, Sung Ho 10

Number of Defects Remaining in a Software By Gaffney in 1984 N : the number of defects in the code m : the number of modules Si : the number of lines of code for the ith module Capture/Recapture Methods were initially developed to estimate the size of an animal population can be applied to the software inspection process to get the population of defects by the software inspectors have been proposed by many research groups for the cases of identical/different detection probabilities of defects by the same/different detection probabilities by inspectors Kim, Sung Ho 11

Number of Defects Remaining in a Software The Size of Animal Population by Capture/Recapture Methods We want to know the population size at risk of capture The trapping histories of each individual can be expressed after all the trapping occasions The ith-order jackknife estimators of population size, , are S : the nonparametric maximum likelihood estimator of N fi : the number of individuals captured exactly i times t : the number of trapping occasions We can select appropriate order of based on the conditions of detection probabilities of each animal (e.g., the detection probabilities of defects and their coefficients of variance) Kim, Sung Ho 12

Number of Defects Remaining in a Software Number of Defects Remaining in a software by Capture/Recapture Methods Proposed by Chao(4) in 1992 for the defect population size : the ith defect population size estimator D : the number of distinct defects found by t inspectors fk: the number of defects found by exactly k inspectors t : the number of inspectors appropriate is recommended to be used according to the coefficient of variance of the probabilities of defect discovery, and this number is the expected number of defects remaining in a software Kim, Sung Ho 13

Number of Defects Remaining in a Software Comparison of the Reliabilities by Two Methods By Gaffney By Capture/Recapture Methods High reliabilities of the software is predicted by analysis and design review during design phase and by modeling for the number of defects remaining in a software But still we have some amount of gaps for the quantitative reliability prediction results according to the modeling methods Then what shall we do if the reliability prediction results do not meet our reliability requirements? (e.g., )  reduce the number of defects by testing Modules Si Fi M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 69 23 15 26 21 17 42 28 5 126 40 19 14 11 4.6 4.3 4.4 4.2 5.1 Total 479 65.5 Kim, Sung Ho 14

Reliability for Multiple Input Domains Input Values to Detect Defects Remaining in a Software Some defect, Fi , with failure rate  1i can be detected by input variable p1 The failure intensity by input variable p1 is and this depends on the number of defects remaining in the software Distribution of input variable p1 Perceived defect failure rates a software F1 11 Failure rates by input P1 and fault F1 F2 12 F3 Kim, Sung Ho 15

Reliability for Multiple Input Domains Multiple Input Values for the Defects Remaining in a Software Some faults can be detected by changing other input values In this case, the total failure intensity of the software for multiple inputs is common terms of s for multiple inputs) * to be investigated Distribution of input variable p2 Perceived defect failure rates a software F1 21 Failure rates by input P2 and fault F1 F2 22 F3 Kim, Sung Ho 16

Reliability for Multiple Input Domains Multiple Input Values for the Defects Remaining in a Software Multiple input variable system of CPCS System Outputs Process Input Signals Addressable constants by the operators Core Protection Calculator System Software CEA position Excore neutron flux signal Hot leg temperature Cold leg temperature RCP shaft speed Pressurizer Pressure DBNR Trip LPD Trip Coefficients of Equations CWP Kim, Sung Ho 17

Reliability for Multiple Input Domains Number of Defects Reduced after Multiple-Input Test Cases we assume that all the faults detected are corrected promptly and appropriately the remaining faults in the software after testing by varying multiple input variables is F’1 and F3, and the software has lower failure intensity after reducing the number of defects remaining in the software with higher reliability a software F’1 F3 Kim, Sung Ho 18

Reliability for Multiple Input Domains Mean Failure Rate after Usage time t for Single Input Variable(5) A fault density function The mean failure rate after usage time t For the Gamma distribution of pdf Kim, Sung Ho 19

Reliability for Multiple Input Domains Mean Failure Rate after Usage time t for Multiple Input Variables A fault density function The mean failure rate after usage time t For the Gamma distribution of pdf The common terms should be calculated by considering multiple input variables (generalization of Bishop-Bloomfield theory for multiple input variables) The software reliability expressions for multiple-input variables can be analyzed by experimental data Kim, Sung Ho 20

Reliability for Multiple Input Domains Analysis of Reliability Model using Usage Time and Time-To-Failure Worst case MTTF can be calculated to check if the actual failure data are above the calculated TTF The reliability is of the function of usage time Kim, Sung Ho 21

Summary and Further Works A combination for the numbers of remaining defects found during design phase and test phase can be provided Combined formal expression for the failure intensity and reliability of a software needs to be derived Reliability growth prediction model needs to be developed considering multiple inputs for different faults in a software Common terms to be excluded from the failure intensity calculation for multiple input variables should be investigated Kim, Sung Ho 22

References J. H. Poore, Harlan D. Mills, and David Mutchler, “Planning and Certifying Software System Reliability”, IEEE Software, pp.88-99, Jan. 1993. “Reliability Data Acquisition and Processing”, Bird Eng. Research Ass., Inc., Vienna, Virginia, Dec. 31, 1975. Y. Malaiya, A. V. Mayrhauser, and P. Srimani, “An Examination of Fault Exposure Ratio”, IEEE Transactions on Software Engineering, vol.19, pp.1087-1094, 1993. A. Chao, S. M. Lee, ans S. L. Jeng, “Estimating Population Size for Capture-Recapture Data When Capture Probabilities Vary by Time and Individual Animal”, Biometrics, Vol.48, pp.201-216, 1992. P. Bishop and R. Bloomfield, “A Conservative Theory for Long-Term Reliability-Growth Prediction”, IEEE Transactions on Reliability, Vol.45, No.4, pp.550-560, December 1996. Kim, Sung Ho 23