1 Software Reliability Growth Models Incorporating Fault Dependency with Various Debugging Time Lags Chin-Yu Huang, Chu-Ti Lin, Sy-Yen Kuo, Michael R. Lyu *, and Chuan-Ching Sue *Computer Science & Engineering Department The Chinese University of Hong Kong Shatin, Hong Kong The 28 th Annual International Computer Software and Application Conference (COMPSAC 2004)

2 Outline Introduction Reviews of software fault detection and correction process Considering failure dependency & debugging time lag in software reliability modeling Numerical examples Conclusions

3 Introduction The software is becoming increasingly complex and sophisticated. Thus reliability will become the main goal for software developers. Software reliability is defined as the probability of failure-free software operation for a specified period of time in a specified environment. Over the past 30 years, many Software Reliability Growth Models (SRGMs) have been proposed for estimation of reliability growth of products during software development processes.

4 Introduction (contd.) One common assumption of conventional SRGMs is that detected faults are immediately removed. In practice, this assumption may not be realistic in software development. The time to remove a fault depends on the complexity of the detected faults, the skills of the debugging team, the available man power, or the software development environment, etc. Therefore, the time delayed by the detection and correction processes should not be neglected.

5 Introduction (contd.) On the other hand, there are two types of faults in a software system: mutually independent faults and mutually dependent faults. –mutually independent faults are on different program paths. –mutually dependent faults can be removed only if the leading faults are removed. In this paper, we will incorporate the ideas of failure dependency and time-dependent delay function into software reliability growth modeling.

6 Basic Assumptions of Most SRGMs 1. The fault removal process follows the Non- homogeneous Poisson Process (NHPP). 2. The software system is subject to failures at random times caused by the manifestation of remaining faults in the system. 3. All faults are independent and equally detectable. 4. Each time a failure occurs, the corresponding fault is immediately and perfectly removed. No new faults are introduced.

7 Issues of Fault Detection and Correction Processes in SRGMs It is noted that the assumption (4) may not be realistic in practice. Actually, most existing SRGMs can be reinterpreted as delayed fault-detection models that can model the software fault detection and correction processes. We can remove the impractical assumption that the fault-correction process is perfect and establish a corresponding time-dependent delay function to fit the fault-correction process.

8 Definition 1 Given a fault-detection and fault-correction process, one defines the delay-effect factor, φ(t), to be a time-dependent function that measures the expected delay in correcting a detected fault at any time.

9 Definition 2 An SRGM is called a delayed-time NHPP model if it obeys the following assumptions: 1.The fault detection process follows the NHPP. 2.The software system is subject to failures at random times caused by the manifestation of remaining faults in the system. 3.All faults are independent and equally detectable. 4.The rate of change of mean value function (MVF) of the number of detected faults is exponentially decreasing. 5.The detected faults are not immediately removed. The fault correction process lags the fault detection process by a delay-effect factor φ(t).

10 Fault Detection and Correction Processes in SRGMs Based on the above assumptions (1)-(4), the original MVF of NHPP model is where a is the expected number of initial faults, and r is the fault detection rate. From the assumption (5) in definition 2 and above equation, the new MVF can be depicted as We thus derive the following theorem.

11 Theorem 1 Given a delay-factor, φ(t),we have: 1) The fault-detection intensity of the delayed-time NHPP SRGM is where a>0 and r>0. 2)

12 Three Conventional SRGMs In the following, we will review three conventional SRGMs based on NHPP can be directly derived from Definition 1, Definition 2, and Theorem 1. 1) Goel-Okumoto model 2) Yamada delayed S-shaped model 3) Yamada Weibull-type testing-effort function model

13 Goel-Okumoto model This model, first proposed by Goel and Okumoto, is one of the most popular NHPP model in the field of software reliability modeling. If, then (Definition 1) a) we have (Theorem 1) b) mean value function (MVF): (Definition 2)

14 Yamada delayed S-shaped model The Yamada delayed S-shaped model is a modification of the NHPP to obtain an S-shaped curve for the cumulative number of failures detected such that the failure rate initially increases and later decays. If, then (Definition 1) a) we have (Theorem 1) b) MVF: (Definition 2).

15 Yamada Weibull-type testing-effort function model Yamada et al. proposed a software reliability model incorporating the amount of test-effort expended during the software testing phase. The distribution of testing-effort can be described by a Weibull curve. If, then (Definition 1) a) we have (Theorem 1) b) Mean value function: (Definition 2).

16 Considering Failure Dependency in Software Fault Modeling Assumption: 1.The fault detection process follows the NHPP. 2.The software system is subject to failures at random times caused by the manifestation of remaining faults in the system. 3.The all detected faults can be categorized as leading faults and dependent faults. Besides, the total number of faults is finite. 4.The mean number of leading faults detected in the time interval (t, t+Δt] is proportional to the mean number of remaining leading faults in the system. Besides, the proportionality is a constant over time.

17 Considering Failure Dependency in Software Fault Modeling (contd.) 5.The mean number of dependent faults detected in the time interval (t, t+Δt) is proportional to the mean number of remaining dependent faults in the system and to the ratio of leading faults removed at time t and the total number of faults. Besides, the proportionality is a constant over time. 6.The detected dependent fault may not be immediately removed and it lags the fault detection process by a delay-effect factor φ(t). That is, φ(t) is the time delay between the removal of the leading fault and the removal of the dependent fault(s). 7.No new faults are introduced during the fault removal process.

18 Considering Failure Dependency in Software Fault Modeling (contd.) From assumption (3) & (4), we have a= a 1 + a 2. Besides, a 1 = P× a and a 2 = (1 － P)× a., and m(t)= m 1 (t) + m 2 (t) Where: a : expected number of initial faults. a 1 : total number of leading faults. a 2 : total number of dependent faults. P : portion of the leading faults. m(t) : the MVF of the expected number of initial faults. m 1 (t) : the MVF of the expected number of leading faults detected in time (0, t]. m 2 (t) : the MVF of the expected number of dependent faults detected in time (0, t].

19 Considering Failure Dependency in Software Fault Modeling (contd.) From assumption (4), the number of detected leading faults is proportional to the number of remaining leading faults. Thus we obtain the following differential equation., a 1 >0, 0<r<1. Solving the above differential equation under the boundary condition m 1 (t) =0, we have

20 Considering Failure Dependency in Software Fault Modeling (contd.) Similarly, from assumption (5) ~ (7), we obtain the following differential equation. In the following, we will give a detail description of possible behavior of φ(t).

21 Case 1 If, we obtain  From assumption (3), the MVF of the expected number of initial faults is

22 Case 2 If, we obtain  From assumption (3), the MVF of the expected number of initial faults is

23 Case 3 If, we obtain Similarly, we can obtain the MVF of the expected number of initial faults by means of the same steps.

24 Numerical Examples Two Real Software Failure Data Sets –Ohba’s data set (1984) –Data set from RVLIS project in Taiwan (2003) Criteria for Model’s Comparison –Mean Square of Fitting Error –Mean Error of Prediction –Noise Performance Analysis –We only consider Case 1 & Case 2 as illustration. –Parameters of all models are estimated by using the method of LSE and MLE.

25 Data Description The first data set (DS1) : –The system was a PL/I database application software consisting of approximately 1,317,000 lines of code. During 19 weeks, CPU hours were consumed and about 328 software faults were removed.

26 Data Description (contd.) WEEKCNFWEEKCNFWEEKCNFWEEKCNF Table 1: Number of failures per week from DS1. Where CNF: Cumulative number of failures.

27 Data Description (contd.) The second data set (DS2): –RVLIS, a detection system used to monitor the level of water within the reactor vessel. The coding language is VersaPro 2.03 and the development platform is GE FANUC PLC It took 25 weeks to complete the test. During the test phase, 230 software faults were removed.

28 WEEKCNFWEEKCNFWEEKCNFWEEKCNF Table 2: Number of failures per week from DS2. Where CNF: Cumulative number of failures. Data Description (contd.)

29 Criteria for Model’s Comparison (1) Noise: where r i is the predicted failure rate. Mean Square of Fitting Error (MSE): where m i is the observed number of faults by time t i. –A smaller MSE indicates a smaller fitting error and better performance.

30 Criteria for Model’s Comparison (2) Mean Error of Prediction (MEOP): where n i is the observed cumulative number of failures at time s i and m i is the predicted cumulative number of failures at time s i, i=k, k+1,…, n.

31 Performance Analysis Table 3: Comparison results of different SRGMs for Case1-DS1. ModelarθPMSEMEOPNoise Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Goel-Okumoto model —— Delay S-shaped model ——

32 Performance Analysis (contd.) Case 1 — DS1 –The proposed model estimates P=0.55. That is, the software may contain two categories of faults, 55% are leading faults and 45% are dependent faults. –The possible values of P are also discussed and listed in Table 3. –The proposed model provides the lowest MSE & MEOP if compared to the Goel-Okumoto model and the Yamada delay S-shaped model. –We can see that the proposed model provides the best fit and prediction for the second data set.

33 Table 4: Comparison results of different SRGMs for Case1-DS2. ModelarθPMSEMEOPNoise Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Goel-Okumoto model —— Delay S-shaped model —— Performance Analysis (contd.)

34 Performance Analysis (contd.) Case 1 — DS2 –The proposed model estimates P=0.71. That is, the software may contain two categories of faults, 71% are leading faults and 29% are dependent faults. –The possible values of P are also discussed and listed in Table 4. –The proposed model provides the lowest MSE & MEOP if compared to the Goel-Okumoto model and the Yamada delay S-shaped model. –We can see that the proposed model provides the best fit and prediction for the second data set.

35 ModelarθPMSEMEOPNoise Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Goel-Okumoto model —— Delay S-shaped model —— Table 5: Comparison results of different SRGMs for Case2-DS1. Performance Analysis (contd.)

36 Performance Analysis (contd.) Case 2 — DS1 –The proposed model estimates P=0.72 for this data set. And it indicates that the software may contain two categories of faults, 72% are leading faults and 28% are dependent faults. –The possible values of P are also discussed and listed in Table 5. –The proposed model almost provides the lowest MEOP if compared to the Goel-Okumoto model and the Yamada delay S-shaped model. –Overall, the MVF of proposed model provides a good fit to this data.

37 ModelarθPMSEMEOPNoise Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Equation of Case Goel-Okumoto model —— Delay S-shaped model —— Table 6: Comparison results of different SRGMs for Case2-DS2. Performance Analysis (contd.)

38 Performance Analysis (contd.) Case 2 — DS2 –The proposed model estimates P=0.77 for this data set. And it indicates that the software may contain two categories of faults, 77% are leading faults and 23% are dependent faults. –The possible values of P are also discussed and listed in Table 6. –On the other hand, we know that the inflection S- shaped model is based on the dependency of faults by postulating the assumption: some of the faults are not detectable before some other faults are removed. Therefore, it may provide us some information for reference.

39 Performance Analysis (contd.) Case 2 — DS2 (contd.) –After the simulation, we find that the estimated value of inflection rate (which indicates the ratio of the number of detectable faults to the total number of faults in the software) is for DS2. It indicates that the growth curve is slightly S-shaped. –On the average, the proposed model performs well in this actual data.

40 Conclusions In this paper, we incorporate both failure dependency and time-dependent delay function into software reliability assessment. Specifically, all detected faults can be categorized as leading faults and dependent faults. Moreover, the fault-correction process can be modeled as a delayed fault-detection process and it lags the detection process by a time-dependent delay. Experimental results show that the proposed framework to incorporate both failure dependency and time-dependent delay function for SRGM has a fairly accurate prediction capability.