Chin-Yu Huang Department of Computer Science National Tsing Hua University Hsinchu, Taiwan Optimal Allocation of Testing-Resource Considering Cost, Reliability, and Testing-Effort The 10th International Symposium Pacific Rim Dependable Computing (PRDC 2004)

2 Outline Introduction Software Reliability Growth Model with Generalized Logistic Testing-Effort Function Testing Resource Allocation for Software Module Testing Experimental Studies and Results —Numerical Example and Sensitivity Analysis Conclusions

3 Introduction A software testing process consists of several testing stages including module testing, integration testing, system testing, and installation testing. The quality of the tests usually corresponds to the maturity of the software test process, which in turn relates to the maturity of the overall software development process. Most popular and commercial software products are complex systems and composed of a number of modules.

4 Introduction (contd.) Project managers should know how to allocate the specified testing-resources among all the modules and develop quality software with high reliability. We provide a systematic method for the project managers to allocate specific amount of testing- resource expenditures for each module under some constraints: minimizing the cost of software development, with a given fixed amount of testing- effort and a reliability objective.

5 SRGMs Software Reliability represents a customer- oriented view of software quality and is dynamic rather than static. SRGMs describe failures as a random process, which is characterized in either times of failures or the number of failures at fixed time. We use an SRGM with generalized logistic testing-effort function to describe the time- dependency behaviors of detected faults and the testing-resource expenditures spent during module testing.

6 Generalized Logistic Testing-Effort Function The generalized logistic testing-effort function (LTEF) was proposed by Huang, Kuo, and Lyu (1999) and can be depicted as where N is the total amount of testing effort to be eventually consumed, a is the consumption rate of testing-effort expenditures, A is a constant, and k is a structuring index with a large value for modeling well-structured software development efforts.

7 SRGM with Generalized Logistic Testing-Effort Function where m(t) is the expected mean number of faults detected in time (0, t), W k (t) is the current testing-effort consumption at time t, a is the expected number of initial faults, and r is the error detection rate per unit testing-effort at testing time t & r>0. This SRGM with Generalized LTEF was proposed by Huang, Kuo, and Lyu (1999) and the mean value function is

8 SRGM with Generalized Logistic Testing-Effort Function (contd.) When t→∞, the expected number of faults to be detected is The reliability of a software system is defined as the ratio of the cumulative number of detected faults at time t to the expected number of initial faults: (if A >> N)

9 Testing-Resource Allocation for Module Testing Assumptions : 1)The software system is composed of N independent software modules that are tested individually. The number of faults remaining in each module can be estimated by an SRGM with generalized LTEF. 2)For each module, the failure data have been collected and the parameters of each module can be estimated. 3)The total amount of testing resource expenditures available for the module testing processes is fixed and denoted by W.

10 Testing-Resource Allocation for Module Testing Assumptions (contd.): 4)If any of the modules is faulty, the whole software system fails. 5)The system manager has to allocate the total testing resources W to each software module to minimize the number of faults remaining in the system during the testing period. 6)The desired software reliability after the testing phase is greater than or equal to the reliability objective R 0.

11 Mean Value Function of a Software System with N Modules The number of detected faults in the system can be estimated by where v i is a weighting factor to measure the relative importance of a fault removal from module i in the future. If v i =1 for all i =1, 2,…, N, the objective is to minimize the total number of faults remaining in the software system after the testing phase.

12 Mean Value Function of a Software System with N Modules (contd.) Therefore, the number of remaining faults can be estimated by

13 Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective The objective function is : minimizing the cost of software testing Subject to the constraints : (1) The total amount of testing resource expenditures for all modules would not be more than available resource W. (2) Reliability of each module should not be less than R 0  guarantee that reliability of system will not be less than R 0.

14 Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective The objective function is: Minimize: Subject to the constrains:, i=1, 2,..., N.

15 Therefore, we have:, i= 1,2,…,N Let We can have,,i=1,2,….,N and,where Let, we can transform the above equations to The following two equation: Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective

16 Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective Minimize: Subject to,i=1,2,..., N. where

17 According to the Lagrange multiplier, above equations be simplified as follows: Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective Minimize:

18 Based on the Kuhn-Tucker conditions (KTC), the necessary conditions for a minimum value of above equation are in existence. The results will be applied in the Algorithm for solving our problem. Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective

19 Algorithm 1 Step 1: Set l=0. Step 2: Calculate the following equations,i=1, 2,..., N- l. Step 3: Rearrange the index i such that Step 4: IF then stop (i.e., the solution is optimal) Else ; l=l+1 End-IF. Step 5: Go to Step 2.

20 Observation 1 The optimal solution has the following form :

21 Observation 2 Algorithm 1 always converges in, at worst, N-1 steps and thus the value of objective function at the optimal solution is

22 Numerical Examples Suppose the total amount of testing-effort expenditures W is given. We apply the proposed model to actual software failure data and have to allocate the expenditures to each module to minimize the expected cost. All the parameters a i and r i for each software module have been estimated by using the maximum likelihood estimation (MLE) or the least squares estimation (LSE).

23 Table 1 : the estimated values of a i, r i, v i, and . Moduleaiai riri  vivi × × × × × × × × × × An example for Proposed Algorithm

24 Sensitivity analysis If a 1 is increased by 40%, then the estimated value of optimal testing-effort expenditure for module 1 is changed from 7632 to 8400 and its relative change (RC) is about 10% increment). for modules 2, 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are decreased by about 0.95%, 0.95%, 1.52%, 0.67%, 1.93%, 2.87%, 2.30%, and 4.49%, respectively.

25 Sensitivity analysis (contd.) Figure 1: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% increase to a 1.

26 If a 1 is decreased by 30% the estimated value of optimal testing-effort expenditure for module 1 is changed from 7632 to 6818 and its RC is (about 10.66% decrement). for modules 2, 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are increased by about 1.01%, 1.02%, 1.64%, 0.71%, 2.06%, 3.05%, 2.44%, and 4.86%, respectively. Sensitivity analysis (contd.)

27 Sensitivity analysis (contd.) Figure 2: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% decrease to a 1.

28 Sensitivity analysis (contd.) If a 1 & a 2 both are increased by 40% the estimated values of optimal testing-effort expenditure for modules 1 and 2 are changed from 7632 to 8370 (about 9.67% increment) and 3158 to 3764 (about 19.18% increment), respectively. for modules 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are decreased by about 1.75%, 2.79%, 1.23%, 3.52%, 5.25%, 4.19%, and 8.22%, respectively.

29 Sensitivity analysis (contd.) Figure 3: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% increase to a 1 & a 2.

30 Conclusions In this paper, we proposes a method to optimize the software testing-resource allocation problem —minimizes the cost of software development, with a given fixed amount of testing-effort and a reliability objective. We develop a comprehensive strategy for module testing in order to help software project managers make the best decisions in practice.

31 Conclusions (contd.) An extensive sensitivity analysis is presented to study the effects of various principal parameters on the optimization problem of testing-resource allocation. Using Algorithm 1, project managers can allocate limited testing-resource easily and efficiently and thus achieve the lowest cost objective during software module and integration testing.