1 POP Quiz T/F Defect Removal Effectiveness and Defect Removal Models are not true Predictive Models Define DRE What is a Checklist? What is it for? What is Risk Management? What is it for? What’s the difference between proactive and reactive risk management? Briefly describe a Pareto Diagram

2 Software Quality Engineering CS410 Class 7 Defect Removal Effectiveness

3 Defect Removal A method for identifying and removing defects at every phase of the development life cycle. Results of defect removal: –Reduction in product expense –Reduction in product development time –Increase in development team productivity –Increase in product quality

4 Defect Removal Defect Removal History –1960’s “code and test” methodology cost overruns and schedule delays the only defect removal step was test –1970’s formal reviews and inspections emerged productivity and quality improved –(mid) 1980’s Defect Removal, Defect Removal Effectiveness, and Defect Removal Metrics, appear in literature

5 Defect Removal Effectiveness General Equation For Effectiveness Where: –‘Defects Present At Stage’ are defects at entry to stage plus defects injected during stage, and counted as: Defects Found + Defects Not Found –‘Defects not found’ is determined at later stages –Must account for where defects should have been caught (I.e. defect must have existed in that stage to be counted against effectiveness of stage)

6 Defect Removal Effectiveness Example: fig. 6.4 p. 159 Defects at stage entry = total defects for all stages so far minus the total defects removed so far. Also equal to the latent defects from the previous stage. Defect Removal effectiveness of stage = defects removed by stage / defects total at stage end.

7 Defect Removal Effectiveness Quality Planning –Defect Removal Effectiveness by stages provides a clear indicator of where quality improvements should be focused. –Release to release monitoring (comparing) can also give a good feel for process capability. –Historical data can be used for control charting and process tracking.

8 Defect Removal Model (DRM) A quality management tool which can help to answer “what if”? For example: What if we improved our code inspection DRE by 5%? What would be the resulting quality improvement on the product? Summarizes the interrelations among three metrics: Defect Injection (input) Defect Removal Effectiveness (input) Defect Removal (output) Table 6.3 p. 166

9 Defect Removal Model (DRM) Code Inspection DRE increased by 5% reduces defects to field from.8/KLOC to.71/KLOC

10 Two-Phase Model Remus and Zilles 1979 Model assumes –There are only two phases of defect removal Front End - Development activities, reviews etc. (MPs) Back End - Formal testing (PTRs) –DRE of two phases is the same u = MP/PTR (ratio of front end to back end DRE) Q = Defects released to field TD = Total Defects (MP + PTR + Q) Q = PTR / (u-1) Q = MP / u(u-1) Q = TD / u 2

11 Two-Phase Model –Model Example: –We can see that the more defects we remove at the Front End the less defects we expect to ship to the field. –The model works as long as MP > PTR

12 Cost Effectiveness Cost of defect removal must be considered for efficient quality planning Defect removal at front end is less expensive due to less removal effort, and less rework times less expensive (Fagan) 50-80% cost reduction by early defect removal -Freedman and Weinberg Inspections (Reviews and Walkthroughs) are the key front end defect removal method

13 Inspections Michael Fagan (IBM) first introduced formal inspection in 1976 as a way to improve product quality, and the manageability of the development process Five (six) part process 1. Planning/Overview - For communication and involvement 2. Preparation - For education of inspectors 3. Inspection Meeting - To find defects 4. Rework - To fix defects 5. Follow-up - To track process and quality

14 Inspections Knight and Myers (1991) introduced the phased inspection method Each phase of inspection focuses on different property of product (I.e. portability, reusability, or maintainability) Also incorporates two types of phases –Single-inspector phase - rigidly formatted process that performs unambiguous checks –Multi-inspector phase - multiple independent examinations of quality aspects are performed and then compared

15 Defect Removal Summary Matrix approach to Defect Removal Effectiveness works well (can setup a spreadsheet) The shorter the time between defect origin and defect removal, the more effective and less expensive the process will be Front end defect removal (Inspections) are the most cost effective. Defect Removal Effectiveness and Defect Removal Models are not true Predictive Models (next chapter)

16 Pop Quiz What’s the difference between Single- inspector phase and Multiple-inspector phase? T/F The shorter the time between defect origin and defect removal, the more effective and less expensive the process will be Explain the results documented here:

17 Software Quality Engineering CS410 Class 7b Reliability Models Rayleigh Model

18 Reliability Models Reliability models are used to estimate the reliability of a software product when it becomes available to the customer Reliability equates to latent defects Important for two reasons: –It is an objective statement of the product quality –It is a resource planning tool for the maintenance phase

19 Reliability Models Static Reliability Models –Uses other attributes of the project or program modules to estimate the number of defects in the software –General Form: y = f(x 1, x 2,,, x k ) + e Said: The number of defects (y) is dependant on the attributes (x) of the product and the process by which it is produced, plus some error (e) due to unknowns which inherently exist.

20 Reliability Models –General Form (cont.): y = f(x 1, x 2,,, x k ) + e y is said to be the dependant variable x’s are the independent variables the coefficients of the independent variables are estimated based on data from previous products the values of the independent variables are measured and then plugged into the equation to determine the estimated dependant variable

21 Reliability Models –Static models are considered static because the coefficients of the parameters are based on previous projects (I.e. historical data) –The current product is treated as an additional observation in the same population of products. –Static models are better suited for hypothesis testing (trends, correlation), and not as suitable for product-level reliability estimating

22 Reliability Models Dynamic Reliability Models –Based on statistical distributions, and uses the current development defect patterns to estimate end product quality. –Parameters are estimated based on multiple data points gathered to date from the product of interest. –The resulting model is specific to the product for which the projection of reliability is attempted (intended).

23 Reliability Models Dynamic models are all expressed as a function of time in development or it’s logical equivalent (development phase - I.e. fig 7.2 p. 182) Two categories of Dynamic Models: –Models for entire development process Example: Rayleigh Model –Models for the back-end formal testing phase Chapter 8

24 Rayleigh Model Rayleigh model is a member of the Weibull distribution family Weibull distributions are used for predicting reliability and probability distribution Two standard functions for graphing Weibull

25 Rayleigh Model Rayleigh is a special case of the Weibull where the shape parameter (m) equals 2: The formulas represent a standard distribution. The total area under the curve is 1.

26 Rayleigh Model It has been empirically well established that software projects follow a life-cycle pattern described by the Rayleigh density curve. Applications of the model can be used for staffing requirements over time, and defect removal patterns. The model has been validated with real systems and their resulting actual defect rates.

27 Rayleigh Model Assumptions of the model: 1. The defect rate observed during the development process is positively correlated with the defect rate in the field. Assuming the defect removal effectiveness (or the organization) remains unchanged, then a higher curve (more defects) during development means a higher defect injection rate and hence a higher field defect rate.

28 Rayleigh Model Assumptions of the model: 2. Given the same error injection rate, if more defects are discovered and removed earlier then fewer will remain in later stages and the field quality will be better. Fig 7.4 p. –The assumptions relate to the “Do it right the first time” principle. If each step of the process is executed properly then the end product quality will be good due to fewer errors injected and errors detected early.

29 Rayleigh Model Assumptions a different way: –If the error injection rate is low then the chance of errors making it to the field is reduced due to the lower number of total errors present. –If the process is detecting errors in the front- end then the chance of the errors making it to the field is reduced due to the higher chance that errors are detected and require minimal rework. Point: Heavy rework can detract from finding additional errors.

30 Rayleigh Model Implementation of Rayleigh Model Most statistical software packages support Weibull Distributions. Applications can be developed due to the clearly defined algorithms for Weibull. COTS (Commercial Off The Shelf) products can also be used: –Software Life-cycle Model tool (SLIM) –Software Error Estimation Reporter (STEER) table 7.2 p. 192

31 Rayleigh Model Reliability –Modeling reliability refers to the degree of change in the model output due to change fluctuations in the input data. –Confidence interval of estimate is determined by the model reliability and is generally increased with larger sample sizes. Predictive Validity –Accuracy of model estimates –Input data must be accurate and reliable

32 Rayleigh Model Predictive Validity (cont.) –To establish high Predictive Validity, model estimates and actual outcomes must be compared and empirical validity must be established. –The validity of software reliability models is context specific. A certain model may work well for a specific organization or development structure, but not for others. –No universally good software reliability model exists.

33 Rayleigh Model Summary Rayleigh model is a special case of Weibull Rayleigh is a formal parametric model which can be used for projecting latent defects in a software product Software projects generally follow a life- cycle pattern described by the Rayleigh curve Rayleigh model should be calibrated using an adjustment factor, based on actual data from previous software projects