S Neuendorf 2004 Prediction of Software Defects SASQAG March 2004 by Steve Neuendorf

S Neuendorf 2004 Affiliations –Independent Consultant –David Consulting Group –Human Systems Knowledge Networks

S Neuendorf 2004 introduce which are observed as FAULTSDEFECTSFAILURES are discovered in products Before release Internal Defects or Errors (External) Defects Yes No Faults, Defects and Failures There is considerable disagreement about the definitions of defects, errors, faults and failures. (Fenton)

S Neuendorf 2004 Defect Prediction # Created? # Found? # Fixed? Release # Found? # Left? # Created # Found # Left

S Neuendorf 2004 Defect Addition Rates PhaseDefects per FP Requirements1.00 Design1.25 Coding1.75 Documentation0.60 Bad Fixes0.40 Total5.00 Jones 5

S Neuendorf 2004 Defect Removal Efficency 55%60%65%70%75%80%85%90%95%100% MalpracticeUS AverageBest in Class Defects per FP Defect Removal Efficiency Jones 5

S Neuendorf 2004 Other Considerations Not much information about the relationship between defects and failures No relationship between defect(s) and failure severity No consistency in terminology (BSOD = Bug?)

S Neuendorf 2004 Defect Prediction Many types of Models –Process Models –Multivariate Models –Size and Complexity metrics –Belief Models –...

S Neuendorf 2004 The Components of a Process

S Neuendorf 2004 Process Models Actual Performance Predicted Performance (e.g., E, Q or T) Process Model Current Capability Management Teams Tools Techniques Technology Environment

S Neuendorf 2004 Process Models Phase Containment Models –Rely on history to identify How many defects were produced in each phase, How many defects from that phase were discovered and corrected (Phase Containment) –Predict defects for each phase and track discovery and removal. Assume that defects predicted and not found were passed to the next phase. –Simple, easy to implement with common tools.

S Neuendorf 2004 Multivariate Models Analogous to Parametric Estimating Models. (e.g., COQUALMO) Uses any of many variables and analysis of the relationships of the values for those variables and the results observed in historic projects. Good if you have a good match for the projects from which the model is created.

S Neuendorf 2004 Size and Complexity metrics Given enough “Defects per” data, all that is needed is the “per What” to predict defects – Right? Size and Complexity do not cause defects – faults do.

S Neuendorf 2004 Using Testing Metrics Look at total defects found in each phase pre and post release, including analysis as to where the defect originated. Use statistics to determine total defects introduced in each phase and use that as a predictor. With a stable environment, very high confidence (95%) is claimed. Good opportunity for Benchmarking Approach.

S Neuendorf 2004 Using Process Quality Data SEI CMM LevelsDefect PotentialsRemoval EfficiencyDelivered Defects 1585% % % % %0.01 Jones 5

S Neuendorf 2004 Bayesian Belief Networks (BBN) Seems to be the area of most interest. Combines history with reality. Old Concept (1763) – New life (we got tools) Graphical network of the probabilistic relationships between variables. –Uses expert beliefs about the relationships assess the the impact of evidence on the probabilities of outcomes

S Neuendorf 2004 Staff Experience Problem Complexity Use of Standard Code Complexity Use Reliability # of Latent Defects Coders’ Performance Adapted from Fenton 4 Example BBN

S Neuendorf 2004 Node probability table (NPT) for the node 'Reliability' Uselowmedhigh Defectslowmedhighlowmedhighlowmedhigh Reliability low med high

S Neuendorf 2004 References 1.A Bayesian Approach to Deriving Parameter Values for a Software Defect Predictive Model; Robert Stoddard, John Henderson, Ph.D.; Sixth International Conference on the Applications of Software Measurement. 2.A Critique of Software Defect Prediction Models; Norman Fenton, Martin Neil; Centre for Software Reliability 3.Predicting Software Errors and Defects; Mark Criscione, et.al. 4.Predicting Software Quality using Bayesian Belief Networks; Martin Neil & Norman Fenton 5.Estimating Software Costs, Jones, McGraw-Hill, 1999