Measuring and Estimating Software Defects Loren Stroup UCF EEL6883: Software Engineering II

Outline Introduction To Topic Current Measures for Estimating Defects Preventative Methods Management Decisions Capability Maturity Model and Defect Prevention Personal Work Experience Conclusion

Introduction Software is in demand As software complexity has increased so has the need for software quality The goal of software reliability is to make software behave the way it is intended

Introduction: Software Defects Over Time Figure shows number of defects over time Most defects found during debugging, unit testing, and system level testing At some point, software defects will continue to rise and an upgrade is needed IBM studies have shown software defects found after coding are 10x more costly to correct than found during testing

Introduction: Software Defect Costs Driven by the amount an organization invests in software development U.S. businesses were investing roughly $250 billion in software development annually in 1996 A single post-release defect can cost between $20,000 and $40,000 Improving quality and reducing software defects has a conservative return of $7.50 to every $1 Software quality budget of $40,000 can return a savings of $300,000 in preventing most software defects

Current Measures For Estimating Defects Defect Detection Rate Ratio of the number of defects found per the size of the work product The number of defects per page of code or per 1000 SLOC Compared to previous projects and established thresholds A high DDR could mean experienced coders doing the peer review or latent defects from previous release of software A low DDR could mean inexperienced coders are review the software or more experienced coders are doing the development

Current Measures For Estimating Defects (cont) Defect Removal Efficiency Percentage of software defects that are removed by code reviews, inspections, and tests In use since the 1960s Current U.S. average is about 85% Companies such as IBM, AT&T, and Hewlett-Packard have gotten to 99%

Current Measures For Estimating Defects (cont) Table shows defect removal efficiency for several activities No one activity has the best efficiency overall All the activities working together help software quality

Current Measures For Estimating Defects (cont) Defect Density Number of defects per size of the software project Size of project measured in LOC Useful metric in determining if a software release is ready for delivery to the customer

Preventative Methods Software Inspections (peer reviews) Introduced by Michael Fagan in the 1970s at IBM Involves reviewing a software work product by a group of trained individuals Reviewers review the software code changes and relay comments back to the author for clarification or further work Initial costs are in training individuals on the process Software developers are reviewing instead of coding

Preventative Methods Pair Programming Two software developers work together at one computer One developer is coding, the other developer is reviewing Potential to find defects earlier because the code is under constant review On average, PP leaves 15% fewer defects than conventional methods

Management Decisions Implement fixes prior to delivery Depends on severity and schedule Deliver a new software release Hold off the fixes until another software release, depends on severity Operation and Maintenance work Deliver a software patch release “out-of-cycle” releases Release accepted by customer, but patches will be delivered for any defects found during additional testing

Capability Maturity Model and Defect Prevention CMM level 2 Key process area is software quality assurance Starting to be process-oriented Main goals are planning quality assurance guidelines and enforcing them Beginning foundation to defect prevention

Capability Maturity Model and Defect Prevention CMM level 3 Key process area is peer reviews Focus is on the cause of software defects Processes begin to be repeatable CMM level 5 Defect prevention is a key process area Defect prevention is a strategy applied to the software lifecycle that identifies root causes of defects and prevents them from recurring

Capability Maturity Model and Defect Prevention Figure is a cost analysis of defect prevention Clearly, defects found earlier saves money Team can be set up dedicated to the defect prevention activities

Personal Work Experience Software development experience of 2 years Software test experience of 3 years Use peer reviews to minimize software defects Run unit tests and integration tests with test team witness Been a part of 2 major software releases and several patch releases Tough to anticipate all behaviors of system Must pay special attention to code changes when dealing with older code

Conclusion Defects are unavoidable but can be limited with the right tools Special metrics make it easier to plan releases and diagnose problems more quickly Preventing defects is a key process area in many of the CMM levels It is very beneficial in the long run for a company to invest money in processes to reduce defects for the time of release of the software

Cited Works Pan, Jiantao. Software Reliability. Website, 1999, Wikipedia. Software quality. Website, 2008, Subramaniam, Bala. Eective Software Defect Tracking. Website, 1999, /subramaniam.asp. /subramaniam.asp Lett, Steven H. Using Peer Review Data to Manage Software Defects Web Document, 2008, defects.pdf. defects.pdf Jones, Capers. "Software Defect Removal Efficiency", in Computer, April 1996, Software Productivity Res. Inc., IEEE. McConnel, Steve. "Gauging Software Readiness With Defect Tracking", in IEEE Software, May/June 1997, Construx Software, IEEE. Software Engineering Institute. Software Inspections. Website, 2007,

Cited Works (cont) Wikipedia. Software inspection. Website, 2008, Wikipedia. Pair programming. Website, 2008, Padberg, Frank and Muller Matthias M. "Analyzing the Cost and Benet of Pair Programming", in Proceedings of the Ninth International Software Metrics Symposium, 2003, The Computer Society, IEEE. Wikipedia. Capability Maturity Model. Website, 2008, Paulk, Mark C., Weber, Charles V., Garcia Suzanne M., Chrissis, Mary Beth, Bush, Marilyn. Software Quality Assurance. Website, Narayana, Purushotham. "Software Defect Prevention - In a Nutshell", in iSixSigma, , iSixSigma LLC, Web Document:

Questions! If you have any questions, feel free to contact me