What Do We Know about Defect Detection Methods P. Runeson et al.; "What Do We Know about Defect Detection Methods?", IEEE Software, May/June Page(s):

Summary A survey of defect detection studies comparing inspection and testing techniques A survey of defect detection studies comparing inspection and testing techniques Yields practical recommendations: Yields practical recommendations: Use inspections for requirements and design defects Use inspections for requirements and design defects Use testing for code Use testing for code

Article Vocab Review Evidence-based software engineering Evidence-based software engineering Determines which methods to use and for what purpose Determines which methods to use and for what purpose EBSE involves relevant questions, surveying and appraising available evidence EBSE involves relevant questions, surveying and appraising available evidence Integrates and evaluates new practices in a target environment Integrates and evaluates new practices in a target environment Used as basis of article to help determine most efficient and effective methods for defect detection Used as basis of article to help determine most efficient and effective methods for defect detection

Article Vocab Review (con’t) Defect Defect Relates to one or more underlying faults in an artifact such as code Relates to one or more underlying faults in an artifact such as code Maps to single faults (faults and defect are used interchangeably) Maps to single faults (faults and defect are used interchangeably) Artifact Artifact Requirements Requirements Design Design Code Code Using industrial artifacts improves the studies generalizabilty Using industrial artifacts improves the studies generalizabilty

Types of Defects Grammatical defects are much different from missing requirements in a requirements specification Grammatical defects are much different from missing requirements in a requirements specification Testing and inspection methods may deem better or worse depending on the type of defect Testing and inspection methods may deem better or worse depending on the type of defect Omission (missing) vs. Commission (incorrect) Omission (missing) vs. Commission (incorrect) Unimportant, unimportant, crucial Unimportant, unimportant, crucial Classify defects on origins : Classify defects on origins : Requirements Requirements Design Design Code Code

Actors & Technique Who is the reviewer or tester Who is the reviewer or tester Freshman student in an experiment Freshman student in an experiment Experienced software engineer Experienced software engineer Technique Technique Relative to inspection and testing Relative to inspection and testing Inspection – families of verification techniques Inspection – families of verification techniques Testing – structural (white box) & functional (black box) Testing – structural (white box) & functional (black box)

Purpose What's the purpose of the inspection and testing activity? What's the purpose of the inspection and testing activity? Validation – assuring the correct system is developed Validation – assuring the correct system is developed Verification – assuring the system meets its specifications Verification – assuring the system meets its specifications Both? Both? Detection (for later isolation) vs. Isolation (isolate the underlying fault) Detection (for later isolation) vs. Isolation (isolate the underlying fault)

Defect detection activities Consider defect origins: Consider defect origins: Primary defect detection (1) Primary defect detection (1) Secondary defect detection (2) Secondary defect detection (2)

Evaluation criteria Efficiency - # of defects found per time unit spent on verification Efficiency - # of defects found per time unit spent on verification Effectiveness – the share of the existing defects found Effectiveness – the share of the existing defects found

The Survey (finally)!!!!!

Data from the studies

The issues covered from Table 2&3 Requirements defects Requirements defects Design defects Design defects Code defects Code defects Different defect types Different defect types Efficiency versus effectiveness Efficiency versus effectiveness

Requirements defects – Thank god for the experts! Different requirements inspection methods; same conclusion: Different requirements inspection methods; same conclusion: Spending effort up front to establish a good set of requirements is more efficient than developing a system on basis of incorrect requirements and then reworking it Spending effort up front to establish a good set of requirements is more efficient than developing a system on basis of incorrect requirements and then reworking it

Design defects Is inspecting design documents or testing the implemented function more efficient? Is inspecting design documents or testing the implemented function more efficient? Inspections were significantly more effective and efficient than testing Inspections were significantly more effective and efficient than testing More than half of the defects (53.5 %) found during inspection More than half of the defects (53.5 %) found during inspection Less found during testing (41.8 %) Less found during testing (41.8 %) Bottom line : design inspection is the winner! Bottom line : design inspection is the winner!

Code defects Which is better? Code inspection or testing (functional or structural)? Which is better? Code inspection or testing (functional or structural)? Which is more effective? Which is more effective? Which is more efficient? Which is more efficient? No clear winner No clear winner We’ll see why next slide We’ll see why next slide

Average effectiveness for code defect detection

Average efficiency for code defect detection

Survey issues Different defect types Different defect types Absence or presence of different types of defects might affect efficiency and effectiveness Absence or presence of different types of defects might affect efficiency and effectiveness Studies report defects by differing types and cannot be used to investigate whether the differences depend on fault type, not only the technique Studies report defects by differing types and cannot be used to investigate whether the differences depend on fault type, not only the technique Frequency of defects vary widely Frequency of defects vary widely Only a fraction of defects were found among studies – affects defect type frequency Only a fraction of defects were found among studies – affects defect type frequency Classification schemes involve subjective judgment Classification schemes involve subjective judgment Conclusion: jury is out as to the effect of defect types on the performance of inspection versus testing Conclusion: jury is out as to the effect of defect types on the performance of inspection versus testing

Survey issues (con’t) Effectiveness and efficiency Effectiveness and efficiency Only 25 to 50 percent of defects found during inspection Only 25 to 50 percent of defects found during inspection 30 to 60 percent found during testing 30 to 60 percent found during testing More than HALF defects remain!! More than HALF defects remain!! Conclusion: Primary defect detection methods’ low effectiveness and efficiency values is that secondary detection methods play a larger role that the surveys conclude Conclusion: Primary defect detection methods’ low effectiveness and efficiency values is that secondary detection methods play a larger role that the surveys conclude

The results are in! For requirements defects, no empirical evidence exists at all, but the fact that costs for requirements inspections are low compared to implementing incorrect requirements indicates that reviewers should look for requirements defects through inspection.

The results are in! For design specification defects, the case studies and one experiment indicate that inspections are both more efficient and more effective than functional testing.

The results are in! For code, functional or structural testing is ranked more effective or efficient than inspection in most studies. Some studies conclude that testing and inspection find different kinds of defects, so they’re complementary. Results differ when studying fault isolation and not just defect detection.

The results are in! Verification’s effectiveness is low; reviewers find only 25 to 50 percent of an artifact’s defects using inspection, and testers find 30 to 60 percent using testing. This makes secondary defect detection important. The efficiency is in the magnitude of 1 to 2.5 defects per hour spent on inspection or testing.

Results Summary Table

Questions ???