1. Estimating the number of components with defects post-release that showed no defects in testing
C. Stringfellow
A. Andrews
C. Wohlin
H. Peterson
Jeremy Mange

2. Motivation For nearly any product, defects will appear after release
The defects that were not detected in testing point to problems in the test process
We would like to know how many of these post-release defects to expect

3. Motivation
Using these estimates, release / re-test decisions can be made
The authors wish to compare methods of post-release defect estimation

4. Defect Estimation
The paper compares three methods of this type of estimation
Experience-based
Capture-recapture
Curve-fitting
Experience-based requires historical data, Capture-Recapture and Curve-fitting do not

5. Experience-based Estimation
Requires some sort of historical data:
Defect data
Change data
Historical data can be from previous products or prior releases of the same product
If from previous products, they must be similar for these estimations to work

6. Experience-based Estimation
Defect data
Number of faults in a module can be estimated using the defect history of that module
Change data
Number of faults in a module can be estimated using the number of file changes
Recent changes can be weighted more heavily

7. Capture-recapture Estimation
A type of model originally designed for reviews and inspections
Compares the number and types of defects found from multiple test sites
Applies statistical estimation to the data based on assumptions about test sites

8. Capture-recapture Estimation
Differences between test sites are accounted for in two areas:
Ability to find defects
Probabilities of finding specific defects
This yields four models:
Same ability, same probabilities
Same ability, different probabilities
Different ability, same probabilities
Different ability, different probabilities

9. Capture-recapture Estimation

10. Curve-fitting Estimation
Fits a mathematical curve to the data points to estimated remaining defects
Two basic types:
Decreasing – plot number of test sites that found each defect, sorted in decreasing order
Increasing – plot cumulative number of defects found by each testing event

11. Curve-fitting Estimation
With both increasing and decreasing models, the fitted curve is used to predict the number of remaining defects
Both exponential and linear prediction models exist for each type

12. Approach 1. Collect the data
For each discovered defect, count the number of test sites that discovered it

13. Approach 2. Apply non-historical methods
Both capture-recapture and curve-fitting models are applied
These provide estimates of the number of total defects

14. Approach 3. Apply experience-based method
Use historical data to estimate the number of post-release defects
For each past project (or release), calculate:

15. Approach 4. Using the calculations, estimate post- release defects
For non-historical models, subtract the number of defects found in testing from the calculated total defects estimate

16. Approach 5. Compare this estimate to a decision threshold value
If the number of expected post-release defects is acceptable, the product can be released
If not, further testing must be performed
Of course, the actual release decision will be based on many criteria, this is just one of those

17. Results
This approach was carried out in a case study on a medical record system
Results:
Capture-recapture and curve-fitting: 5%-20% error
Experience-based: 3%-5% error

18. Conclusion
If data for similar projects is available, experience-based models provide the best estimations for post-release defects
However, non-historical (capture-recapture and curve-fitting) models also provide fairly accurate estimates independent of past data
Post-release defect estimates should be used in release decisions

19. Questions?