1. Tom Ostrand Elaine Weyuker Bob Bell AT&T Labs – Research
Using Static Analysis to Determine Where to Focus Dynamic Testing Effort
Tom Ostrand
Elaine Weyuker
Bob Bell
AT&T Labs – Research

2. GOAL: Predict Which Files in a Software System are Most Likely to Contain Faults

3. Why is this important?
Because this should allow us to
build highly dependable software
systems more economically by
allowing us to better allocate testing
effort and resources, including
personnel. Prioritize testing.

4. Subgoal 1: Determine Whether Faults are Non-uniformly Distributed
We did a large case study using 12 successive releases (about 3 years in the field) of an inventory control system containing roughly ½ million LOC

5. Number of Files
Release Number

6. Distribution of Faults
Percent Faulty Files
Release Number

7. Numbers of Files and Faults
Release Number

8. Number of Files and Number of Files Containing any Faults

9. Subgoal 2: Identify which characteristics are associated with the files that contained the largest numbers of faults

10. Considered Structural Characteristics of Software
Size (LOC)
Complexity (cyclomatic number)
Age
New to the current release, and if not, whether it was changed during prior release?
The number and magnitude of changes made.
The number of observed faults during earlier releases.
The number of faults observed during early development stages.
Programming language used.

11. Findings
Size is the best predictor of fault-proneness when assessed in terms of number of faults; not a good predictor if considering fault density.
New files and files that had been changed in the last release were significantly more likely to contain faults than files that had not changed.
Other weaker factors include file age, number of faults in previous release, program type and release number.

12. Fault Density: New, Changed, Unchanged

13. Doing the Actual Prediction
We used the identified characteristics, properly weighted, to build a statistical model (negative binomial regression model), and did the prediction for the releases 3 through 12.

14. Percent of Faults Correctly Predicted by 20% of Files Selected by Model (Average = 80%)
Release

15. Final Thoughts Used static analysis to influence dynamic testing.
By determining file fault-proneness, testers can prioritize testing effort.
Now have another 5 releases to analyze, and another 2 systems. Initial indications are that results are similar or better as system ages.