1. Software Reliability Estimates/ Projections, Cumulative & Instantaneous Presented by Dave Dwyer With help from: Ann Marie Neufelder, John D. Musa, Martin Trachtenberg, Thomas Downs, Ernest O. Codier and Faculty of Rivier College Grad. School Math and Computer Science Rivier College Mathematics & Computer Science Lecture Series March 23, 2006

2. 03/23/2006Software Reliability2 Martin Trachtenberg (1985): Simulation shows that, with respect to the number of detected errors: –Testing the functions of the software system in a random or round- robin order gives linearly decaying system error rates. –Testing each function exhaustively one at a time gives flat system- error rates –Testing different functions at widely different frequencies gives exponentially decaying system error rates [Operational Profile Testing], and –Testing strategies which result in linear decaying error rates tend to require the fewest tests to detect a given number of errors.

3. 03/23/2006Software Reliability3 Thomas Downs (1985): “In this paper, an approach to the modeling of software testing is described. A major aim of this approach is to allow the assessment of the effects of different testing (and debugging) strategies in different situations. It is shown how the techniques developed can be used to estimate, prior to the commencement of testing, the optimum allocation of test effort for software which is to be nonuniformly executed in its operational phase.”

4. 03/23/2006Software Reliability4 There are Two Basic Types of Software Reliability Models Predictors - predict reliability of software at some future time. Prediction made prior to development or test as early as concept phase. Normally based on historical data. Estimators - estimate reliability of software at some present or future time based on data collected from current development and/or test. Normally used later in life cycle than predictors.

5. 03/23/2006Software Reliability5 A Pure Approach Reflects the True Nature of Software The execution of software takes the form of the execution of a sequence of M paths. The actual number of paths affected by an arbitrary fault is unknown and can be treated as a random variable, c. Not all paths are equally likely to be executed in a randomly selected execution profile.

6. 03/23/2006Software Reliability6 12 3 M x1 x2 2 paths affected by x1 1 path affected by x2 xN ‘N’ total faults initially ‘M’ total paths Start ‘c’ paths affected by an arbitrary fault

7. 03/23/2006Software Reliability7 Further...  In the operational phase of many large software systems, some sections of code are executed much more frequently than others.  Faults located in heavily used sections of code are much more likely to be detected early.

8. 03/23/2006Software Reliability8 Downs (IEEE Trans. on SW Eng. April, 1985) Showed that approximations can be made  Each time a path is selected for testing, all paths are equally likely to be selected. The actual number of paths affected by an arbitrary fault is a constant

9. 03/23/2006Software Reliability9 My Data Assumptions Cumulative 8 Hr. test shifts are recorded VS the number of errors. Each first instance is plotted  The last data point will be at the end of the test time, even though there was no error, because a long interval without error is more significant than an interval with an error.

10. 03/23/2006Software Reliability10 Other Assumptions  Only integration & system test data are used.  Problems will be designated as priority 1, 2 or 3 (Ref DoD-STD-2167A) where:  Priority 1: “Prevents mission essential capability”  Priority 2: “Adversely affects mission essential capability with no alternative workaround”  Priority 3: “Adversely affects mission essential capability with alternative workaround”

11. 03/23/2006Software Reliability11 Downs Showed: ~ faults/path j = (N – j) , where: –N = the total number of faults, –j = the number of corrected faults, –  = -r log(1 – c/M), r = the number of paths executed/unit time, c = the average number of paths effected by each fault and M = the total number of paths

12. 03/23/2006Software Reliability12 Failure Rate is proportional to failure number, Downs: j  (N – j)r(c/M)

13. 03/23/2006Software Reliability13 Failure rate plots against failure number for a range of non-uniform testing profiles, M 1, M 2 paths and N 1, N 2 initial faults in those paths. (Logarithmic?)

14. 03/23/2006Software Reliability14 Imagine two main segments Segment 1 Segment 2

15. 03/23/2006Software Reliability15 After testing segment 1, someone asks: Given 10 faults found, what’s the reliability of the code? Responses: –Don’t know how many other faults remain in section 1, let alone are in section 2 –Don’t know how often sections 1, 2 are used. –Did we plot failure intensity vs faults? –Why didn’t we test to the operational profile?

16. 03/23/2006Software Reliability16 By reference to Duane’s derivation for hardware reliability, (Ref. E. O. Codier, RAMS - 1968)

17. 03/23/2006Software Reliability17 Failure Intensity (Instantaneous Failure Rate) Derivation - Hardware & Software Duane’s Instantaneous for HWDave’s Instantaneous for SW ‘Failure Intensity’ Similar Result

18. 03/23/2006Software Reliability18 Priority 1 Data Plotted

19. 03/23/2006Software Reliability19 Priority 1 and 2 Data Plotted

20. 03/23/2006Software Reliability20 Point Estimates vs Instantaneous

21. 03/23/2006Software Reliability21

22. For copy of paper, e-mail request to: david.j.dwyer@baesystems.com