1. Main Title Slide Software Reliability Estimates/ Projections, Cumulative & Instantaneous - Dave Dwyer With help from: Ann Marie Neufelder, John D. Musa, Martin Trachtenberg, Thomas Downs, Ernest O. Codier and Faculty of Rivier College Grad. School Math and Computer Science

2. 10/29/20022 Martin Trachtenberg (1985): Simulation shows that, with respect to the number of detected errors: –Testing the functions of the software system in a random or round- robin order gives linearly decaying system error rates. –Testing each function exhaustively one at a time gives flat system- error rates –Testing different functions at widely different frequencies gives exponentially decaying system error rates [Operational Profile Testing], and –Testing strategies which result in linear decaying error rates tend to require the fewest tests to detect a given number of errors.

3. 10/29/20023 Thomas Downs (1985): “In this paper, an approach to the modeling of software testing is described. A major aim of this approach is to allow the assessment of the effects of different testing (and debugging) strategies in different situations. It is shown how the techniques developed can be used to estimate, prior to the commencement of testing, the optimum allocation of test effort for software which is to be nonuniformly executed in its operational phase.”

4. 10/29/20024 There are Two Basic Types of Software Reliability Models Predictors - predict reliability of software at some future time. Prediction made prior to development or test as early as concept phase. Normally based on historical data. Estimators - estimate reliability of software at some present or future time based on data collected from current development and/or test. Normally used later in life cycle than predictors.

5. 10/29/20025 A Pure Approach Reflects the True Nature of Software The execution of software takes the form of the execution of a sequence of M paths. The actual number of paths affected by an arbitrary fault is unknown and can be treated as a random variable, c. Not all paths are equally likely to be executed in a randomly selected execution profile.

6. 10/29/20026 12 3 M x1 x2 2 paths affected by x1 1 path affected by x2 xN ‘N’ total faults initially ‘M’ total paths Start ‘c’ paths affected by an arbitrary fault

7. 10/29/20027 Further...  In the operational phase of many large software systems, some sections of code are executed much more frequently than others.  Faults located in heavily used sections of code are much more likely to be detected early.

8. 10/29/20028 Downs (IEEE Trans. on SW Eng. April, 1985) Showed that Approximations can be Made  Each time a path is selected for testing, all paths are equally likely to be selected. The actual number of paths affected by an arbitrary fault is a constant

9. 10/29/20029 My Data Assumptions Cumulative 8 Hr. test shifts are recorded VS the number of errors. Each first instance is plotted  The last data point will be at the end of the test time, even though there was no error, because a long interval without error is more significant than an interval with an error.

10. 10/29/200210 Other Assumptions  Only integration & system test data are used.  Problems will be designated as priority 1, 2 or 3 (Ref DoD-STD-2167A) where:  Priority 1: “Prevents mission essential capability”  Priority 2: “Adversely affects mission essential capability with no alternative workaround”  Priority 3: “Adversely affects mission essential capability with alternative workaround”

11. 10/29/200211 Downs Showed ~ faults/path j = (N – j) , where: –N = the total number of faults, –j = the number of corrected faults, –  = -r log(1 – c/M), r = the number of paths executed/unit time, c = the average number of paths effected by each fault and M = the total number of paths

12. 10/29/200212 Failure Rate is proportional to failure number, Downs: j  (N – j)r(c/M).

13. 10/29/200213 Failure rate plots against failure number for a range of non-uniform testing profiles, M 1, M 2 paths and N 1, N 2 initial faults in those paths. (Logarithmic?)

14. 10/29/200214 Imagine two main segments Segment 1 Segment 2

15. 10/29/200215 After testing segment 1, someone asks: Given 10 faults found, what’s the reliability of the code? Responses: –Don’t know how many other faults remain in section 1, let alone are in section 2 –Don’t know how often sections 1, 2 are used. –Did we plot failure intensity vs faults? –Why didn’t we test to the operational profile?

16. 10/29/200216 By reference to Duane’s derivation for hardware reliability, (Ref. E. O. Codier, RAMS - 1968)

17. 10/29/200217 Instantaneous Failure Intensity for Software

18. 10/29/200218 Priority 1 Data Plotted

19. 10/29/200219 Priority 1 and 2 Data Plotted

20. 10/29/200220 Point Estimates vs Instantaneous

21. 10/29/200221

22. For copy of paper, e-mail request to: david.j.dwyer@baesystems.com