1. Confidence intervals in software reliability testing
Alessandro Di Bucchianico (LaQuSo, Eindhoven University of Technology)
Ed Brandt and Rob Henzen (Refis, Netherlands)
ENBIS-5
Newcastle
September 15, 2005

2. Goals of this talk
show how to obtain confidence intervals for software reliability predictions from NHPP models
apply results to case study

3. Introduction of LaQuSo and Refis
Overview of this talk
Introduction of LaQuSo and Refis
Case Dutch Ministry of Transport, Public Works and Water Management
Software reliability models
Confidence intervals for NHPP models:
asymptotics
simulation
goodness-of-fit tests
Conclusions

4. LaQuSo: Laboratory for Quality Software
university based laboratory
started at the Eindhoven University of Technology
Radboud University (Nijmegen) has recently joined as partner
statistics and probability group in math department at TU/e is one of the participating groups
started in January 2004: 10 fte; will grow to 50 fte
case-study driven in cooperation with industry
statistics will be integrated part of testing and verification activities
more information:

5. consultancy company in Bilthoven, the Netherlands activities include:
Refis
consultancy company in Bilthoven, the Netherlands
activities include:
software reliability assessments
measurements systems for IT sector
test audits
for more information, see

6. 2/3 of the Netherlands is below sea level
Context of case
2/3 of the Netherlands is below sea level
protection against sea and rivers by
dunes
dikes
dams
sluices …
hardware reliability of sluices is well understood and documented
control of sluices by huge software systems (reliability??)

7. Sluice (1)

8. Sluice (2)

9. Obtain information on reliability of software system
Goals of case
Case is project of Dutch Ministry of Transport, Public Works and Water Management (see for general information)
Obtain information on reliability of software system
Registration system for defect detection and repair
Predict system reliability with confidence bounds

10. Data available from three tests:
Available data
Data available from three tests:
plant acceptation test
site acceptation test
site acceptation retest
Defect counts
grouped data
severity index
repair status …
Data was collected manually and checked on consistency etc.

11. Data assumptions
Assumptions are results from intensive discussions with project and test engineers
all test intervals have same effort
every test period corresponds to 219 days of actual use
immediate correction of errors (gaps between testing periods allowed for this)
no new error introduced by correction actions

12. Data (severity 1 FAT)

13. Software reliability models
Main differences with hardware reliability:
no wear
no burn-in
exact reproducibility of errors
Hundreds of reliability growth models available
Dedicated software for software reliability exists (not always reliable, though):
Casre
Smerfs …

14. Initial Model Selection
Models available in standard software reliability packages (Smerfs, Casre) were judged on several criteria (assumptions or properties), including:
upper bound on number of errors
interval data
length of test intervals
distribution of errors
shape of failure intensity …
The list of selected models included two NHPP models (Goel-Okumoto and Yamada S-shaped)

15. Nonhomogeneous Poisson processT1 T2 T3 T4 t
N(t )=4
This is a Type II model (cf. Langberg/Singpurwalla (1985)) that in general cannot be described easily in terms of time between failures.
Special case: Poisson process

16. NHPP models Several choices for  have been introduced:
Goel-Okumoto, Musa
delayed S-shaped
inflection S-shaped
hyperexponential
logarithmic

17. NHPP models: inference for grouped data
data consists of counts in time intervals:
ni = # detected failures in time interval (ti-1,ti]
likelihood function (t0=0):
(t) = cumulative hazard rate at time t = expected number of failures at time t
if  has parametric form, then maximizing L yields ML estimates for parameters
(t) = d/dt (t) = hazard rate at time t

18. NHPP models with 2 parameters: inference for parameters
Assume  depends on 2 parameters a and e
ML-estimators have no closed form
asymptotic distribution through Fisher information:

19. NHPP models with 2 parameters: inference for function of parameters
assume  depends on two parameters a and b
asymptotic distribution of functions of a and b through Fisher information and delta method:
examples of functions of parameters include:
probability of no failure in certain time period
failure intensity at t=t0

20. Simulation NHPP processT1 T2 T3 T4 t
N(t )=4
Conditional on the event N(t)=n, the T1,…,Tn are distributed as the order statistics of a sample of size n from a distribution with density (t) / (t).
Hence, simulating a sample from a distribution with density (t) / (t) can be used to simulate an NHPP process with intensity (t)

21. Goodness-of-fit NHPP processT1 T2 T3 T4 t
N(t )=4
Conditional on the event N(t)=n, the T1,…,Tn are distributed as the order statistics of a sample of size n from a distribution with density (t) / (t).
Hence, the Kolmogorov goodness-of-fit test based on the empirical distribution function may be used to perform a GOF test.

22. goodness-of-fit: OK at 5% level
Back to case study
parameter estimates and 95% confidence intervals for Goel-Okumoto model (a(1-exp(b t)):
a : ( , )
b = ( , )
goodness-of-fit: OK at 5% level
important question from Dutch politics: 95% confidence interval for probability of no failure in 1 year:
( , 1 ) (thus confirmation of suspicion by Ministry officials that defect system is not good enough for required probabilities)

23. Conclusions
asymptotic confidence intervals for functions of parameters in NHPP models may obtained from Fisher information
testing registration of Dutch water works not sufficient to obtain high-precision estimates of software reliability

24. Literature
Rijkswaterstaat report (confidential)
Systematic description of software reliability models, manuscript in progress (ADiB + Refis)
Xie and Hong (2001), Handbook of statistics 20 (Advances in Reliability),