Confidence intervals in software reliability testing

Slides:



Advertisements
Similar presentations
Software Reliability Engineering Tool Nederlandse Testdag 27 november 2008 Alessandro di Bucchianico, Lime – TU/e Ed Brandt, Refis VOF.
Advertisements

LaQuSo is an activity of Technische Universiteit Eindhoven Confidence intervals in software reliability testing Confidence intervals in software reliability.
Sta220 - Statistics Mr. Smith Room 310 Class #14.
Chapter 11- Confidence Intervals for Univariate Data Math 22 Introductory Statistics.
Software Reliability Engineering
A Unified Scheme of Some Nonhomogenous Poisson Process Models for Software Reliability Estimation C. Y. Huang, M. R. Lyu and S. Y. Kuo IEEE Transactions.
Goodness of Fit of a Joint Model for Event Time and Nonignorable Missing Longitudinal Quality of Life Data – A Study by Sneh Gulati* *with Jean-Francois.
1 Software Reliability Growth Models Incorporating Fault Dependency with Various Debugging Time Lags Chin-Yu Huang, Chu-Ti Lin, Sy-Yen Kuo, Michael R.
Jump to first page STATISTICAL INFERENCE Statistical Inference uses sample data and statistical procedures to: n Estimate population parameters; or n Test.
2. Point and interval estimation Introduction Properties of estimators Finite sample size Asymptotic properties Construction methods Method of moments.
4-1 Statistical Inference The field of statistical inference consists of those methods used to make decisions or draw conclusions about a population.
Statistics and Probability Theory Prof. Dr. Michael Havbro Faber
1 Fundamentals of Reliability Engineering and Applications Dr. E. A. Elsayed Department of Industrial and Systems Engineering Rutgers University
Testing models against data Bas Kooijman Dept theoretical biology Vrije Universiteit Amsterdam master course WTC.
INFERENTIAL STATISTICS – Samples are only estimates of the population – Sample statistics will be slightly off from the true values of its population’s.
Overview Software Quality Assurance Reliability and Availability
ECE355 Fall 2004Software Reliability1 ECE-355 Tutorial Jie Lian.
Chapter 22. Software Reliability Engineering (SRE)
SOFTWARE RELIABILITY MODELING
Software faults & reliability Presented by: Presented by: Pooja Jain Pooja Jain.
Pop Quiz How does fix response time and fix quality impact Customer Satisfaction? What is a Risk Exposure calculation? What’s a Scatter Diagram and why.
Software Reliability Model Deterministic Models Probabilistic Models Halstead’s software metric McCabe’s cyclomatic complexity metrix Error seeding Failure.
SENG521 (Fall SENG 521 Software Reliability & Testing Software Reliability Tools (Part 8a) Department of Electrical & Computer.
1SAS 03/ GSFC/SATC- NSWC-DD System and Software Reliability Dolores R. Wallace SRS Technologies Software Assurance Technology Center
Software Reliability SEG3202 N. El Kadri.
Prof. Dr. S. K. Bhattacharjee Department of Statistics University of Rajshahi.
Education Research 250:205 Writing Chapter 3. Objectives Subjects Instrumentation Procedures Experimental Design Statistical Analysis  Displaying data.
Lecture 4: Statistics Review II Date: 9/5/02  Hypothesis tests: power  Estimation: likelihood, moment estimation, least square  Statistical properties.
Chapter 8 : Estimation.
Methods for Estimating Defects Catherine V. Stringfellow Mathematics and Computer Science Department New Mexico Highlands University October 20, 2000.
Copyright © 2006 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill/Irwin 8-1 Chapter Eight Audit Sampling: An Overview and Application.
1 Introduction to Statistics − Day 4 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Lecture 2 Brief catalogue of probability.
Understanding Basic Statistics Fourth Edition By Brase and Brase Prepared by: Lynn Smith Gloucester County College Chapter Nine Hypothesis Testing.
1 ES Chapters 14 & 16: Introduction to Statistical Inferences E n  z  
Topics Semester I Descriptive statistics Time series Semester II Sampling Statistical Inference: Estimation, Hypothesis testing Relationships, casual models.
In Bayesian theory, a test statistics can be defined by taking the ratio of the Bayes factors for the two hypotheses: The ratio measures the probability.
PEP-PMMA Training Session Statistical inference Lima, Peru Abdelkrim Araar / Jean-Yves Duclos 9-10 June 2007.
Tony Hyun Kim 9/26/ MW Probability theory 1.Independent events 2.Poisson distribution 2. Experimental setup 3. Results 1.Comparison to.
1 Borgan and Henderson: Event History Methodology Lancaster, September 2006 Session 8.1: Cohort sampling for the Cox model.
A Software Cost Model with Reliability Constraint under Two Operational Scenarios Satoru UKIMOTO and Tadashi DOHI Department of Information Engineering,
Slide (Ch.22) 1 Tian: Software Quality Engineering Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement Jeff Tian Chapter.
Modeling and Simulation CS 313
Chapter Nine Hypothesis Testing.
Audit Sampling: An Overview and Application
Inference for the Mean of a Population
HST 583 fMRI DATA ANALYSIS AND ACQUISITION
ESTIMATION.
Making inferences from collected data involve two possible tasks:
Review of Testing a Claim
Chapter 3 INTERVAL ESTIMATES
Point and interval estimations of parameters of the normally up-diffused sign. Concept of statistical evaluation.
Modeling and Simulation CS 313
Chapter 9: Inferences Involving One Population
7-1 Introduction The field of statistical inference consists of those methods used to make decisions or to draw conclusions about a population. These.
Flooding Walkdown Guidance
Software Reliability Models.
Software Test Termination
Chapter 6 Hypothesis tests.
Software Reliability Engineering
This teaching material has been made freely available by the KEMRI-Wellcome Trust (Kilifi, Kenya). You can freely download,
Interval Estimation.
Econometric Models The most basic econometric model consists of a relationship between two variables which is disturbed by a random error. We need to use.
Discrete Event Simulation - 4
Confidence Intervals Chapter 10 Section 1.
Counting Statistics and Error Prediction
Chapter 10: Estimating with Confidence
Introduction to Estimation
EQT 272 PROBABILITY AND STATISTICS ROHANA BINTI ABDUL HAMID
How Confident Are You?.
Professor Ke-Sheng Cheng
Presentation transcript:

Confidence intervals in software reliability testing Alessandro Di Bucchianico (LaQuSo, Eindhoven University of Technology) Ed Brandt and Rob Henzen (Refis, Netherlands) ENBIS-5 Newcastle September 15, 2005

Goals of this talk show how to obtain confidence intervals for software reliability predictions from NHPP models apply results to case study

Introduction of LaQuSo and Refis Overview of this talk Introduction of LaQuSo and Refis Case Dutch Ministry of Transport, Public Works and Water Management Software reliability models Confidence intervals for NHPP models: asymptotics simulation goodness-of-fit tests Conclusions

LaQuSo: Laboratory for Quality Software university based laboratory started at the Eindhoven University of Technology Radboud University (Nijmegen) has recently joined as partner statistics and probability group in math department at TU/e is one of the participating groups started in January 2004: 10 fte; will grow to 50 fte case-study driven in cooperation with industry statistics will be integrated part of testing and verification activities more information: www.laquso.com

consultancy company in Bilthoven, the Netherlands activities include: Refis consultancy company in Bilthoven, the Netherlands activities include: software reliability assessments measurements systems for IT sector test audits for more information, see www.refis.nl

2/3 of the Netherlands is below sea level Context of case 2/3 of the Netherlands is below sea level protection against sea and rivers by dunes dikes dams sluices … hardware reliability of sluices is well understood and documented control of sluices by huge software systems (reliability??)

Sluice (1)

Sluice (2)

Obtain information on reliability of software system Goals of case Case is project of Dutch Ministry of Transport, Public Works and Water Management (see www.rws.nl for general information) Obtain information on reliability of software system Registration system for defect detection and repair Predict system reliability with confidence bounds

Data available from three tests: Available data Data available from three tests: plant acceptation test site acceptation test site acceptation retest Defect counts grouped data severity index repair status … Data was collected manually and checked on consistency etc.

Data assumptions Assumptions are results from intensive discussions with project and test engineers all test intervals have same effort every test period corresponds to 219 days of actual use immediate correction of errors (gaps between testing periods allowed for this) no new error introduced by correction actions

Data (severity 1 FAT)

Software reliability models Main differences with hardware reliability: no wear no burn-in exact reproducibility of errors Hundreds of reliability growth models available Dedicated software for software reliability exists (not always reliable, though): Casre Smerfs …

Initial Model Selection Models available in standard software reliability packages (Smerfs, Casre) were judged on several criteria (assumptions or properties), including: upper bound on number of errors interval data length of test intervals distribution of errors shape of failure intensity … The list of selected models included two NHPP models (Goel-Okumoto and Yamada S-shaped)

Nonhomogeneous Poisson process T1 T2 T3 T4 t N(t )=4 This is a Type II model (cf. Langberg/Singpurwalla (1985)) that in general cannot be described easily in terms of time between failures. Special case: Poisson process

NHPP models Several choices for  have been introduced: Goel-Okumoto, Musa delayed S-shaped inflection S-shaped hyperexponential logarithmic

NHPP models: inference for grouped data data consists of counts in time intervals: ni = # detected failures in time interval (ti-1,ti] likelihood function (t0=0): (t) = cumulative hazard rate at time t = expected number of failures at time t if  has parametric form, then maximizing L yields ML estimates for parameters (t) = d/dt (t) = hazard rate at time t

NHPP models with 2 parameters: inference for parameters Assume  depends on 2 parameters a and e ML-estimators have no closed form asymptotic distribution through Fisher information:

NHPP models with 2 parameters: inference for function of parameters assume  depends on two parameters a and b asymptotic distribution of functions of a and b through Fisher information and delta method: examples of functions of parameters include: probability of no failure in certain time period failure intensity at t=t0

Simulation NHPP process T1 T2 T3 T4 t N(t )=4 Conditional on the event N(t)=n, the T1,…,Tn are distributed as the order statistics of a sample of size n from a distribution with density (t) / (t). Hence, simulating a sample from a distribution with density (t) / (t) can be used to simulate an NHPP process with intensity (t)

Goodness-of-fit NHPP process T1 T2 T3 T4 t N(t )=4 Conditional on the event N(t)=n, the T1,…,Tn are distributed as the order statistics of a sample of size n from a distribution with density (t) / (t). Hence, the Kolmogorov goodness-of-fit test based on the empirical distribution function may be used to perform a GOF test.

goodness-of-fit: OK at 5% level Back to case study parameter estimates and 95% confidence intervals for Goel-Okumoto model (a(1-exp(b t)): a : ( 13.2 , 19.5989 ) b = ( 0.000818358 , 0.00318164 ) goodness-of-fit: OK at 5% level important question from Dutch politics: 95% confidence interval for probability of no failure in 1 year: ( 0.799462 , 1 ) (thus confirmation of suspicion by Ministry officials that defect system is not good enough for required probabilities)

Conclusions asymptotic confidence intervals for functions of parameters in NHPP models may obtained from Fisher information testing registration of Dutch water works not sufficient to obtain high-precision estimates of software reliability

Literature Rijkswaterstaat report (confidential) Systematic description of software reliability models, manuscript in progress (ADiB + Refis) Xie and Hong (2001), Handbook of statistics 20 (Advances in Reliability), 707-731.