Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Modelling Engin Deveci

Boğaziçi University Software Reliability Modelling Computer Engineering Agenda What is Software Reliability? Software Failure Mechanisms –Hardware vs Software Measuring Software Reliability Software Reliability Models Statistical Testing Conclusion

Boğaziçi University Software Reliability Modelling Computer Engineering What is Software Reliability the probability of failure-free software operation for a specified period of time in a specified environment Probability of the product working “correctly” over a given period of time. Informally denotes a product’s trustworthiness or dependability.

Boğaziçi University Software Reliability Modelling Computer Engineering What is Software Reliability Software Reliability is an important attribute of software quality, together with –functionality, –usability, –performance, –serviceability, –capability, –installability, –maintainability, –documentation.

Boğaziçi University Software Reliability Modelling Computer Engineering What is Software Reliability Software Reliability Modeling Prediction Analysis Reliability Measurement Defect Classification Trend Analysis Field Data Analysis Software Metrics Software Testing and Reliability Fault-Tolerance Fault Trees Software Reliability Simulation Software Reliability Tools From Handbook of Software Reliability Engineering, Edited by Michael R. Lyu, 1996

Boğaziçi University Software Reliability Modelling Computer Engineering What is Software Reliability Software Reliability is hard to achieve, because the complexity of software tends to be high. While the complexity of software is inversely related to software reliability, it is directly related to other important factors in software quality, especially functionality, capability.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Failure Mechanisms Failure cause: Software defects are mainly design defects. Wear-out: Software does not have energy related wear- out phase. Errors can occur without warning. Repairable system concept: Periodic restarts can help fix software problems. Time dependency and life cycle: Software reliability is not a function of operational time. Environmental factors: Do not affect Software reliability, except it might affect program inputs. Reliability prediction: Software reliability can not be predicted from any physical basis, since it depends completely on human factors in design.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Failure Mechanisms Redundancy: Can not improve Software reliability if identical software components are used. Interfaces: Software interfaces are purely conceptual other than visual. Failure rate motivators: Usually not predictable from analyses of separate statements. Built with standard components: Well-understood and extensively-tested standard parts will help improve maintainability and reliability. But in software industry, we have not observed this trend. Code reuse has been around for some time, but to a very limited extent. Strictly speaking there are no standard parts for software, except some standardized logic structures.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Failure Mechanisms

Boğaziçi University Software Reliability Modelling Computer Engineering Software Failure Mechanisms

Boğaziçi University Software Reliability Modelling Computer Engineering Measuring Software Reliability Don’t define what you won’t collect.. Don’t collect what you won’t analyse.. Don’t analyse what you won’t use..

Boğaziçi University Software Reliability Modelling Computer Engineering Measuring Software Reliability Measuring software reliability remains a difficult problem because we don't have a good understanding of the nature of software Even the most obvious product metrics such as software size have not uniform definition.

Boğaziçi University Software Reliability Modelling Computer Engineering Measuring Software Reliability Current practices of software reliability measurement can be divided into four categories: –Product metrics –Project management metrics –Process metrics –Fault and failure metrics

Boğaziçi University Software Reliability Modelling Computer Engineering Measuring Software Reliability Different categories of software products have different reliability requirements: –level of reliability required for a software product should be specified in the SRS document. A good reliability measure should be observer independent, –so that different people can agree on the reliability.

Boğaziçi University Software Reliability Modelling Computer Engineering Measuring Software Reliability LOC, KLOC, SLOC, KSLOC McCabe's Complexity Metric Test coverage metrics ISO-9000 Quality Management Standards MTBF –Once a failure occurs, the next failure is expected after 100 hours of clock time (not running time).

Boğaziçi University Software Reliability Modelling Computer Engineering Measuring Software Reliability Failure Classes –Transient: Transient failures occur only for certain inputs. –Permanent: Permanent failures occur for all input values. –Recoverable: When recoverable failures occur the system recovers with or without operator intervention. –Unrecoverable: the system may have to be restarted. –Cosmetic: May cause minor irritations. Do not lead to incorrect results. –Eg. mouse button has to be clicked twice instead of once to invoke a GUI function.

Boğaziçi University Software Reliability Modelling Computer Engineering Measuring Software Reliability Errors do not cause failures at the same frequency and severity. –measuring latent errors alone not enough The failure rate is observer-dependent No simple relationship observed between system reliability and the number of latent software defects. Removing errors from parts of software which are rarely used makes little difference to the perceived reliability. removing 60% defects from least used parts would lead to only about 3% improvement to product reliability. Reliability improvements from correction of a single error depends on whether the error belongs to the core or the non-core part of the program. The perceived reliability depends to a large extent upon how the product is used. In technical terms on its operation profile.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Software reliability models have emerged as people try to understand the characteristics of how and why software fails, and try to quantify software reliability Over 200 models have been developed since the early 1970s, but how to quantify software reliability still remains largely unsolved There is no single model that can be used in all situations. No model is complete or even representative.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Most software models contain the following parts: –assumptions, –factors, –a mathematical function relates the reliability with the factors. is usually higher order exponential or logarithmic.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Software modeling techniques can be divided into two subcategories: –prediction modeling –estimation modeling. Both kinds of modeling techniques are based on observing and accumulating failure data and analyzing with statistical inference.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models ISSUESPREDICTION MODELSESTIMATION MODELS DATA REFERENCEUses historical dataUses data from the current software development effort WHEN USED IN DEVELOPMENT CYCLE Usually made prior to development or test phases; can be used as early as concept phase Usually made later in life cycle(after some data have been collected); not typically used in concept or development phases TIME FRAMEPredict reliability at some future time Estimate reliability at either present or some future time

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models There are two main types of uncertainty which render any reliability measurement inaccurate: Type 1 uncertainty: –our lack of knowledge about how the system will be used, i.e. its operational profile Type 2 uncertainty: –reflects our lack of knowledge about the effect of fault removal. When we fix a fault we are not sure if the corrections are complete and successful and no other faults are introduced Even if the faults are fixed properly we do not know how much will be the improvement to interfailure time.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Step Function Model –The simplest reliability growth model: a step function model –The basic assumption: reliability increases by a constant amount each time an error is detected and repaired. –Assumes: all errors contribute equally to reliability growth highly unrealistic: –we already know that different errors contribute differently to reliability growth.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Jelinski and Moranda Model –Realizes each time an error is repaired reliability does not increase by a constant amount. –Reliability improvement due to fixing of an error is assumed to be proportional to the number of errors present in the system at that time.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Littlewood and Verall’s Model –Assumes different fault have different sizes, thereby contributing unequally to failures. –Allows for negative reliability growth –Large sized faults tends to be detected and fixed earlier –As number of errors is driven down with the progress in test, so is the average error size, causing a law of diminishing return in debugging

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Variations exists –LNHPP (Littlewood non homogeneous Poisson process) model Goel – Okumoto (G-O) Imperfect debugging model –GONHPP Musa – Okumoto (M-O) Logarithmic Poisson Execution Time model

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Applicability of models: –There is no universally applicable reliability growth model. –Reliability growth is not independent of application. –Fit observed data to several growth models. Take the one that best fits the data.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Observed failure intensity can be computed in a straightforward manner from the tables of failure time or grouped data (e.g. Musa et al. 1987).

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Example: (136 failures total): Failure Times (CPU seconds): 3, 33, 146, 227, 342, 351, 353,444, 556, 571, 709, 759, , Data are grouped into sets of 5 and the observed intensity, cumulative failure distribution and mean failure times are computed, tabulated and plotted.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Two common models are the "basic execution time model“ and the "logarithmic Poisson execution time model" (e.g. Musa et al. 1987).

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Basic Execution Time Model –Failure intensity λ(τ) with debugging time τ: –where λ0 is the initial intensity and ν0 is the total expected number of failures (faults).

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models where μ(τ) is the mean number of failures experienced by time τ.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models In this case the Logarithmic- Poisson Model fits somewhat better than the Basic Execution Time Model. In some other projects BE model fits better than LP model.

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models Additional expected number of failures, Δμ, that must be experienced to reach a failure intensity objective where λP is the present failure intensity, and λF is the failure intensity objective. The additional execution time, Δτ, required to reach the failure intensity objective is

Boğaziçi University Software Reliability Modelling Computer Engineering Software Reliability Models After fitting a model describing the failure process we can estimate its parameters, and the quantities such as the total number of faults in the code, future failure intensity and additional time required to achieve a failure intensity objective.

Boğaziçi University Software Reliability Modelling Computer Engineering Statistical Testing The objective is to determine reliability rather than discover errors. Uses data different from defect testing.

Boğaziçi University Software Reliability Modelling Computer Engineering Statistical Testing Different users have different operational profile: –i.e. they use the system in different ways –formally, operational profile: probability distribution of input Divide the input data into a number of input classes: –e.g. create, edit, print, file operations, etc. Assign a probability value to each input class: –a probability for an input value from that class to be selected.

Boğaziçi University Software Reliability Modelling Computer Engineering Statistical Testing Determine the operational profile of the software: –This can be determined by analyzing the usage pattern. Manually select or automatically generate a set of test data: –corresponding to the operational profile. Apply test cases to the program: –record execution time between each failure –it may not be appropriate to use raw execution time After a statistically significant number of failures have been observed: –reliability can be computed.

Boğaziçi University Software Reliability Modelling Computer Engineering Statistical Testing Relies on using large test data set. Assumes that only a small percentage of test inputs: –likely to cause system failure. It is straight forward to generate tests corresponding to the most common inputs: –but a statistically significant percentage of unlikely inputs should also be included. Creating these may be difficult: –especially if test generators are used.

Boğaziçi University Software Reliability Modelling Computer Engineering Conclusions Software reliability is a key part in software quality Software reliability improvement is hard There are no generic models. Measurement is very important for finding the correct model. Statistical testing should be used but it is not easy again… Software Reliability Modelling is not as simple as described here.

Boğaziçi University Software Reliability Modelling Computer Engineering Thank you. Q&A