Statistical Testing Jonas Abromaitis IFM-0/2

Introduction NIST* developed techniques for applying statistical methods to derive best sample test cases and provide a statistical level of confidence or probability that a program implements its functional specification correctly NIST* - National Institute of Standards and Technology

Introduction The concept of statistical testing was invented by Harlan Mills (IBM) The central idea is to use software testing as a means to assess the reliability of software as opposed to a debugging mechanism

Black-Box Conformance Testing Consists of developing a suite of tests to determine the correctness of an implementation that is built to the requirements of a functional specification.

Black-Box Conformance Testing Control-Flow Testing category - the objective is to model the functional specification as a directed graph and find input values for a minimal number of test cases to ensure that every achievable path is traversed by at least one of the test cases.

Markov chain model It allows test input sequences to be generated from multiple probability distributions, making it more general than many existing techniques

Markov chain model Analytical results associated with Markov chains facilitate informative analysis of the sequences before they are generated, indicating how the test is likely to unfold.

Usability testing Forecast Analyze Compare Change UI …...

Usage model Characterizes the population of intended uses of the software in the intended environment Statistical testing based on a software usage model ensures that the failures that will occur most frequently in operational use will be found early in the testing cycle.

Usage model Based on the software specification Can be developed in parallel with the software

Usage model Improves the specification, gives an analytical description of the specification, quantifies the testing costs and, with statistical testing, provides a basis from which inferences of software reliability may be made.

Usage model 1. Review and clarify the software specification 2. Identify expected users of the software, expected uses of the software, and expected system environment 3. Define a stratification of user and environment parameters

Usage model 4. Determine the desired levels of usage model granularity 5. Iteratively develop usage model structure 6. Verify the correctness of the structure against the specification

Usage model 7. Iteratively develop a probability distribution for the model 8. Verify the correctness of the probability distribution against any information available concerning intended usage of the software

Metrics (reliability) Probability of failure on demand Time between failures Failures count over some period of time

Metrics (usability) State life duration Probability of reaching state 'X' Count of reaching state 'X' Reaching some state only once or n times Total state switches during program run- time

How do we get probabilities? Specified! No?... Other similar systems! There is none?... Based on field data (there might be no info) Assumptions about expected use (I guess that my guess is correct!) Worst case... Just split probabilities equally

Binomial Trials The binomial model describes the number of successes in n independent trials, where each trial has probability of success p.

So what is it all about? Statistical testing Find small number of test cases to do the grand job Find how correct it is, rather than find bugs Know who, how and where will use the soft Don't wait for too late (early testing) You know the metrics – you know the project development state

Questions ?

Thank you 4 your attention! Questions for audience: How does Markov Chain graph looks like? How to find states reach probabilities? How is statistical testing associated with Black Box testing? Why would you want to know the reliability of the soft?