1. Statistical Testing
Jonas Abromaitis IFM-0/2

2. Introduction
NIST* developed techniques for applying statistical methods to derive best sample test cases and provide a statistical level of confidence or probability that a program implements its functional specification correctly
NIST* - National Institute of Standards and Technology

3. Introduction
The concept of statistical testing was invented by Harlan Mills (IBM)
The central idea is to use software testing as a means to assess the reliability of software as opposed to a debugging mechanism

4. Black-Box Conformance Testing
Consists of developing a suite of tests to determine the correctness of an implementation that is built to the requirements of a functional specification.

5. Black-Box Conformance Testing
Control-Flow Testing category - the objective is to model the functional specification as a directed graph and find input values for a minimal number of test cases to ensure that every achievable path is traversed by at least one of the test cases.

6. Markov chain model
It allows test input sequences to be generated from multiple probability distributions, making it more general than many existing techniques

7. Markov chain model
Analytical results associated with Markov chains facilitate informative analysis of the sequences before they are generated, indicating how the test is likely to unfold.

8. Usability testing
Forecast
Analyze
Compare
Change UI
…...

9. Usage model
Characterizes the population of intended uses of the software in the intended environment
Statistical testing based on a software usage model ensures that the failures that will occur most frequently in operational use will be found early in the testing cycle.

10. Usage model Based on the software specification
Can be developed in parallel with the software

11. Usage model
Improves the specification, gives an analytical description of the specification, quantifies the testing costs and, with statistical testing, provides a basis from which inferences of software reliability may be made.

12. Usage model 1. Review and clarify the software specification
2. Identify expected users of the software, expected uses of the software, and expected system environment
3. Define a stratification of user and environment parameters

13. Usage model 4. Determine the desired levels of usage model granularity
5. Iteratively develop usage model structure
6. Verify the correctness of the structure against the specification

14. Usage model
7. Iteratively develop a probability distribution for the model
8. Verify the correctness of the probability distribution against any information available concerning intended usage of the software

15. Metrics (reliability)
Probability of failure on demand
Time between failures
Failures count over some period of time

16. Metrics (usability) State life duration
Probability of reaching state 'X'
Count of reaching state 'X'
Reaching some state only once or n times
Total state switches during program run- time

17. How do we get probabilities?
Specified! No?...
Other similar systems! There is none?...
Based on field data (there might be no info)
Assumptions about expected use (I guess that my guess is correct!)
Worst case... Just split probabilities equally

18. Binomial Trials
The binomial model describes the number of successes in n independent trials, where each trial has probability of success p.

19. So what is it all about? Statistical testing
Find small number of test cases to do the grand job
Find how correct it is, rather than find bugs
Know who, how and where will use the soft
Don't wait for too late (early testing)
You know the metrics – you know the project development state

20. Questions ?

21. Thank you 4 your attention!
Questions for audience:
How does Markov Chain graph looks like?
How to find states reach probabilities?
How is statistical testing associated with Black Box testing?
Why would you want to know the reliability of the soft?