1. When to Test Less Authors: Tim Menzies and Bojan Cukic Bojan Cukic Presenter: Genet Balaker

2. Agenda Introduction The Theory of Testing The Realty of Testing The Average Shape of Software Implementation ImplementationConclusion

3. Introduction Testing software in the large is different from testing it in the small Small projects often lack funds to set up software test environments and have to depend on manual and simple automatic testing schemes.

4. Introduction (cont.) a program's shape can help determine whether rapid testing with limited resources will be as effective as elaborate and expensive testing regimes. for many small-scale projects, a small number of randomly selected tests will adequately probe the software

5. Theory of Testing Black-box testing to detect faults in a software How many random black-box tests do we need to find a fault? A randomly chosen input has odds x that it stumble across some fault, This input will miss that fault with odds of 1-x If we conduct N random black-box tests, the odds of a failure not occurring is (1-x) N, so, the probability y of finding a fault in N random tests is so, the probability y of finding a fault in N random tests is y = 1- (1-x) N y = 1- (1-x) N

6. Theory of Testing (cont.) Black-box testing can be expensive. Formal-methods testing Benefit of formally checking a system is that such formal proofs can find faults more systematically than standard testing. A single formal first-order query is equivalent to many black-box test inputs. The cost of such rigorous formal analysis ca be impractically high

7. Theory of Testing (cont.) We can only use formal methods to test small, critical portions of the system. Even when applying formal verification, its recommended to use black-box testing. Another approach to testing small-scale software is white-box testing. Test inputs exercise K different partitions. Test inputs exercise K different partitions. Each partition exercises one interesting feature of a program Each partition exercises one interesting feature of a program

8. Theory of Testing (cont.) The odds of detecting a fault with white- box methods are nearly the same as black-box methods. White-box method using k partitions is only k times better at finding errors than black-box method testing, but it slightly reduces the number of tests required Building the partitions is time consuming.

9. The Realty of Testing Given the mathematics of black-box testing, software-in-the-small project doesn’t crash more often. Program shape = the shape of the pathways within the program Numerous and tangled like spaghetti Numerous and tangled like spaghetti Few and not complex Few and not complex What should the shape of the software be to adequately test it using limited resources?

10. The realty of Testing (cont.) Studies suggest that simple shapes are common Du-pathways A link from where a variable is defined to where it is used A link from where a variable is defined to where it is used As the du-pathways shrink, the number of inputs required to reach part of the program also shrinks As the du-pathways shrink, the number of inputs required to reach part of the program also shrinks Control flow diagrams They link program statements: while, repeat, and for statements They link program statements: while, repeat, and for statements They grow as the program grow They grow as the program grow A worst-case control-flow A worst-case control-flow Saturation effect Consistent with programs containing either many portions with simple shapes or many portions that are so twisted in shape Consistent with programs containing either many portions with simple shapes or many portions that are so twisted in shape

11. Shape of a software If it is possible to show a software has the right shape, it can be tested quickly. What’s the right shape of a software? What’s an average shape of a software?

12. Shape of a software (cont.) Assumptions about the tested program’s structure to find out the shape Programs are networks connecting system concepts Programs are networks connecting system concepts A fault explanation tree is a tree whose leaves are inputs and whose root is fault A fault explanation tree is a tree whose leaves are inputs and whose root is fault Testing is a process of trying to generate a fault explanation tree from the program network Testing is a process of trying to generate a fault explanation tree from the program network An explanation tree has edges and nodes An explanation tree has edges and nodes Testing is the process of extracting NAYO trees (no, and, yes and or) Testing is the process of extracting NAYO trees (no, and, yes and or)

13. Implementation Based on the above assumptions, it’s possible to simulate the construction of an explanation tree across a program It is possible to characterize the tree’s shape by the number of tests N required to reach it’s root to find a fault Complex shaped trees require large N

14. Implementation (cont.) simple if 0 < N < 10 2 simple if 0 < N < 10 2 Shape = moderate if 10 2 <= N <10 4 hard 10 4 <= N <10 6 hard 10 4 <= N <10 6 overly complex if N >= 10 6 overly complex if N >= 10 6

15. Implementation Given in inputs to a NAYO network with V nodes, the odds of hitting a fault straight away from the inputs is X o = in/v (A) The probability of reaching an and node with andp parents is the probability of reaching all its parents: of reaching all its parents: X and = xi andp (B) where xi is the probability we computed in the prior step of the simulation (and the base case of x i = 0 is computed from Equation A). The probability of reaching an or node with orp parents is the probability of not missing any of its parents; that is, of not missing any of its parents; that is, X or = 1 − (1 − x) orp (C)

16. Implementation (cont.) If the ratio of and nodes in a NAYO network is andf, then the ratio of or nodes in the same network is or f = 1 – andf. The odds of reaching some random node x j is the weighted sum of the probabilities of reaching and nodes or or nodes x j = andf * xand * orf * xor (D) The number of tests required to be 99% sure of finding a fault with probability x j : y = 0.99 = 1 − ((1 − x j ) N ) (E) Therefore: N = log(1- 0.99) / log(1-x j )

17. Implementation (cont.) Results from the Simulation Runs ____________________________________ Classification Threshold Percent Simple 0 < N < 10 2 36 Moderate 10 2 <= N < 10 4 19 Hard 10 4 <= N < 10 6 25 Overly complex N > =10 6 20

18. Implementation (cont.) Over half of the time, 100 random tests will yield as much information as much more prolonged series of tests 10,000 randomly selected tests will probe the program as much as very prolonged series of tests will.

19. conclusion Software-in-the-small projects should routinely execute overnight test runs in which tens of thousands of test cases are generated at random software-in-the-small projects, a cost- cutting heuristic is to avoid elaborate and expensive testing regimes.