When to Test Less Presented by Lan Guo

Introduction (1) Methods of software testing: functional, coverage, and user-oriented Phases of software testing: unit, integration, product, or regression White-box (coverage) testing uses the structure of the software to measure the quality of testing.

Introduction (2) –Statement coverage –Decision coverage –Data flow coverage –Mutation testing Black-box testing aims at the specific features of the software. –Generate an operational profile, which is a list of occurrence probabilities of each element in the input domain of the program

Introduction (3) –The operational profile is dependent on the input from the customer. Difficulties in black-box testing –new software, new feature, no customer base –inadequate test set –coarse features, interacting features –multiple and unknown user groups

Introduction (4) There is a saturation effect associated with ALL testing methods. Testing effort Reliability functional decision Data flow mutation

Introduction (5) Elaborate and expensive testing does not tend to find more faults than inexpensive manual or simple automatic testing schemes.

Theory of Testing (1) In black-box testing, the probability to find a fault in N random tests is y = 1 - (1 - x) N x: the probability of finding a fault for a random input Black-box testing can be expensive. If x = , 46,000 tests are needed to be 99% sure to reveal a fault.

Theory of Testing (2) Formal-methods testing is even more expensive compared to black-box testing. While-box testing only slightly reduces the number of tests required by black-box testing. In addition, the partition cost is very high.

Reality of Testing (1) Software-in-the-small projects produce software without rigorous testing. The software does not crash more often. A program’s shape is the shape of the pathways within the program. The more tangled the pathways, the more complex the program shape.

Reality of Testing (2) The simple shapes are common, which is demonstrated by du-pathways, control-flow diagrams and saturation effect. Saturation effect is consistent with programs containing either many portions with simple shapes--so the portions are easily reached-- or many portions that are so twisted in shape that we will never reach them.

Average Shape of Software Right shape: the program is either too complex to test or too simple to test elaborately. In either case, there is no point to conduct a lengthy and expensive test. Most programs are indeed the right shape for simple testing. Proof: A mathematical model of building a pathway across a program from randomly selected inputs to some randomly selected state.

Assumption 1 Programs are networks connecting system concepts. (NAYO) Optimizing compiler builds a network (control and data-flow graphs) from the code. –An execution trace of a procedural program shows what statements are executed, which subroutines are care called, and in what order. –A verification condition generator creates a proof tree (execution tree), including preconditions and post- conditions

Assumption 2 A fault explanation tree is a tree whose leaves are inputs and whose root is a fault. If we arrive at a state which should not happen in this tree, it is a bug in our system.

Assumption 3 Testing is a process of trying to generate a fault explanation tree from the program network. If we can’t generate such tree, we are confident that there is no fault in our program.

Assumption 4 NAYO tree (explanation tree) N: no edge A: and node Y: yes edge O: or node No edge only connects or nodes Yes edge connects both and nodes and or nodes

Assumption 5 Testing is the process of extracting NAYO trees (the explanations) from NAYO networks (the program).

Mathematics Equations (1) Given in inputs to a NAYO network with V nodes, the probability to find a fault is: x o = in/V (A) The probability of reaching an and node with andp parents is the probability of reaching all its parents: x and = x i andp (B)

Mathematics Equations (2) The probability of reaching an or node with orp parents is the probability of not missing all of its parents: x or = 1 - (1 - x o ) orp (C) If the ratio of and nodes in a NAYO tree is andf, then the ratio of or nodes in the same tree is orf = 1 - andf. The probability to reach some random node x j is: x j = andf * x and + orf * x or (D)

Mathematics Equations (3) Rearrange Equation A to isolate the number of tests required to be 99% sure of finding a fault with probability x j is: y = 0.99 = 1- ((1 - x j ) N ) Therefore, N = log( ) log(1-x j )

Implementation We can approximately categorize the tree’s shape by the number of tests N required to reach that tree’s root to find a fault. Results from the Simulation Runs ClassificationThresholdPercent Simple Moderate Hard Overly complex 0 < N < <=N< <=N<10 6 N >=

Conclusion Black-box mathematics is blind to the internal structures of the program. If we include internal structure in the analysis, the result is more optimistic. The average case analysis of the shape of programs indicates that simple and RANDOM testing can be very valuable.

Related Research Simple and random testing can probe non- determinate systems. Limited testing is adequate for knowledge based systems.

Potential Future Research Extending static code analysis tools (optimizer and verification condition generator) to extract the parameters in the mathematical model from real-world programs, in order to monitor the change of program’s shape. Offering design guideline to avoid hard- shaped programs.

References (1) T. Menzies and B. Cukic, “When to Test Less”, IEEE Software, September/October 2000, pp D. Hamlet and R. Taylor, “Partition Testing Does not Inspire Confidence”, IEEE Trans. Software Eng., Vol. 16, No. 12, Dec. 1990, pp J. Horgan and A. Mathur, “Software Testing and Reliability”, The Handbook of Software Reliability Eng., M.R. Lyu, ed., McGraw-Hill, New York, 1996, pp

References (2) T. Menzies and B. Cukic, “Adequacy of limited testing for knowledge based systems”, International Journal on Artificial Intelligence Tools, Vol. 9, No. 1, 2000, pp T. Menzies, B. Cukic, H. Singh, and J. Powell, “Testing Nondeterminate Systems”