1 740f02frankl25 Evaluating Testing Methods by Delivered Reliability Frankl, Hamlet, Littlewood, Strigini IEEE TOSE Aug98

2 740f02frankl25 Partition Testing u What is a partition? u How do you choose the partitions? u What is a good partition?

3 740f02frankl25 Fault Detection Probability u Probability of a testing methodology finding a fault (if it existed)

4 740f02frankl25 Operational Profile u What is an operational profile? u How do you construct an operational profile? u What are some difficulties with operational profiles?

5 740f02frankl25 Tests, Specifications, meets u Test or test case single value of program input functional program - one input produces an output u Specification - S set of input-output pairs u Program meets specification iff for all x in spec, actual output matches spec output

6 740f02frankl25 Failures, failure points u “A program P with specification S fails on input x iff P does not meet S at x.” u “the event is called a failure” u “the input responsible is a failure point” u “The program’s failure set is the collection of all failure points”

7 740f02frankl25 Faults u “the part of a source program that causes a failure” (IEEE glossary) u Not well defined u usually multiple ways to correct a failure

8 740f02frankl25 Fixing failures u “assume that all testers, upon observing a test failure, choose fixes that eliminate exactly the same failure region” u Why is this assumption important?

9 740f02frankl25 Operational Testing u Q - probability distribution over input domain Q:D -> [0,1] and  Q(t) = 1  labels  -phi(failure) and  -sigma(success)   (t) = 1 if  and 0 if    - failure probability for a randomly drawn point is  Q 

10 740f02frankl25 Cost of testing u Fixed budget T d debug tests or T r operational tests u T d log(1-d) < T k log(1-q) u for small q and d dT d > qT r u Why does he assume that T d > T r ?

11 740f02frankl single failure region, with sub u Debug with subdomains E(  ) = q  (1-d i ) T u Operational E(  ) = q(1-q) T

12 740f02frankl25 Triangle Code cin >> a >> b >> c; type = “ scalene ” if (a == b || a == c || b == c) type = “isosceles”; if (a == b && b == c) type = “equilateral”; if (a>=b+c||b>=a+c||c>=a+b) type = “not a triangle”; if(a<=0||b<=0||c<=0) type = “bad inputs”; cout<< type

13 740f02frankl25 Triangle subdomains u subdomains u (1-d i1 ) T i 1 …(1-d in ) T i n < (1-q) T

14 740f02frankl25 Class discussion u What would be better subdomains

15 740f02frankl25 Multiple Failure,debug w/o sub  E(  ) =  q i (1-d i ) T  E(  ) =  q i (1-q i ) T

16 740f02frankl25 When is …. u A. debug better than operational? u B. vice-versa? u C. reliability worse but detection better?