1841f06detprob3 Testing Basics Detection probability

2841f06detprob3 Methodology to evaluate testing criterion u Pick sample program, sample faults, sample operational profile, two criteria. u For each criterion, randomly generate test suites that satisfy the criterion. u Calculate the percentage of the suites that detect each of the faults. u The criterion that has the higher percentage is better.

3841f06detprob3 Triangle Example cin >> a >> b >> c ; type = “scalene”; if (a == b || a == c || b == c) type= “isosceles”; if (a == b && a == c) type = “equilateral”; if (a >= b+c || b >= a+c || c >= a+b) type=“not a triangle”; if (a <= 0 || b <= 0 || c <= 0) type=“bad input”; cout<< type;

4841f06detprob3 Control Flow Graph Operational profile 3,3,3abcdegiequi 3,3,4abcegiisos 3,3,5abcegiisos 3,3,6abcefginot 3,4,3abcegiisos 3,4,4abcegiisos 3,4,5acegiscal 3,4,6acegiscal All inputs are equally likely

5841f06detprob3 What are the failure probability for each color (separately)? cin >> a >> b >> c ; type = “scalene”; if (a == b || a == c && b == c) type= “isosceles”; if (a == b || a == c) type = “equilateral”; if (a >= b+c || b >= a+c || c > a+b) type=“not a triangle”; if (a <= 0 || b <= 0 || c <= 0) type=“bad input”; cout<< type; Blue GreenRed

6841f06detprob3 TTYP – probability of detection u What is the probability of detection with one randomly chosen test case? u What is the probability of detection with two randomly chosen test cases?

8841f06detprob3 TTYP – per path u What is the probability of detection with one randomly chosen test case per path? u What is the probability of detection with an equal number of randomly chosen test cases?

9841f06detprob3 TTYP – smaller subdomains u What might be better smaller subdomains?

10841f06detprob3 TTYP - subdomains u Are paths the best subdomains? u Would a functional decomposition be better? u Should we re-define the term subdomain?

11841f06detprob3 fault 1 fault 2 fault 3 fault 4 criterion criterion criterion criterion Comparing Criteria

12841f06detprob3 Solving this choice u What are the assumptions? u Analogy with dice?

13841f06detprob3 TTYP – smaller subdomains u What might be better smaller subdomains? u Would MCC (multiple condition coverage) be better subdomains

14841f06detprob3 TTYP2 – C0 and C1 coverage u How do we deal with C0 and C1 coverage since they are not subdomain testing methodologies?

15841f06detprob3 TTYP3 u How could you estimate the det prob of C0 or C1 testing?

16841f06detprob3 Marble Problem u Assume that there is a bag of marbles from which marbles are drawn with replacement. u What is the maximum likelihood estimate of p (the probability of drawing a purple) marble if you draw exactly n purple marbles in a row? u ? P such that (p) n (1-p) is max or u ? P such that (p) n =.5

17841f06detprob3 Evaluating Testing Methods by Delivered Reliability Frankl, Hamlet, Littlewood, Strigini IEEE TOSE Aug98 For Tuesday, Sep 4– study through section 2.3