1. Evaluating Testing Methods by Delivered Reliability
Frankl, Hamlet, Littlewood, Strigini
IEEE TOSE Aug98

2. From last Thursday
How could you estimate detection probability for C0?

3. About Frankl’s article
What parts are unclear?
Any questions?

4. 3.3 single failure region, with sub
Debug with n subdomains
E(Q) = qP(1-di)T/n
Operational
E(Q) = q(1-q)T

5. Assumptions: multiple failure model

6. Multiple Failure,debug w/o sub
E(Q) = Sqi(1-di)T
E(Q) = Sqi(1-qi)T

7. Example 2 – assume 100 tests in domain
Fault 1
Fault 2
Fault 3
Fault 4
Fault 5
Tests in failure
1,2
3,4,5,6
7,8,9
10-19 20
What is E(Q) for debugging testing?
What is E(Q) for operational testing if tests 1-20 are twice as
likely as tests ?

9. Empirical Tool Abstract domain to integers 1-100
Abstract failure sets into number ranges
Run N (upto 5K) test tests
Each test set may have subdomains or n random tests
Determine frequency of each detect/notdetect combination
Input q of each detect/notdetect combination
Calculate E(Q)

10. Example Code for(i=1;i<MAX;i++){ answer = 0; int j;
for (j=1;j<6;j++){
test = 1+ rand()%80;
if (test <= 10) answer = answer + 1;
if (test > 20 && test <= 25 ) {
answer = answer + 2;
error[2] = 1; }
if ( test > 30 && test <=35 && error[2] == 0) answer = answer + 2;
if (test > 35 && test <= 36){answer = answer + 4;
error[4] = 1; }
if (test > 36 && test <= 37) answer = answer + 8;
if (test> 30 && test <= 32 && error[4] == 0) answer = answer + 4;
if (test > 60 && test <= 62 && error[4] == 0) answer = answer + 4;
if (test > 62 && test <= 63) answer = answer + 16;
found[answer]++;

12. For Tuesday, Sep 12 Study Frankl through section 3.5
Finish empirical tool and try subdomain testing for the example in slides

13. 1 min paper
What is the muddiest point about Frankl’s paper?