Evaluating Testing Methods by Delivered Reliability Frankl, Hamlet, Littlewood, Strigini IEEE TOSE Aug98

From last Thursday How could you estimate detection probability for C0?

About Frankl’s article What parts are unclear? Any questions?

3.3 single failure region, with sub Debug with n subdomains E(Q) = qP(1-di)T/n Operational E(Q) = q(1-q)T

Assumptions: multiple failure model

Multiple Failure,debug w/o sub E(Q) = Sqi(1-di)T E(Q) = Sqi(1-qi)T

Example 2 – assume 100 tests in domain Fault 1 Fault 2 Fault 3 Fault 4 Fault 5 Tests in failure 1,2 3,4,5,6 7,8,9 10-19 20 What is E(Q) for debugging testing? What is E(Q) for operational testing if tests 1-20 are twice as likely as tests 21-100?

Empirical Tool Abstract domain to integers 1-100 Abstract failure sets into number ranges Run N (upto 5K) test tests Each test set may have subdomains or n random tests Determine frequency of each detect/notdetect combination Input q of each detect/notdetect combination Calculate E(Q)

Example Code for(i=1;i<MAX;i++){ answer = 0; int j; for (j=1;j<6;j++){ test = 1+ rand()%80; if (test <= 10) answer = answer + 1; if (test > 20 && test <= 25 ) { answer = answer + 2; error[2] = 1; } if ( test > 30 && test <=35 && error[2] == 0) answer = answer + 2; if (test > 35 && test <= 36){answer = answer + 4; error[4] = 1; } if (test > 36 && test <= 37) answer = answer + 8; if (test> 30 && test <= 32 && error[4] == 0) answer = answer + 4; if (test > 60 && test <= 62 && error[4] == 0) answer = answer + 4; if (test > 62 && test <= 63) answer = answer + 16; found[answer]++;

For Tuesday, Sep 12 Study Frankl through section 3.5 Finish empirical tool and try subdomain testing for the example in slides

1 min paper What is the muddiest point about Frankl’s paper?