1. Evaluating Testing Methods by Delivered Reliability Frankl, Hamlet, Littlewood, Strigini IEEE TOSE Aug98 Section 3.4

2. 3.2 SFR, w/o subdomains  d =  tinF V(t)  P(  ) = 1-(1-d) T  P(  q) = (1-d) T  E(  ) = 0* P(  ) +q* P(  q) u = q(1-d) T 2 841f07frankl7sep13

4. 3.3 single failure region, with sub u Debug with n subdomains E(  ) = q  (1-d i ) T/n u Operational E(  ) = q(1-q) T

5. Assumptions: multiple failure model

6. Multiple Failure,debug w/o sub  E(  ) =  q i (1-d i ) T  E(  ) =  q i (1-q i ) T

7. 7 740f06frankl7 Example 1 – assume 100 tests in domain Fault 1 Fault 2 Fault 3 Fault 4 Fault 5 Fault 6 Tests In Fault 1,23,4,5, 6 7,8,910-192030-35

8. Example 2 – Debug w/o subdomains? Flt 1Flt 2Flt 3Flt 4Flt 5 SD 110/200/20 SD 20/105/100/10 SD 30/105/101/10 0/10 SD 40/20 2/200/20 SD 50/20 2/200/201/20 assume disjoint

9. Example 3 – subdomain testing Flt 1Flt 2Flt 3Flt 4Flt 5 SD 110/200/20 SD 20/105/100/10 SD 30/105/101/10 0/10 SD 40/20 2/200/20 SD 50/20 2/200/201/20

10. Example 3 – Solve empirically Flt 1Flt 2Flt 3Flt 4Flt 5 SD 110/200/20 SD 20/105/100/10 SD 30/105/101/10 0/10 SD 40/20 2/200/20 SD 50/20 2/200/201/20

11. Empirical Tool u Abstract domain to integers 1-100 u Abstract failure sets into number ranges u Run N (upto 5K) test tests Each test set may have subdomains or n random tests Determine frequency of each detect/notdetect combination Input q of each detect/notdetect combination  Calculate E(  )

12. Example Code for(i=1;i<MAX;i++){ answer = 0; int j; for (j=1;j<6;j++){ test = 1+ rand()%80; if (test <= 10) answer = answer + 1; if (test > 20 && test <= 25 ) { answer = answer + 2; error[2] = 1; } if ( test > 30 && test <=35 && error[2] == 0) answer = answer + 2; if (test > 35 && test <= 36){answer = answer + 4; error[4] = 1;} if (test > 36 && test <= 37) answer = answer + 8; if (test> 30 && test <= 32 && error[4] == 0) answer = answer + 4; if (test > 60 && test <= 62 && error[4] == 0) answer = answer + 4; if (test > 62 && test <= 63) answer = answer + 16; } found[answer]++;

13. For Tuesday, Sep 18 u Read rest of article u Build tool u Do experiments for subdomain testing with multiple faults.