1. Software Test Termination
1209

2. Testing is expensive

3. When do we terminate testing?

4. Finding all the bugs

5. Fault Seeding Seed (add) errors to code
Use standard techniques to identify errors
Estimate total errors based on fraction of seeded errors found

6. Fault Seeding ST: total seeded faults SD: Seeded faults detected
NT: Total faults that were not seeded
ND: Detected faults that were not seeded
SD/ST = Fraction of seeded faults found
ND/NT = Fraction of non-seeded faults found
SD/ST = ND/NT

7. Example: We seed 20 faults in a system.
The testing group finds 38 faults, 8 of which were seeded.
How many unseeded faults remain?

8. Problems with this approach

9. Software is good enough

10. Measures of Software Quality: Usability
Ship when product is usable enough based on customer usability studies

11. Measures of Software Quality: Reliability, Availability, Maintainability

12. Measures MTTF: Mean time to failure MTTR: Mean time to repair
MTBF: Mean time between failure
MTBF = MTTF + MTTR
Failure
Failure
System repaired
Time
TTF
TTR
TBF

13. Software Reliability Models
Statistical models of reliability
Track the first N failures
Look for a probability distribution that matches this data
Example models include
Basic execution time
Logarithmic Poisson Execution Time

14. Failures, not faults Time Execution time (more accurate)
Calendar time (more useful)

15. Non-static failure behavior
Faults are repaired when the system fails
The probability of failure changes over time

16. Developing models Track failures and execution time
Time to failure
Time between failure
Fit to statistical distribution
Estimate distribution parameters
Estimate reliability for some future time

17. Need to Ship: Factors Market-driven need (need to be first to market)
Business need
Need to market product to generate revenue
Lack of resources (money) to continue testing
Cost/benefit analysis is positive
Political factors

18. Retire the System
“retirement” here may mean “terminate maintenance”