Software Test Termination 1209

Testing is expensive

When do we terminate testing?

Finding all the bugs

Fault Seeding Seed (add) errors to code Use standard techniques to identify errors Estimate total errors based on fraction of seeded errors found

Fault Seeding ST: total seeded faults SD: Seeded faults detected NT: Total faults that were not seeded ND: Detected faults that were not seeded SD/ST = Fraction of seeded faults found ND/NT = Fraction of non-seeded faults found SD/ST = ND/NT

Example: We seed 20 faults in a system. The testing group finds 38 faults, 8 of which were seeded. How many unseeded faults remain?

Problems with this approach

Software is good enough

Measures of Software Quality: Usability Ship when product is usable enough based on customer usability studies

Measures of Software Quality: Reliability, Availability, Maintainability

Measures MTTF: Mean time to failure MTTR: Mean time to repair MTBF: Mean time between failure MTBF = MTTF + MTTR Failure Failure System repaired Time TTF TTR TBF

Software Reliability Models Statistical models of reliability Track the first N failures Look for a probability distribution that matches this data Example models include Basic execution time Logarithmic Poisson Execution Time

Failures, not faults Time Execution time (more accurate) Calendar time (more useful)

Non-static failure behavior Faults are repaired when the system fails The probability of failure changes over time

Developing models Track failures and execution time Time to failure Time between failure Fit to statistical distribution Estimate distribution parameters Estimate reliability for some future time

Need to Ship: Factors Market-driven need (need to be first to market) Business need Need to market product to generate revenue Lack of resources (money) to continue testing Cost/benefit analysis is positive Political factors

Retire the System “retirement” here may mean “terminate maintenance”