1. Good Enough Testing James Bach james@satisfice.com http://www.satisfice.com The author thanks ST Labs, Inc. for supporting the work that led to some of the materials in this presentation.

2. Paradigms of Good Enough... n Not Too Terrible: “we’re still in business” n Infallibility: “anything we do is good” n Righteous Exhaustion: “perfection or bust” n Customer Oracle: “customers seem to like it” n Defined Process: “we follow a Good Process” n Static Quality: “we satisfy The Requirements” n Accountability: “we fulfill our promises” n Advocacy: “we make every reasonable effort” n Dynamic Quality: “we fulfill the mission”

3. Dynamic Quality Paradigm Perfect Awful unnecessary quality unacceptable quality Item A Item B It’s more important to work on Item B. Further improvement would not be a good use of resources. Further improvement is necessary. Good enough quality bar floating line

4. A Heuristic for Good Enough 1. X has sufficient benefits. 2. X has no critical problems. 3. Benefits of X sufficiently outweigh problems. 4. In the present situation, and all things considered, improving X would be more harmful than helpful. Benefits Problems All conditions must apply.

5. Good Enough... n …with what level of confidence? n …to meet ethical obligations? n …in what time frame? n …compared to what? n …for what purpose? n …or else what? n …for whom? Perspective is Everything

6. You can use this to: n Argue persuasively for improvement, even in the midst of chaos. n Plan ahead for high quality. n Understand what problems are keeping otherwise smart, caring people from doing excellent work. n Notice when “perfect” or “as good as humanly possible” is not good enough. n Structure a dialog about acceptable quality.

7. Why not Quantify Good Enough? Law of Medium Numbers For medium number systems, we can expect that large fluctuations, irregularities, and discrepancies with any theory will occur more or less regularly. -- Gerald Weinberg Because quality is an emergent property of a complex system, any quantification of quality is incomplete.

8. Heuristics are an alternative to quantitative models... n We can measure some things. n We can observe some things. n We can reason about those observations and measurements. A heuristic is useful method that sometimes doesn’t work. Heuristic means “serving to discover”

9. Test Project Dynamics: Star Model

12. CHOI VENS Motivation Capability Test Project Dynamics: Givens vs. Choices n Motivation: What testing does the situation require? n Capability: Can we perform that testing in this situation?

13. Analyzing a Test Process n Are GIVENS good enough? n Do CHOICES about process exploit the GIVENS and address the MISSION well enough? n Is MISSION is achieved well enough? How do you know?

14. MISSION: The most important part n Find important problems n Assess quality n Certify to standard n Fulfill process mandates n Satisfy stakeholders n Assure accountability n Advise about QA n Advise about testing n Advise about quality n Maximize efficiency n Minimize time n Minimize cost The quality of testing depends on which of these possible missions matter and how they relate. Many debates about the goodness of testing are really debates over missions and givens.