Redefining Boundaries of Boundary Testing James Bach Satisfice, Inc. This material benefited from review and conversation at the 4 th Workshop on Heuristic and Exploratory Techniques. The following people contributed to that conference: Jon Bach, James Bach, Robert Sabourin, Karen Johnson, Cem Kaner, Henrik Andersson, Keith Stobie, Scott Barber, David Gilbert, Doug Hoffman, Mike Kelly, Harry Robinson, Ross Collard, Dawn Haynes, Timothy Coulter, and Michael Bolton

4.3.2 Boundary value analysis (K3) “Behavior at the edge of each equivalence partition is more likely to be incorrect, so boundaries are an area where testing is likely to yield defects. The maximum and minimum values of a partition are its boundary values. When designing test cases, a value on each boundary is chosen.” ISTQB Foundation Syllabus

Scenario: Dear Tester, I just added a document description field to the metadata (“File | Properties”). It should take up to bytes of text, and truncate anything more than that. Thanks!

32K?  Ambiguity in units and limits

(64K) 32K  Ambiguity in units and limits  Foreign boundaries (upstream)  Unknown boundaries ? 64K 32K  Ambiguity in units and limits

32K  Ambiguity in units and limits  Foreign boundaries (upstream)  Dynamic boundaries  Unknown boundaries ? (64K+  ) 64K+  32K

(64K+  ) 32K(16K) 32K  Ambiguity in units and limits  Foreign boundaries (upstream/downstream)  Dynamic boundaries  Unknown boundaries 16K ? 64K+ 

32K (64K+  ) (16K) K16K  Ambiguity in units and limits  Foreign boundaries (upstream/downstream)  Dynamic boundaries  Quasi-functional boundaries  Unknown boundaries  Undeclared boundaries 255 ? 64K+ 

32K (64K+  ) (16K) K16K 255  Ambiguity in units and limits  Foreign boundaries (upstream/downstream)  Dynamic boundaries  Quasi-functional boundaries  Unknown boundaries  Undeclared boundaries  Mis-declared boundaries (2K) ? 64K+  2K

The Boundary Risk Hypothesis: All other things being equal, any given thing is more likely to be misclassified, mishandled, or misdirected when “near” a suspected boundary than when “far away.” This hypothesis arises because we are aware of specific mechanisms of boundary-related failure, over and above other mechanisms. But the actual boundaries in a product may not be the ones we are told about. That’s why we must explore.

Toward a Better Definition of Boundary Testing  Boundary testing: Any testing to the extent that it involves the discovery or evaluation of boundaries or boundary-related behavior.  Boundary: 1. A dividing point between two otherwise contiguous regions of behavior; or 2. A principle or mechanism by which things are classified into different sets.

To Test Boundaries Well Focus and De-focus  De-focusing strategy for boundary testing  Don’t “believe” what you read about specific boundaries, nor what you are told.  Study product behavior while driving inputs/outputs toward infinity and zero. (Stopping heuristics: flat line, sufficient drama, or cost exceeds value.)  Look for subtle clues: performance degradation; obscure data corruption; garbage.  You never know that a boundary test is finished. The failure may not become manifest right away, and not necessarily in the function you are testing at that moment. Explore around.  Focusing strategy for boundary testing  Study product claims about boundaries.  Interview programmers about boundaries.  Don’t just test on the boundary, test in “epsilons” off the boundary.  Envision data moving through a multi-path maze of “boundary gates” in the product. Are those gates consistent with each other, and with the purposes of the product?  Push boundaries while considering all known routes data may take in their lifecycle.  Pay special attention to interactions between sub-systems written by different people.  Perform bisection search to home in on specific boundaries.