Software Testing Methodologies (STM) Unit 4 – Domain Testing Compiled with reference from: Software Testing Techniques: Boris Beizer Craft of Software Testing: Brain Marrick Narasimha Rao.P

Domain Testing Domains & Paths Nice & Ugly Domains Domain Testing We will see in this part of Unit 3: Domains & Paths Nice & Ugly Domains Domain Testing Domains & Interfaces Testing Domains & Testability ref boris beizer

Transaction-Flow Testing U3 Contents of the unit Domains Model, as a set, paths Predicates closure, dimensionality, assumptions Nice & ugly domains specified & implemented domains Nice domains – properties, testing implications Ugly domains, simplification by programmers & testers Domain testing Bugs & testing for those bugs Testing procedure Variations, tools, effectiveness Domains & Interface Testing Range, closure, span Interface range, domain compatibility Finding the values Domains & Testability Linearizing, coordinate transformations. Canonical program form ref boris beizer

Domain Testing - Domains & Paths U3 Domain Testing Model D1 D2 INPUT CLASSIFY DO CASE 1 OUTPUT D3 (x,y,z,…) { Outcome } Dn DO CASE 2 (1, 2, 3, 4, 5,…) DO CASE 3 DO CASE n ref boris beizer

Domain Testing Domain Testing Views Programs as input data classifiers Verifies if the classification is correct ref boris beizer

Domain Testing - Domains & Paths U3 Domain Testing Model Two Views Based on Specs Functional Testing Based on Implementation information Structural Technique ref boris beizer

Domain Testing - Domains & Paths U3 Domain Testing Model Variables Simple combinations of two variables Numbers from - to + Input variables as numbers Loop-free Programs ref boris beizer

Domain Testing - Domains & Paths U3 Domain Testing Model Structural Knowledge is not needed. Only Specs. For each Input Case, A Hypothetical path for Functional testing An Actual path for Structural testing ref boris beizer

Domain Testing - Domains & Paths U3 Domain (simplified) A Single Connected Set of numbers No arbitrary discrete sets Defined by the boundaries One or more boundaries Specified by predicates Bugs likely at the boundaries One or more variables D1 ref boris beizer

Domain Testing - Domains & Paths U3 Predicates Interpretation Structural Testing - CFG Functional Testing - DFG Specifies the Domain Boundary n Predicates in Sequence  2n domains Or, just two domains A .AND. B .AND. C ref boris beizer

Domain Testing - Domains & Paths U3 Paths At least One PATH thru the Program More paths for disconnected domains ref boris beizer

Domain Testing - Domains & Paths U3 Summary: Domain for a loop-free program corresponds to a set of numbers defined over the input vector For every domain, there is at least one path thru the routine, along which that domain’s processing is done The set of interpreted predicates traversed on that path (ie., the path’s predicate expression) defines the domain’s boundaries. ref boris beizer

Domain Testing - Domains & Paths - Domain Closure Boundary: Closed / Open X >= MIN X <= MAX MAX MIN D2 D3 D1 Closed D2 X >= MIN X < MAX D3 D1 Closed Open MIN MAX ref boris beizer

Domain Testing - Domains & Paths - Domain Closure Boundary: Closed / Open MIN MAX D1 D2 D3 X > MIN X < MAX Open ref boris beizer

Domain Testing - Domains & Paths - Domain Dimensionality U3 One dimension per variable At least one predicate Slices thru previously defined Domain Slicing Boundary N-spaces are cut by Hyperplanes ref boris beizer

Domain Testing - Domains & Paths – Bug Assumptions Processing is OK. Domain definition may be wrong.  Boundaries are wrong.  Predicates are wrong. Once input vector is set on the right path, it’s correctly processed. More bugs causing domain errors … ref boris beizer

Domain Testing - Domains & Paths – Bug Assumptions.. Double-zero representation Floating point zero check Contradictory domains Ambiguous domains defined undefined ref boris beizer

Domain Testing - Domains & Paths – Bug Assumptions.. Over-specified domains Boundary errors Boundary closure Shifted X > 3 .AND. X < 2 .AND. Y > 3 ref boris beizer

Domain Testing - Domains & Paths - Bug Assumptions Boundary errors…. Tilted Missing Extra boundary ref boris beizer

Domain Testing - Domains & Paths - Bug Assumptions Closure Reversal Faulty Logic Simplification of compound predicates X >= 3 ref boris beizer

Domain Testing - Domains & Paths - Bug Assumptions Strategy for domain testing in 2-dim ref boris beizer

Domain Testing - Domains & Paths - Restrictions U3 In General… Coincidental Correctness DT cannot detect Example Representative outcome Partition testing Input Equivalence F1(x) : +1 D1 D2 F2(x) : -1 ref boris beizer

Domain Testing - Domains & Paths U3 Domain Testing Model INPUT CLASSIFY DO CASE 1 OUTPUT DO CASE 2 DO CASE 3 DO CASE n D1 D3 (x, y, z, …) (1, 2, 3, 4, 5,…) { Outcome } D2 Dn Function f1 f2 f3 f4 ref boris beizer

Domain Testing - Domains & Paths - - Restrictions U3 Simple boundaries & Compound predicates D1 X 16 X >=0 .AND. X <= 16 Y = 1 .AND. X >=0 .AND. X <= 16 X 16 Y D1 ref boris beizer

Domain Testing - Domains & Paths - - Restrictions U3 Compound predicates…. Impact of .OR. Concave, Disconnected Adjacent domains with same function Example A B C + D E F Eliminate compound predicates ref boris beizer

Domain Testing - Domains & Paths - Restrictions U3 Functional Homogeneity of Bugs Functional form still Retained a x + b y >= c Bugs are only in a, b, c ref boris beizer

Domain Testing - Domains & Paths - - Restrictions U3 Linear Vector Space Linear boundary predicate, Interpreted Simple relational operators Conversion to linear vector space 2-d Polar co-ordinates Polynomials Problems with Non-linear Boundaries ref boris beizer

Domain Testing - Domains & Paths - Restrictions U3 Loop-free Software Predicate for each iteration Loop over the entire transaction Definite loop ref boris beizer

Domain Testing - Domains & Paths – Nice & Ugly domains Nice Domains Requirements Bugs  ill-defined domains Before DT Analyze specs Make the Boundary Specs Consistent & Complete ref boris beizer

Domain Testing - Domains & Paths – Nice domains U3 Implemented Domains Complete, Consistent & Process all inputs Specified Domains Incomplete, Inconsistent Programmer’s / Designer’s Effort ref boris beizer

Domain Testing - Domains & Paths – Nice domains U3 Nice Domains Linear, Complete, Systematic, Orthogonal, Consistently Closed, & Convex U1 U2 V1 V2 D11 D12 D21 D22 ref boris beizer

Domain Testing - Domains & Paths – Nice domains U3 Nice Domains Advantages Ease of DT Fewer Bug Occurrences ref boris beizer

Domain Testing - Domains & Paths – Nice domains U3 Nice Domains Boundaries are Linear Complete Systematic Orthogonal Closure consistency Convex Simply connected ref boris beizer

Domain Testing - Domains & Paths – Nice domains U3 Linear Boundaries Interpreted linear inequalities n-dim Hyperplane: n+1 Points n+1 + 1 Test Cases Non-Linear Transform ref boris beizer

Domain Testing - Domains & Paths – Nice domains U3 Complete Boundaries Span the total number space (-, +) One set of Tests Incomplete… Reasons Tests ref boris beizer

Domain Testing - Domains & Paths - Nice domains U3 Systematic Boundaries Linear Inequalities differing by a constant fj (X) ≥ kj or, fj (X) ≥ g (j, c) g (j, c) = j + k * c Parallel lines Identical Sectors in a Circle DT Test a domain  Tests for other Domains ref boris beizer

Domain Testing - Domains & Paths - Nice domains U3 Orthogonal Boundaries Two boundaries or, boundary sets Parallel to axes DT Each Set Independently # Tests  O (n) Uj Vj ref boris beizer

Domain Testing - Domains & Paths - Nice domains U3 Orthogonal Boundaries Tilted sets transformation Test cases: O(n) Concentric circles with radial lines Rectangular coordinates Polar r ≥ aj .AND. r < aj+1 .AND. ≥ j .AND.  < j+1 ref boris beizer

Domain Testing - Domains & Paths - Nice domains U3 Non-Orthogonal Boundaries Test Intersections  O ( n2 ) + O (n) ref boris beizer

Domain Testing - Domains & Paths - Nice domains U3 Closure Consistency A Simple pattern in all boundary closures Example Same relational operator for systematic boundaries D3 D1 D2 D4 ref boris beizer

Domain Testing - Domains & Paths - Nice domains U3 Convex Domain Line joining any two points lies with in the domain DT n on-points & 1 off-pt Concave “ But, However, Except, Or … “ in Specs Handle with special care ref boris beizer

Domain Testing - Domains & Paths - Nice domains U3 Simply Connected Domain In a single piece 2 complementary domains D1: Convex  D2: Concave, not-Connected Programmer / Designer Convex part First D2 D1 D3 D1 D2 ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Generally, From Bad Specs Programmer / Designer Simplifies => Ugly to Good possibility of Introduction of more bugs ?!? ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Causes Non-linear Boundaries Ambiguities & Contradictions Simplifying the topology Rectifying boundary closures ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Non-linear Boundaries Transform ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Ambiguities Holes in input vector space. Missing boundary Detected by Specification languages & tools. ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Contradictions.. Overlapping of Domain Specs Closure Specs D3 D1 D2 ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Simplifying the Topology…. Complexity ! Concavity, Holes, Disconnectedness ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Simplifying the Topology…. Smoothing out concavity Filling in Holes ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Simplifying the Topology…. Joining the pieces Correct: Connect disconnected boundary segments Extend boundaries to infinity ref boris beizer

Domain Testing - Domains & Paths – Ugly Domains Rectifying Boundary Closures. make closures in one direction for parallel boundaries with closures in both directions Force a Bounding Hyperplane to belong to the Domain. Consistent Direction Inclusion / Exclusion Consistency ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 General DT Strategy Select test points near the boundaries. Define test strategy for each possible bug related to boundary Test points for a domain useful to test its adjacent domain. Run the tests. By post test analysis determine if any boundaries are faulty & if so how? Run enough tests to verify every boundary of every domain ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 DT for Specific Domain Bugs Generally, Interior point Exterior point Epsilon neighborhood Extreme point On point ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 DT for Specific Domain Bugs Off Points Extreme point Boundary point Epsilon neighborhood On Points Domain D1 ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 1-d Domains 2-d Domains Equality & inequality Predicates Random Testing Testing n-dimensional Domains ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Testing 1-d Domains : Bugs with open boundaries A B x A B x Closure Bug A B x x1 Shift left Bug A B x x1 Shift Right Bug ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Testing 1-d Domains : Bugs with open boundaries A B x A B x Missing Boundary Extra Boundary A B x1 x C Bugs with Closed Boundaries : Similar to the above ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Testing 2-d Domains Closure bug Boundary Shift : up / down Tilted Boundary Extra Boundary Missing Boundary ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Domain Testing - Domains & Paths – Domain Testing 2 n : Domains share tests 3 n : no sharing of tests by domains Strategy for domain testing in 2-dim ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Equality & Inequality Predicates An Equality predicate defines a line in 2-d a A c’ c c d B b To avoid bugs ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Random Testing A Point in the center : verifies computation ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Testing n-Dimensional Domains (strategy) n-dimensions, p boundary segments (n+1)*p test cases : n on points & 1 off point Extreme pt shared : 2 * p points Equalities over m-dimensions create a subspace of n-m dimensions Orthogonal domains with consistent boundary closures, orthogonal to the axes & complete boundaries Independent testing ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Procedure for solving for values Simple procedure. Need tools. Identify input variables Identify variables which appear in domain-defining predicates, such as control-flow predicates ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Procedure Interpret all domain predicates in terms of input variables: Transform non-linear to linear Find data flow path Predicate expression with p # predicates. Find # domains : < 2 p Solve inequalities for extreme points Use extreme points to solve for nearby on points … ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Effectiveness Cost effective Bugs on boundaries, extreme points Hardware logic testing – tool intensive ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Domains & Interface Testing Domain Range Range Variable Domain Classify Function / Routine ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Domains & Interface Testing Span compatibility Routine 2 Routine 1 Domain Variable Range for Routine2 Domain for Routine1 ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Domains & Interface Testing Closure compatibility ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Interface Range/Domain Compatibility Testing Test each var independently Find an inverse function Build a super domain ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Finding the values / solving inequalities ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Domains and Testability Linearizing transformations Polynomial Rectangular to polar Generic & rational => Taylor series ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Domains and Testability Perform transformations for better testing. Co-ordinate transformations ref boris beizer

Domain Testing - Domains & Paths – Domain Testing U3 Domains and Testability Canonical Program Form ref boris beizer

Domain Testing – Questions from the previous year’s exams Explain Nice & Ugly Domains. What is domain testing? Discuss applications of domain testing Explain the domain boundary bugs for two dimensional domains What is the purpose of Domain Testing? Give its schematic representation Define the following concepts. Domains 2. Domain closure 3. Domain dimensionality 4. Bug assumptions for Domain Testing Explain simple domain boundaries & compound predicates. Classify what can go wrong with boundaries, then define a test strategy for each case in domain testing. ref boris beizer

Domain Testing To Unit 5…Paths, Path Products & Regular Expressions ref boris beizer