1 Software Testing & Quality Assurance Lecture 14 Created by: Paulo Alencar Modified by: Frank Xu

2 Overview  Structural Testing  Introduction – General Concepts  Flow Graph Testing  Data Flow Testing  Definitions  Some Basic Data Flow Analysis Algorithms  Define/use Testing  Slice Based Testing  Guidelines and Observations  Hybrid Methods  Retrospective on Structural Testing

3 Issues on Structural Testing An important question relates to the point where we stop to test and release the system for beta testing. Some possible answers could be: –When all faults have been removed! –When we run out of time –When continued testing produces no new failures –When continued testing reveals no new faults –When we have exhausted the testing criteria and the test cases we have designed –When we reach a point of diminishing returns –When the mandated coverage has been achieved –When the failure intensity (failures / CPU hour) level drops below a certain point

4 Metrics for Testing Method Comparison (1) Functional testing methods may (eventually) provide a high level of path coverage, but at the cost of high redundancy The question is to define metrics that relate the effectiveness of a functional technique with that of a structural technique (path coverage) To start addressing this question we first observe that functional testing methods are measured according to the number of test cases they produce, while structural testing methods are measured according to the level of the corresponding path coverage they achieve.

5 Metrics for Testing Method Comparison (2) Let’s assume a functional testing method M, and a structural metric S that identifies s coverage elements in the unit under test Furthermore, let us assume that the functional testing method M produces m test cases, and these m test cases traverse n of the s structural coverage elements. We then define the following: –Coverage of methodology M with respect to metric S is defined as C(M,S) = n/s –Redundancy of methodology M with respect to metric S is defined as R(M,S) = m/s –Net redundancy of methodology M with respect to metric S is defined as NR(M,S) = m/n

6 Sample Comparison of Functional Testing Methods with respect to a Structural Metric MethodmnsC(M,S)R(M,S)NR(M,S) nominal Worst case MethodmnsC(M,S)R(M,S)NR(M,S) Output bva Decision Table S: Program Paths s: total number of paths n: number of paths traversed using method M m: number of test cases generated by method M M: Functional method Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen Triangle Problem Commission Problem

7 Sample Comparison of a Functional Testing Method (Output BVA) with respect to Structural Metrics (DD-Path, DU-Path, Slices) MethodmnsC(M,S)R(M,S)NR(M,S) DD-Path DU-Path Slices S: Structural Metric s: total number of paths in structural metric S n: number of paths traversed using method M m: number of test cases generated by method M M: Functional method Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen Commission Problem

8 Testing Sophistication vs. Number of Test Coverage Items DD-Path Basis Path DU-Path Slice low high Sophistication Number of Test Coverage Items Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

9 Test Coverage Items DD-Path Basis Path DU-Path Slice Number of Test Coverage Items 5 33 Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

10 Test Case Sophistication vs. Test Case Identification Effort DD-Path Basis Path DU-Path Slice low high Sophistication Effort to Identify Test Coverage Items Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

11 Remarks Functional testing is prone to gaps and redundancies Structural path-coverage testing considers infeasible paths Data flow testing considers program semantics on Control Flow Graph paths Hybrid approaches that combine functional-structural methods are useful Structural testing techniques are best suited for unit testing

12 Subsumption A test criterion C1 subsumes a test criterion C2 if every test suite that satisfies C1 also satisfies C2 Two test criteria C1 and C2 are equivalent if C1 subsumes C2 and vice-versa What is the subsumption relation among coverage criteria?

13 Subsumption Hierarchy – Control Flow

14 Subsumption Hierarchy – Data Flow