Implementation-Based Testing Review Class 9 Implementation-based Testing CEN 5076 Class 10 – 11/07

CEN 5076 Class /072 Test Adequacy Criteria Definitions from Zhu et al. ‘97: 1.Test Data Adequacy as Stopping Rules: A test data adequacy criterion C is a function C: P X S X T → {true, false}. C(p, s, t) = true means that t is adequate for testing program p against specification s according to criterion C, otherwise t is inadequate.

CEN 5076 Class /073 Test Adequacy Criteria 2.Test Data Adequacy Criteria as Measurements: A test data adequacy criterion C is a function C: P X S X T → [0,1]. C(p, s, t) = r means that the adequate of testing the program p by the test set t w.r.t. specification s is of degree r according to criterion C. The greater the value of r, the more adequate the testing.

CEN 5076 Class /074 Test Adequacy Criteria 3.Test Data Adequacy Criteria as Generators: A test data adequacy criterion C is a function C: P X S → 2 T. A test set t C(p, s) means that t satisfies C w.r.t. p and s, and it is said that t is adequate for (p, s) according to C.

CEN 5076 Class /075 Types of Testing Specification-based (functional) Specifies the required testing in terms of identified features of the specification or the requirements of the software, so that a test set is adequate if all the identified features have been fully exercised. [Zhu ’97] Implementation-based (program-based) Specifies testing requirements in terms of the program under test and decides if a test set is adequate according to whether the program has been thoroughly exercised.

CEN 5076 Class /076 Specification-based Testing cont A specification has two facets: a syntactic structure and a semantic structure. There are two role the specifications plays during software testing: 1.To provide the necessary information to check whether the output of the program is correct, the oracle problem. 2.Provide information to select test cases and to measure test adequacy

CEN 5076 Class /077 Specification-based Testing cont Techniques used to represent the specification: 1.Logic-based 2.State machines 3.Algebraic specs. Model-based Property-oriented

CEN 5076 Class /078 Specification-based Testing cont Test generation for Logic-based models: All-Explicit variants All-Variants, All-True, All-False, All-Primes Each-Condition/All-Conditions BDD-Determinants Variable Negation Nonbinary Variable Domain Analysis

CEN 5076 Class /079 Specification-based Testing cont Test generation for state-based models: Piecewise coverage - all states, all events, all actions. All transitions All n-Transition Sequences All Round-trip Paths M-length Signature

CEN 5076 Class /0710 Exhaustive Strategy M-length Signature All Round-trip Paths All Transition k-tuples All Transitions All StatesAll EventsAll Actions Faults Revealed All possible invalid states All event and action faults All invalid states Up to M extra states All event and action faults All invalid/missing states Some extra states All event and action faults Some missing/invalid states Some event and action faults Fig. State-based test power hierarchy

CEN 5076 Class /0711 Specification-based Testing cont 3.Algebraic Specifications Algebraic spec consists of a set of operations that the operations of the software must satisfy. An algebraic spec has two parts: -Syntactic - consists of functions name and their signatures (the input and output types). -Semantic part – consists of a list of axioms describing relation among functions

CEN 5076 Class /0712 Specification-based Testing cont Syntax part: -Functions that return values of type T (type of algebraic spec) are called constructors and transformers. -Functions that return values other than T are called observers. Semantic part: -Allows for a list of preconditions describing the domains of functions. -Allows functions to return an error if element is outside of the domain.

CEN 5076 Class /0713 Specification-based Testing cont An equation in an algebraic spec. consists of two terms as two sides of the eqn. A term is constructed from three types of symbols: variables, constants, and operators. Each term has two interpretations in the context of testing. 1.As a sequence of calls to the operations that implement the operators specified in the spec. 2.As a value that is the result of the sequence of operations.

CEN 5076 Class /0714 Specification-based Testing cont For a algebraic spec to be useful it must be consistent and sufficiently complete. -Consistent must not contain any contradictory axioms i.e., no contradiction should be derivable. -Sufficiently complete if for every sequence w in W the result of applying each observer is defined. W is the set of all operation sequences consisting of constructors and transformers. See handout for example of an algebraic spec for a priority queue ASTOOT Doong and Frankl ‘94

CEN 5076 Class /0715 Specification-based Testing cont The regularity hypothesis [Bouge et al. 1986] assumes that for some complexity degree k, if a program satisfies an equation t(x) = t’(x) on all data of complexity not higher than k, then the program satisfies the eqn on all data. No way to determine the complexity k s.t. only the test cases of complexity less than k need to be tested.

CEN 5076 Class /0716 Specification-based Testing cont Bouge et al Bernot et al.1991 suggest that the selection of test cases should be based on partitioning the set of ground terms according to their complexity so that the regularity and uniformity hypotheses in the subsets in the partitioned are assumed. The complexity of a test case is the depth of nesting of the operators in the ground term. Ground terms are terms without variables.

CEN 5076 Class /0717 Specification-based Testing cont The selection of test cases should first consider constants specified by the spec, then all the values generated by one application of operations on constants, the values generated by two applications on constants etc..

CEN 5076 Class /0718 Specification-based Testing cont Constructing Test Cases for Classes: Test cases are usually identified from the class specification. The spec can be stated in OCL, natural language, and/or state transition diagrams. Test case construction from pre- and postconditions -Create test cases that covers all possible combinations of situations in which a precondition can hold and a postcondition can be achieved. P. 169 – 176 in text.

CEN 5076 Class /0719 Specification-based Testing cont It is possible to express the adequacy for the spec in terms of the number of pairs of pre- and postconditions that have been covered.

CEN 5076 Class /0720 Implementation-based Testing Specifies testing requirements in terms of the program under test and decides if a test set is adequate according to whether the program has been thoroughly exercised. There are two main groups of implementation-based structural test adequacy criteria. 1.Control-flow 2.Data-flow

CEN 5076 Class /0721 Control Flow Adequacy Criteria Flow Graph (FG) [Binder 00] -A segment is represented by a node in the FG (a sphere). A segment is one or more contiguous statements with no conditionally executed statements. -A conditional transfer of control is a branch. A branch is represented by an outgoing edge in the FG. -The entry point of a method is represented by the entry node, which is a node no incoming edges. The exit point of a method is represented by the exit node, which has no outbound edges.

CEN 5076 Class /0722 Control Flow Adequacy Criteria cont Example: Source code 1.public int Fun(int x){ 2. k = 0; 3. while (x <= 10 && k < 3){ 4.if (x%2 != 0) 5.k = k + 1; 6.x = x + 1; 7. } 8. if (x < 0){ 9.x = 10; 10.k = 0; 11. } 12. return k; } K=0 x<=10 && k<3 x%2 != 0 x=x+1 k=k+1 return k B C D E F I Entry Flow graph A X<0 X = 10 K = 0 T F T T F F G H

CEN 5076 Class /0723 Control Flow Adequacy Criteria cont Types of code coverage based on FG model: 1.Statement coverage – A set P of execution paths satisfies the statement coverage criterion iff for all nodes n in the FG, there is at least one path p in P s.t. n is on the path p. Generate test data to execute every stmt in the program at least once. Give a value(s) of x in the example that satisfies statement coverage.

CEN 5076 Class /0724 Control Flow Adequacy Criteria cont 2.Branch coverage – A set P of execution paths satisfies the branch coverage criterion iff for all edges e in the FG, there is at least one path p in P s.t. p contains edge e. Generate test data to exercise the true and false outcomes of every decision. Give a value(s) of x in the example that satisfies branch coverage.

CEN 5076 Class /0725 Control Flow Adequacy Criteria cont 3.Path coverage – A set P of execution paths satisfies the path coverage criterion iff P contains all execution paths from the begin node to the end node in the FG. Generate test data to exercise all paths in the program. Note every path through a loop is considered to be different and distinct. Give a value(s) of x in the example that satisfies path coverage.

CEN 5076 Class /0726 Control Flow Adequacy Criteria cont 4.Cyclomatic-Number Criterion – A set P of execution paths satisfies the cyclomatic number criterion iff P contains at least one set of v independent paths, where v = e – n + p is the cyclomatic number of the FG. n - number of vertices, e - is the number of edges, p - is the number of strongly connected components. A graph is strongly connected if, for any two nodes a and b there exists a path from a to b and a path from b to a.

CEN 5076 Class /0727 Control Flow Adequacy Criteria cont Basis path testing – is a way of easily identifying the upper bound for the number of paths necessary in order to achieve branch coverage. v = (e + 1) - n + 1 McCabe Cycolmatic Complexity Number Note this is for a strongly connected FG, add edge from exit node to entry node, hence e + 1. Note some formulae include the extra edge in the “e”. Identify the basis set for the FG example.

CEN 5076 Class /0728 Control Flow Adequacy Criteria cont Types of code coverage based on program text: 5.Multiple Condition Coverage – A test set T is said to be adequate according to the multiple-condition- coverage criterion if, for every condition C, which consists of atomic predicates (p 1, p 2, …,p n ), and all the possible combinations (b 1, b 2, …,b n ) of their truth values, there is al least one test case in T s.t. the value of p i equals b i, i= 1, 2, …,n. Generate test data to exercise all possible combinations of true and false outcomes of conditions within a decision. Create test set for example that satisfies the MCC

CEN 5076 Class /0729 Control Flow Adequacy Criteria cont Points to note: Path coverage not always practical due to loops i.e., infinite number of paths. Statement and branch coverage are not finitely applicable due to dead code and infeasible branches. Can define a finitely applicable version of the criteria.

CEN 5076 Class /0730 Control Flow Adequacy Criteria cont However we may not be able to decide whether a test set satisfies a given feasible adequacy criterion e.g. whether a statement in a program is feasible is undecidable. Research into flow graph criteria has focused on the most important subsets of paths e.g., length-n path coverage, level-i path coverage, cyclomatic-number coverage,

CEN 5076 Class /0731 Data-Flow Adequacy Criteria Data-flow analysis of test adequacy is concerned with the coverage of flow-graph paths that are significant for the data flow in the program. Each variable occurrence is classified as either a definition occurrence or a use occurrence. A definitional occurrence (def) of a variable is where a value is bound to the variable. A use occurrence (use) of a variable is where the value of the variable is referred.

CEN 5076 Class /0732 Data-Flow Adequacy Criteria cont Two types of use occurrences:  Predicate use (p-use) – if the value of a variable is used to decide whether a predicate is true or false.  Computation use (c-use) – variable used in the evaluation of an expression, call-by-value argument or an output statement. Data-flow test adequacy analysis is concerned with subpaths from nodes with a def to nodes with a use. A def-clear path w.r.t a variable x is a path where for all nodes in the path there is no def occurrence of x.

CEN 5076 Class /0733 Data-Flow Adequacy Criteria cont A global c-use of a variable x is where no def of x precedes the c-use within the node in which it occurs. A def of a variable x at node u reaches a c-use at node v iff there is a path from u to v s.t. p =(u, w 1, w 2, …,w n, v) and (w 1, w 2, …,w n ) is a def-clear w.r.t x and the occurrence of x at v is a global c-use. We say that the def of x at u reaches the c-use of x at v through path p.

CEN 5076 Class /0734 Data-Flow Adequacy Criteria cont A def of a variable x at node u reaches a p-use at node v iff there is a path from u to v s.t. p =(u, w 1, w 2, …,w n, v) and (w 1, w 2, …,w n ) is a def-clear w.r.t x and there is a predicate occurrence of x associated with the edge from w n to v. We say that u reaches the predicate use of x on the edge (w n, v) through path p. If there is at least one input datum that can actually cause the execution of the path, then a def feasibly reaches a use of the def.

CEN 5076 Class /0735 Data-Flow Adequacy Criteria cont All Definitions Criterion – A set P of execution paths satisfies the all-defs criterion iff for all defs of x s.t. there is a use of x which is feasibly reached from the def, there is at least one path p in P s.t. p includes a subpath through which the def of x reaches some use of x. All Uses Criterion – A set P of execution paths satisfies the all-uses criterion iff for all defs of x and all uses of x that the def feasibly reaches, there is at least one path p in P s.t. p includes a subpath through which that def reaches the use.

CEN 5076 Class /0736 Data-Flow Adequacy Criteria cont All Definition-Use-Paths Criterion (All DU- Paths) – A set P of execution paths satisfies the all-du-paths criterion iff for all defs of x and all paths q through which that def reaches a use of x, there is at least one path p in P s.t. q is a subpath of p, and q is cycle-free or contains only simple cycles. Note such a path may be infeasible.

CEN 5076 Class /0737 Data-Flow Adequacy Criteria cont Points to note: Data flow testing methods make no distinction between atomic data and structured or aggregate data. Modification and references to an element of a structured datum are regarded as modifications and references to the whole datum. Can lead to errors [Hamlet ’93]. Dynamic data not considered in the data flow testing i.e., a pointer variable may refer to a number of data and vice versa.

CEN 5076 Class /0738 Data-Flow Adequacy Criteria cont For classes there is intra-method, inter-method, intra-class, and inter-class. For each of the data-flow coverage criteria try to identify test cases for the example that provides adequate coverage for feasible paths. Common errors: Using an undefined or uninitialized variable Assigning a value to a variable more than once without an intermediate reference. Sending a modifier message to an object more than once without an intermediate accessor method.

CEN 5076 Class /0739 Data-Flow Adequacy Criteria cont Sending a modifier message to an object more than once without an intermediate accessor method. Deallocating or reinitializing a variable before it is used. Using incorrect values in the signature of a polymorphic message, resulting in incorrect binding. The container problem: Deleting a collection of object, thereby leaving its members orphaned and inaccessible. More likely in C++ than Java.

CEN 5076 Class /0740 All-paths All-du-paths All-uses All-p-uses/ Some-c-uses All-c-uses/ Some-p-uses All-defs All-p-uses All-edges (branch coverage) All-nodes (statement coverage) Partial Ordering of Coverage criteria Rapps and Weyuker ‘85 Criteria A subsumes B AB

CEN 5076 Class /0741 Data-Flow Adequacy Criteria cont The subsume relation is actually a comparison of adequacy criteria according to the severity of the testing method. Does not say anything about the ability to detect faults.

CEN 5076 Class /0742 Review Exam 2 Definitions: software testing, testability, test data selection criteria, test data adequacy criteria, specification-based testing, implementation-based testing. Know the following axioms: antiextensionality, general multiple change, antidecomposition, anticomposition. Examples of tests that are valid but not reliable and vice versa. The formulas will be provided.

CEN 5076 Class /0743 Review Exam 2 cont Software metrics –Be able to describe how software metrics are used in testing –Know at least (3) pre-OO metrics and (3) OO metrics Specification-based testing: –Types of coverage criteria for logic-based systems, state machines, algebraic systems. –Be able to draw a state machine for a given problem and generate test cases to provide specified coverage e.g., all transition k-tuples, all transitions, all states, all events, all actions.

CEN 5076 Class /0744 Review Exam 2 cont Implementation-based testing: –Control flow: statement coverage, branch coverage, path coverage, basis path coverage, multiple condition coverage (MCC). –Data flow: all defs, all def-use-paths, all uses. –Be able to draw a flow graph for a method; identify def-use pairs for the variables in a method. –Understand the concept of infeasible coverage criteria.

CEN 5076 Class /0745 Review Exam 2 cont Implementation-based testing: –Given a specific control flow or data flow criteria be able to generate a test set to provide adequate coverage. E.g., be able to identify the set of basis paths in a flow graph for a method and generate test cases to cover all the basis paths.