Department of CS and Mathematics, University of Pitesti State-based Testing is Functional Testing ! Florentin Ipate, Raluca Lefticaru University of Pitesti, Romania TAIC PART 2007

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Background Over the last years we have developed specification based test generation techniques proven to determine system correctness under certain “design for test” conditions. State-based specifications – stream X-machine (SXM). Industrial evidence of the power and practicality of these techniques. Joint research with the Verification & Testing Research Group of the University of Sheffield, UK (Mike Holcombe).

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Class testing State based languages (e.g. UML statecharts) are widely used to model class behaviour. New challenge: extend our SXM based techniques to class testing. Gather empirical evidence about their effectiveness.

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Empirical evidence Empirical studies show that state-based testing is more effective than random testing but it is not likely to be sufficient unless the state diagram is straightforward. They advocate the use of functional methods to complement state-based testing. Our experiments (in which we evaluated our own SXM based techniques) also back these findings.

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Questions Why ? Can functional testing and state-based testing be separated ?

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Our answers The modest fault detection of state-based class testing is actually due to the inappropriate state diagram used. Functional testing of a class can be reduced to state-based testing, provided that the right state model is produced.

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 How is a state diagram constructed ? Usually: ad-hoc process, using conventional wisdom. Example: a stack is associated with a 3 state diagram. push[0 < L < k-1] pop[1 < L < k] pushErr[L = k] popErr[L = 0] pop[L = k]pop[L = 1] push[L = 0]push[L = k-1] 012

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 How is a state diagram constructed ? Suppose we add a new method, count(l) that finds the number of distinct elements among the last l. A 3-state diagram is no longer appropriate. push[0 < L < k-1] pop[1 < L < k] count pushErr[L = k] count popErr[L = 0] count pop[L = k]pop[L = 1] push[L = 0]push[L = k-1] 012

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Our approach Construct a diagram for each class method to be tested. The construction of the diagram – based on a partitioning of the domain of the class method. Thus, state-based testing is functional testing !

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 State diagram construction Partition the domain according to some functional criterion. Derive (not necessarily disjoint) data sub-domains by “hiding” the inputs. Derive the states of the diagram by removing the overlapping of data sub-domains. Construct the transitions by considering each pair of states and checking whether the pair is related to a method. Split the transitions according to the original partitioning criterion.

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 State diagram construction (example) push : 3 state diagram pop : 3 state diagram count – domain partition  count(l) = 0  count(l) = 1  1 < count(l) < L  count(l) = L

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 State diagram construction (example) push[i→i,2≤i≤k-1] push[i→i+1,2≤i≤k-2] pop[i→i,2≤i≤k-1] pop[i→i-1,3≤i≤k-1] count[p 2 ] count[p 3 ] count[p1] pop[1→0] push[0→1] count[p 2 ] count[p 3 ] count[p 4 ] count[p 2 ] push[1→1] pop[1→1] pop[k→k-1] pop[2→1] push[1→2] push[k-1→k]

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Previous approaches Construct a composite (all-in-one) diagram rather than a separate diagram for each method. A method may induce at most one transition between a given pair of states  Original partitioning criterion is not fully used. Stack example:  9-state composite diagram but  One loop-back count transition for each state, so important test cases can be missed out.

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Test generation Adaptation of the W-method. Transition cover: only the transitions labeled by the method in question are considered. Characterization set: identifies each state of the diagram. Alternative : an abstract oracle associates with each state a predicate rather than a sequence of transitions.

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Experimental results A series of experiments to evaluate the cost effectiveness of state-based testing. Java implementation of several classes, e.g. bounded stack, book from a library, vending machine. Test suites were executed against mutant programs automatically generated by MuJava. Test cost measure = number of mutants killed / total length

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Experimental results Bounded stack: approximately 90 LOC, 219 mutants. Partitioning criterion (count) No. of sequences Total length Mutation score Test cost measure % % % % %2.75 Our approach Composite (all-in-one) diagram

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Experimental results Book class (library): approximately 110 LOC, 136 mutants. No. of sequences Total length Mutation score Test cost measure % %1.64 Our approach Full predicate

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Mutants left undetected Equivalent mutants. For which stronger criteria are needed. Caused by attributes (dueDate) for which no obvious partitioning criterion exists: precise oracle needed. Class mutants (JSI): simple tests in which more than one instance is used.

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Further work Evolutionary methods for automatically converting sequences of methods into actual test data. Deriving the state model using evolutionary methods ?

Department of CS and Mathematics, University of Pitesti TAIC PART 2007 Thank you for your attention !