1. Teooriapäevad, Vanaõue, Sept. 2007 Synthesis of Test Purpose Directed Reactive Planning Tester for Nondeterministic Systems 1 Jüri Vain, Kullo Raiend, Andres Kull, Juhan P. Ernits 1 Accepted in ASE 2007, Atlanta, Georgia, November 5-9, 2007

2. Teooriapäevad, Vanaõue, Sept. 2007 On-the-fly testing: what? and why?  The most appropriate technique for model-based testing of IUT modeled using nondeterministic models  Denotes test generation and execution algorithms that compute successive stimuli at runtime, directed by the test purpose and the observed outputs of the IUT  Advantages: The state-space explosion problem experienced by many offline test generation methods is reduced because only a limited part of the state-space needs to be kept track of at any point in time.  Drawbacks: Exhaustive planning is diffcult due to the limitations of the available computational resources at the time of test execution.

3. Teooriapäevad, Vanaõue, Sept. 2007 On-the-fly testing: scale of methods  Random walk (the simplest): select test stimuli in random inefficient - based on random exploration of the state space leads to test cases that are unreasonably long may leave the test purpose unachieved  RW with additional heuristics: E.g., anti-ant algorithms are applied for guiding the exploration of the state space [Li, Veanes] ........?  State space exploration with exhaustive planning - solving constraint systems at each step witness trace generated by model checking provides possibly optimal selection of the next test stimulus. in case of explicit state MC the size of the model is critical leads to the explosion of the state space specially in models that include "combination lock" or deep loops [J. Rushby].

4. Teooriapäevad, Vanaõue, Sept. 2007 On-the-fly testing: missing link - Reactive Planning Tester  Reactive planning [C. Williams and P. P. Nayak]: can cope with highly dynamic and unpredictable environments. instead of a complete plan with branches, a set of decision rules is produced directing towards the planning goal. just one subsequent input is computed at every step, based on the current context.  Reactive planning tester [J.Vain et al] Model “in-the-loop”: testing is guided by an EFSM model stimulae to IUT: outputs generated by model execution responses from IUT: inputs to the model decision rules of reactive planning are encoded in the guards of model transitions the rules are constructed by offline analysis based on the given IUT model and the test purpose.

5. Teooriapäevad, Vanaõue, Sept. 2007 The model  Key assumption: the IUT model is presented as an output observable nondeterministic state machine: FSM EFSM in which all transition paths are feasible (see Duale,2004 for algorithms of making EFSM feasible)

6. Teooriapäevad, Vanaõue, Sept. 2007 I/O EFSM (I)  Definition 1: (EFSM) M = (S; V; I; O; E), where S - finite set of states, s 0  S - initial state, V - finite set of variables with finite value domains, I - finite set of inputs, O - finite set of outputs, E - set of transitions.  Configuration of M is a pair (s;  ) where s  S    mapping from V to values.

7. Teooriapäevad, Vanaõue, Sept. 2007 I/O EFSM (II)  Transition e =(s; p; a; o; u; q),where s - the source state of the transition, q - target state of the transition (s; q) p - transition guard - a logic formula over V, a - input of M (a  I), o – output of M (o  O), u - update function over V.  Nondeterministic EFSM: more than one outgoing transitions enabled simultaneously

8. Teooriapäevad, Vanaõue, Sept. 2007 Example: nondeterministic EFSM

9. Teooriapäevad, Vanaõue, Sept. 2007 Test purpose (TP):  TP is a specific property of the IUT that the tester is set out to test.  In MBT the TPs are defined in terms of the IUT model elements  Examples: "test a state change from state A to state B in a model", "test whether some selected states of a model are visited", "test whether all transitions are visited at least once in a model", etc.  All TPs of examples can be specified in terms of structural elements of the model that should be traversed during the test execution

10. Teooriapäevad, Vanaõue, Sept. 2007 Encoding Test Purpose in the IUT Model  Trap - a boolean variable assignment attached to the transitions of the IUT model  A trap variable is initially set to false.  The trap update functions attached to the transitions are executed (set trap variables to true) when the transition is visited.

11. Teooriapäevad, Vanaõue, Sept. 2007 Example: nondeterministic EFSM extended with trap variables

12. Teooriapäevad, Vanaõue, Sept. 2007 Model of the tester  EFSM generated from the IUT model attributed with test purpose  I/O conformance of IUT and tester models is required  transition guards encode the rules of online planning  2 types of tester states : active – tester controls the next move passive – IUT controls the next move  2 types of transitions: Observable – source state is passive state, guard: true, Controllable – source state is active state, guard: p S  p T, where p S – guard of the dual transition of IUT, p T – gain guard The gain guard (defined on trap variables) must ensure that only those outgoing are enabled having maximum gain

13. Teooriapäevad, Vanaõue, Sept. 2007 Tester construction algorithm: a sketch  Construct a control structure structure of the tester model where the inputs and outputs are inverted. Split the transitions to pairs of controllable and observable transitions by i/o operation attached.  Extend the tester automaton controllable transitions guards with “place holders” for gain guards  The gain guards are construct with reachability analysis by taking into account the reachability of all trap-equipped transitions from a given state the length of the paths to them the current value (visited or not) of each trap.

14. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the control structure of the tester

15. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the gain guards (GG) (I): intuition  GG must guarantee that each transition enabled by GG is a prefix of some locally optimal (w.r.t. test purpose) path tester should terminate after the test goal is reached or all unvisited traps are unreachable from the current stat to have a quantitative measure of the gain of executing any transition e we define a gain function g e : TR e  B  Q +

16. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the gain guards (II): the gain function  g e = 0, if it is useless to fire the transition e from the current state with the current variable bindings  g e > 0, if fireing transition e from the current state with the current variable bindings visits or leads closer to at least one unvisited trap  g ei > g ej for transitions e i and e j with the same source state, if taking the transition e i leads to unvisited traps with smaller cost than taking the transition e j.  Having gain function g e with given properties define GG: p T  g e = max ek g ek  g e > 0

17. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the gain guards (III): shortest path tree  Reachability problem of trap labeled transitions can be reduced to single- source shortest path problem.  Arguments of the gain function g e are Shortest path tree TR e with root node e V T – vector of trap variables  To construct TR e we create a graph G = (V D,E D ) where the vertices V D of G correspond to the transitions of the M T, the edges E D of G represent the pairs of subsequent transitions sharing a state in M T.

18. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the gain guards (IV): shortest path tree (example) Figure 3: The dual graph of the tester model in Figure 2. Figure 4: The shortest-paths tree (left) and the reduced shortest-paths tree (right) from the transition e c 01 of the graph shown in Figure 3.

19. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the gain guards (IV): the gain function  Represent the reduced tree TR(e i, G) as a set of elementary sub- trees each specified by the production i  j{1,..n} j  Rewrite the right-hand sides of the productions as arithmetic terms: (3) t  i - trap variable t i lifted to type N, c - constant for the scaling of the numerical value of the gain function, d( 0, i ) the distance between vertices 0 and i, where l - the number of hyper-edges on the path between 0 and i w j – weight of j-th hyperedge

20. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the gain guards (IV): the gain function (continuation)  For each symbol i denoting a leaf vertex in TR(e,G) define a production rule (4)  Apply the production rules (3) and (4) starting from the root symbol 0 of TR(e,G) until all non-terminal symbols i are substituted with the terms that include only terminal symbols t  i and d( 0, i )

21. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the gain guards (IV): the gain function (example)

22. Teooriapäevad, Vanaõue, Sept. 2007 Constructing the gain guards (IV): the gain function (example continues)

23. Teooriapäevad, Vanaõue, Sept. 2007 Complexity of constructing and running the tester  The complexity of the synthesis of the reactive planning tester is determined by the complexity of constructing gain functions.  For each gain function the cost of finding the TR E by breadth-first-search is O(|V D | + |E D |) [Cormen], where |V D | = |E T | - number of transitions of M T |E D | - number of transition pairs of M T (is bounded by |E S | 2 )  For all controllable transitions of the M T the upper bound of the complexity of the computations of the gain functions is O(|E S | 3 ).  At runtime each choice by the tester takes O(|E S | 2 ) arithmetic operations to evaluate the gain functions

24. Teooriapäevad, Vanaõue, Sept. 2007 Experimental results

25. Teooriapäevad, Vanaõue, Sept. 2007 Conclusions  Efficiency of planning: number of rules that need to be evaluated at each step is relatively small (number of outgoing transition of current state) the execution of decision rules is significantly faster than finding the test path by MC but nevertheless leads to the test sequence that is lengthwise close to optimal.