1. Conformance Test Suites, Extensionally Arend Rensink University of Twente Dutch Workshop on Formal Testing Techniques University of Twente 13 September 2007

2. DWFTT, September 2007Conformance Test Suites, Extensionally2 Automaton construction Approach Specification Suspension Automaton Test Suite Test Automaton Implementation Under Test Formal Physical Non-deterministic algorithm Closer to actual implementation Finite if spec is regular On-the-fly construction Completeness criteria Direct reasoning & manipulation Current New

3. DWFTT, September 2007Conformance Test Suites, Extensionally3 Background Specification s given by LTS – Distinguished inputs and outputs – Weakly convergent Stable, output-free states: quiescent – Made explicit by  -labelled self-loop – Suspension language Susp(s): traces with  ’s Implementation i modelled by IOTS – Input enabled Conformance (ioco) i ioco s, 8 w 2 Susp(s): Out(i after w) µ Out(s after w)

4. DWFTT, September 2007Conformance Test Suites, Extensionally4 Example Self-closing door Wrong implementation: press open press close  input: press outputs: open close press open press close press  Error:  2 Out(i after press ¢ open)  explicit quiescence input enabled

5. DWFTT, September 2007Conformance Test Suites, Extensionally5 Example (continued) Alternative specs: press open press close press non-determinism under-specification for press press open close press

6. DWFTT, September 2007Conformance Test Suites, Extensionally6 IOCO analyzed The following are equivalent: – 8 w 2 Susp(s): Out(i after w) µ Out(s after w) – 8 w 2 Susp(s), x 2 U  : w ¢ x 2 Susp(i) ) w ¢ x 2 Susp(s) – Susp(i) Å (Susp(s) ¢ U  ) µ Susp(s) – Susp(i) Å (Susp(s) ¢ U  n Susp(s)) = ; So the following are equivalent: – i conforms to s – i has no suspension trace accepted by Susp(s) ¢ U  n Susp(s)

7. DWFTT, September 2007Conformance Test Suites, Extensionally7 Intuition Susp(s): correct behaviour Susp(s) ¢ U  : Just one more response… Susp(s) ¢ U  n Susp(s): Forbidden paths

8. DWFTT, September 2007Conformance Test Suites, Extensionally8 Example press open press close press open press close  open close  close  open press Spec sConstruct: Susp(s) ¢ U  n Susp(s) press open press close open press close  open close close  open  press Example error trace: press ¢ open ¢  final state

9. DWFTT, September 2007Conformance Test Suites, Extensionally9  reduction &  -saturation Given a set of traces W µ L  * – W -  removes all traces from W with multiple successive  ’s (of the form v ¢  ¢   ¢ w) – W +  adds v ¢  + ¢ w for all v ¢  ¢ w 2 W Facts: If W is a suspension language – W +  = W – (W -  ) +  = W Improvement on previous result: – i conforms to s iff – i has no suspension trace accepted by Susp(s) -  ¢ U  n Susp(s) remove  -cycles

10. DWFTT, September 2007Conformance Test Suites, Extensionally10 Test automata Let T be an arbitrary automaton (with  ’s) T accepts i, Susp(i) Å Lang(T) = ; Properties defined for arbitrary T – soundness: i ioco s implies T accepts i – exhaustiveness: T accepts i implies i ioco s Seen previously: – Susp(s) -  ¢ U  n Susp(s) sound & exhaustive for s General criteria (main result, part 1): – T is sound for s if and only if Lang(T) µ (Susp(s) ¢ U  n Susp(s)) ¢ L  * – T is exhaustive for s if Lang(T) ¶ (Susp(s) -  ¯ U  n Susp(s)) No output after 

11. DWFTT, September 2007Conformance Test Suites, Extensionally11 Main result (part 2) Define: T is full if – v ·  w 2 Lang(T) implies v 2 Lang(T) – v ¢ a ¢ w 2 Lang(T) implies v ¢  ¢ a ¢ w 2 Lang(T) – v ¢ a 2 Lang(T) implies v 2 Lang(T) – v ¢ U  µ Lang(T) implies v 2 Lang(T) For all i: fill(T) accepts i iff T accepts i Necessary criterion for exhaustiveness: – If T is sound for s, then T is exhaustive for s only if Lang(fill(T)) ¶ (Susp(s) -  ¯ U  n Susp(s)) fill(T) can be constructed from T

12. DWFTT, September 2007Conformance Test Suites, Extensionally12 Example press open press close press Spec s Susp(s) -  ¯ U  n Susp(s) press open press close  open close  close  open press open press close  open close  close  open press Susp(s) ¢ U  n Susp(s) Straightforward automata constructions only

13. DWFTT, September 2007Conformance Test Suites, Extensionally13 Given: Possible with extensional characterisation Compositionality Test T sound/complete for Spec s Spec s’ Composition (Parallel composition, action refinement, …) Test T’ Composition Desired:

14. DWFTT, September 2007Conformance Test Suites, Extensionally14 Automaton construction From test automata to test suites Specification Suspension Automaton Test Suite Test Automaton Implementation Under Test Formal Physical Non-deterministic algorithm

15. DWFTT, September 2007Conformance Test Suites, Extensionally15 From test automata to test suites Equivalent testing power: – Test automaton T – Set {T i } i with  i Lang(T i ) = Lang(T) Tretmans test cases t as test automata: – Cycle-free – One final accepting state (= fail) – One final non-accepting state (= pass) – Non-final states either stimulus or response Tretmans’ algorithm generates {t i } i s.t.: – Lang(t i ) µ Susp(s) ¢ U  n Susp(s) for all t i – 9 i: w 2 Lang(t i ) for all w 2 Susp(s) ¢ U  n Susp(s)

16. DWFTT, September 2007Conformance Test Suites, Extensionally16 Automaton construction Direct testing with automata Specification Suspension Automaton Test Suite Test Automaton Implementation Under Test Formal Physical Non-deterministic algorithm

17. DWFTT, September 2007Conformance Test Suites, Extensionally17 Direct testing with automata Given a test automaton T – Deterministic, with explicit quiescence – Corresponds to entire test suite Synchronisation (product) of IuT and T: – Stimulus specified by T sent to IuT – Response of IuT mimicked in T Test verdict – Final state of T reached: fail – Response of IuT absent in T: pass – No reachable final state in T: pass – Done testing: pass

18. DWFTT, September 2007Conformance Test Suites, Extensionally18 Conclusion Test automata fill gap in existing theory – Yield precise conditions for completeness – Enable constructions on test suite – Work modulo Tretmans’ “test criteria” (not covered in this presentation) Re-opens discussion on – Interaction between test and IuT – Input enabledness of test cases