1. 1 Jan Tretmans jan.tretmans@esi.nl Embedded Systems Institute Eindhoven, NL Radboud University Nijmegen, NL Model-Based Testing with Labelled Transition Systems

2. © Jan Tretmans 2 Testing Testing: checking or measuring some quality characteristics of an executing object by performing experiments in a controlled way w.r.t. a specification IUT tester specification

3. © Jan Tretmans 3 Types of Testing unit integration system efficiency maintainability functionality white boxblack box Level of detail Accessibility Characteristics usability reliability module portability

4. © Jan Tretmans 4 Automated Model-Based TestingModel-BasedAutomatedTesting model IUT confto model TTCN test cases pass fail test tool test generation tool test execution tool IUT passes tests IUT confto model 

5. © Jan Tretmans 5 Towards Model Based Testing  Increase in complexity, and quest for higher quality software  testing effort grows exponentially with complexity  testing cannot keep pace with development  More abstraction  less detail  model-based development  Checking quality  practice: testing - ad hoc, too late, expensive, lot of time  research: formal verification - proofs, model checking,.... with disappointing practical impact

6. © Jan Tretmans 6 Towards Model Based Testing  Model based testing has potential to combine  practice - testing  theory- formal methods  Model Based Testing :  testing with respect to a (formal) model / specification state model, pre/post, CSP, Promela, UML, Spec#,....  promises better, faster, cheaper testing:  algorithmic generation of tests and test oracles : tools  formal and unambiguous basis for testing  measuring the completeness of tests  maintenance of tests through model modification

7. © Jan Tretmans 7 A Model-Based Development Process informal requirements specification realization design code formalizable validation formal verification testing model- based informal world world of models real world

8. © Jan Tretmans 8 formal world real world Formal Verification model m of i sat m modcheck s m sat s     completesound model checker m modcheck s assumption: m is valid model of i sat specification s implementation i

9. © Jan Tretmans 9 formal world real world Formal Testing specification s implementation i confto IUT passes T s IUT confto s  correctness of test generation ? IUT passes T s IUT fails T s test execution test generation test suite T S implementation i IUT

10. © Jan Tretmans 10 Approaches to Model-Based Testing Several modelling paradigms:  Finite State Machine  Pre/post-conditions  Labelled Transition Systems  Programs as Functions  Abstract Data Type testing ....... Labelled Transition Systems

11. © Jan Tretmans 11 Model-Based Testing for LTS test execution test generation tests model IUT confto Involves: model / specification implementation IUT + models of IUTs correctness tests test generation test execution test result analysis pass / fail

12. © Jan Tretmans 12 Labelled Transition System  S, L, T, s 0  ?coin ?button !alarm ?button !coffee states actions transitions T  S  (L  {  })  S initial state s 0  S Models of Specifications: Labelled Transition Systems

13. © Jan Tretmans 13 ?dub ! choc ?kwart !tea ! coffee ?dub ?kwart ?dub ?kwart ! choc ?dub !tea Example Models ! coffee ?dub !tea ?dub ! coffee ?dub ( Input-Enabled ) Transition Systems

14. © Jan Tretmans 14 i ioco s = def   Straces (s) : out (i after  )  out (s after  ) Intuition: i ioco-conforms to s, iff if i produces output x after trace , then s can produce x after  if i cannot produce any output after trace , then s cannot produce any output after  ( quiescence  ) Correctness Implementation Relation ioco

15. © Jan Tretmans 15 i ioco s = def   Straces (s) : out (i after  )  out (s after  ) Correctness Implementation Relation ioco p  p=  !x  L U  {  }. p !x out ( P )= { !x  L U | p !x, p  P }  {  | p  p, p  P } Straces ( s )= {   (L  {  })* | s  } p after  = { p’ | p  p’ }

16. © Jan Tretmans 16 ?dub ! choc ?kwart !tea ! coffee ?dub ?kwart ?dub ?kwart ! choc ?dub !tea ioco Implementation Relation ioco ! coffee ?dub !tea s ioco  ?dub ! coffee ?dub

17. © Jan Tretmans 17 Test Cases  ‘quiescence’ label   tree-structured  finite, deterministic  final states pass and fail  from each state  pass, fail :  either one input !a  or all outputs ?x and  ?coffee !dub !kwart ?tea ?coffee ?tea  !dub  pass fail pass Model of a test case = transition system :

18. © Jan Tretmans 18 Algorithm To generate a test case from transition system specification s 0 compute T(S), with S a set of states, and initially S = s 0 after  ; 1end test case pass For T(S), apply the following recursively, non-deterministically: 2supply input !a T ( S after ?a   ) ioco Test Generation Algorithm allowed outputs or  : !x  out ( S ) forbidden outputs or  : !y  out ( S ) 3observe output fail T ( S after !x ) fail allowed outputsforbidden outputs ?y  ?x

19. © Jan Tretmans 19 ?coffee  ?tea pass fail ?coffee pass fail  ?tea    Test Generation Example s ?dub !coffee ?dub test !dub

20. © Jan Tretmans 20 Test Execution Example Two test runs : t  i dub  pass  i' t  i dub coffee  i passes t ?coffee  ?tea pass fail ?coffee pass fail  ?tea test !dub i ?dub !coffee ?dub

21. © Jan Tretmans 21 Test Result Analysis Completeness of ioco Test Generation For every test t generated with algorithm we have:  Soundness : t will never fail with correct implementation i ioco s implies i passes t  Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s implies  t : i fails t

22. © Jan Tretmans 22 Formal Testing with Transition Systems exec : TESTS  IMPS   (OBS) gen : LTS   (TTS) T s  TTS s  LTS IUT  IMPS ioco i IUT  IOTS passes : IOTS  TTS  {pass,fail} Proof soundness and exhaustiveness:  i  IOTS. (  t  gen(s). i passes t )  i ioco s Test hypothesis :  IUT  IMP.  i IUT  IOTS.  t  TTS. IUT passes t  i IUT passes t pass / fail =

23. © Jan Tretmans 23 Variations on a Theme i ioco s     Straces(s) : out ( i after  )  out ( s after  ) i  ior s     ( L  {  } )* : out ( i after  )  out ( s after  ) i ioconf s     traces(s) : out ( i after  )  out ( s after  ) i ioco F s     F : out ( i after  )  out ( s after  ) i mioco smulti-channel ioco i uioco s universal ioco i wioco s non-input-enabled ioco i sioco ssymbolic ioco i (r)tioco s(real) timed tioco (Aalborg, Twente, Grenoble, Bordeaux,....) i ioco r srefinement ioco i hioco shybrid ioco...

24. © Jan Tretmans 24 ? money ? button1 ? button2 ! coffee ! tea test case fai l ! money ! button2 ? tea fai l ? coffee pass  n: int  [ n  35 ] -> [ n  50 ] -> with data model and hybrid and time c := 0 c < 10 c < 15 [ c  5 ] -> c := 0 d V t / dt = 3 d V c / dt = 2 V c := 0 [V c = 10 ] -> V t := 0 [V t = 15 ] -> ? Testing Transition Systems:StatusExtensions ?coin1 ?coin3 ?coin2 and action refinement

25. © Jan Tretmans 25 !x ?ok !err ?but !err ?but !ok ?ok ?er r !x ?err !y ?ok ?err ?ok ?err ?ok ?err ?but  !x i 1 ioco s 1 i 2 ioco s 2 ioco s 1 ||s 2 i 1 ||i 2 ok err but XyXy ok err but XyXy Component Based Testing  ?but !y ?but

26. © Jan Tretmans 26 Compositional Testing Component Based Testing i1i1 i2i2 s2s2 s1s1 ioco i 1 ioco s 1 i 2 ioco s 2 s 1 || s 2 i 1 || i 2 If s 1, s 2 input enabled - s 1, s 2  IOTS - then ioco is preserved !

27. © Jan Tretmans 27 Variations on a Theme: uioco ?a ?b !z ?b?a !y !x i ioco s     Straces(s) : out ( i after  )  out ( s 0 after  ) s0s0 s1s1 s2s2 out ( s 0 after ?b ) =  but ?b  Straces(s) : under-specification : anything allowed after ?b out ( s 0 after ?a ?a ) = { !x } and ?a ?a  Straces(s) but from s 2, ?a ?a is under-specified : anything allowed after ?a ?a ?

28. © Jan Tretmans 28 Variations on a Theme: uioco ?a ?b !z ?b?a !y !x s0s0 s1s1 s2s2 i uioco s     Utraces(s) : out ( i after  )  out ( s 0 after  ) Now s is under-specified in s 2 for ?a : anything is allowed. Utraces(s) = {   Straces (s) |   1 ?a  2 = ,  s': s  1 s'  s' ?a } ioco  uioco

29. © Jan Tretmans 29 Variations on a Theme: uioco LILU LILU  LI LI  ?a ?b !z ?b?a !y !x s0s0 s1s1 ?b  ?a  s2s2 i uioco s     Utraces(s) : out ( i after  )  out ( s 0 after  ) Alternatively, via chaos process  for under-specified inputs

30. © Jan Tretmans 30 Testing Components method invocations IUT component || IUT component method invocations methods invoked method call IUT component method returned method called method return IUT component method invocation

31. © Jan Tretmans 31 Testing Components tester method call IUT component method return method call method return L I = offered methods calls  used methods returns L U = offered methods returns  used methods calls specification s  LTS(L I, L U )

32. © Jan Tretmans 32 Testing Components tester method call IUT component method return method call method return Input-enabledness:  s of IUT,  ?a  L I : s ?a ? No ! ?

33. © Jan Tretmans 33 i uioco s = def   Utraces (s) : out (i after  )  out (s after  ) Correctness Implementation Relation wioco in (s after  ) = { a?  L I | s after  must a? } i wioco s = def   Utraces (s) : out (i after  )  out (s after  ) and in (i after  )  in (s after  ) s after  must a? =  s’ ( s  s’  s’ a? )

34. © Jan Tretmans 34 Formal Testing with Transition Systems exec : TESTS  IMPS   (OBS) gen : LTS   (TTS) T s  TTS s  LTS IUT  IMPS ioco i IUT  IOTS passes : IOTS  TTS  {pass,fail} Proof soundness and exhaustiveness:  i  IOTS. (  t  gen(s). i passes t )  i ioco s Test hypothesis :  IUT  IMP.  i IUT  IOTS.  t  TTS. IUT passes t  i IUT passes t pass / fail =

35. © Jan Tretmans 35 Test Assumption IUT input a?  L I output x?  L U quiescence  Sequencing of inputs, outputs, and  : Input-enabledness:  s of IUT,  ?a  L I : IUT  s ?a IUT behaves as an IOTS (input-enabled LTS)

36. © Jan Tretmans 36 Comparing Transition Systems Testing Equivalences S1S2S1S2 environment  Suppose an environment interacts with the systems:  the environment tests the system as black box by observing and actively controlling it;  the environment acts as a tester;  Two systems are equivalent if they pass the same tests.

37. © Jan Tretmans 37 Formal Testing : Test Assumption Test assumption :  IUT.  i IUT  MOD.  t  TEST. IUT passes t  i IUT passes t IUT i IUT test t

38. © Jan Tretmans 38 Completeness of Formal Testing IUT passes T s  def  t  T s. IUT passes t Test hypothesis :  t  TEST. IUT passes t  i IUT passes t Proof obligation :  i  MOD. (  t  T s. i passes t )  i imp s IUT passes T s  IUT confto s ? Definition : IUT confto s IUT confto s i IUT imp s   t  T s. i IUT passes t   t  T s. IUT passes t  IUT passes T s 

39. © Jan Tretmans 39 ! choc ?dub !tea ioco Test Assumption ! coffee ?dub !tea s More tests may be needed, starting in initial state:  meta-assumption: reliable restart

40. © Jan Tretmans 40 ?dub ! choc ?dub !tea ! choc ?dub !tea Alternative Test Assumption ioco Test:1. do ?dub 2. make core dump 3. make many copies of core dump 4. continue test with each copy An “Ambramsky”-test can distinguish them

41. © Jan Tretmans 41 Alternative Test Assumption ioco ?kwart ?dub ?kwart !tea !choc ?dub ?kwart ?dub ?kwart !tea ! choc With test ?dub.?kwart.undo you can distinguish them

42. © Jan Tretmans 42 Concluding  Testing can be formal, too (M.-C. Gaudel, TACAS'95)  Testing shall be formal, too  A test generation algorithm is not just another algorithm :  Proof of soundness and exhaustiveness  Definition of test assumption and implementation relation  For labelled transition systems :  ioco for expressing conformance between imp and spec  a sound and exhaustive test generation algorithm  tools generating and executing tests: TGV, TestGen, Agedis, TorX,....

43. © Jan Tretmans 43 Model based formal testing can improve the testing process : model is precise and unambiguous basis for testing  design errors found during validation of model longer, cheaper, more flexible, and provably correct tests  easier test maintenance and regression testing automatic test generation and execution  full automation : test generation + execution + analysis extra effort of modelling compensated by better tests Perspectives

44. © Jan Tretmans 44 Thank You