1. 1 Jan Tretmans jan.tretmans@esi.nl Embedded Systems Institute Eindhoven Radboud University Nijmegen Model-Based Testing

2. © Jan Tretmans 2 Model Based Testing Contents  Model-based testing  What and when  Formal model-based testing  Testing versus verification  Testing as a decision procedure  Test assumption, implementation relation, tests  Model-based testing with transition systems  Implementation relation ioco  Test generation for ioco  Test Tools  On-the-fly testing  Conclusions  The future, perspectives

3. © Jan Tretmans 3 Model Based Testing Contents  Model-based testing  What and when  Formal model-based testing  Testing versus verification  Testing as a decision procedure  Test assumption, implementation relation, tests  Model-based testing with transition systems  Implementation relation ioco  Test generation for ioco  Test Tools  On-the-fly testing  Conclusions  The future, perspectives

4. © Jan Tretmans 4 Towards Model Based Testing  Increase in complexity, and quest for higher quality software  testing effort grows exponentially with complexity  testing cannot keep pace with development  More abstraction  less detail  model based development  Checking quality  practice: testing - ad hoc, too late, expensive, lot of time  research: formal verification - proofs, model checking,.... with disappointing practical impact

5. © Jan Tretmans 5 Towards Model Based Testing  Model based testing has potential to combine  practice - testing  theory- formal methods  Model Based Testing :  testing with respect to a (formal) model / specification state model, pre/post, CSP, Promela, UML, Spec#,....  promises better, faster, cheaper testing:  algorithmic generation of tests and test oracles : tools  formal and unambiguous basis for testing  measuring the completeness of tests  maintenance of tests through model modification

6. © Jan Tretmans 6 Automated Model Based TestingAutomatedModel BasedTesting model spec IUT confto spec TTCN test cases pass fail test tool test generation tool test execution tool IUT passes tests IUT confto spec     soundexhaustive

7. © Jan Tretmans 7 A Formal Development Process informal requirements specification realization design code formalizable validation formal verification testing formal informal world formal world real world

8. © Jan Tretmans 8 Model Based Testing Contents  Model-based testing  What and when  Formal model-based testing  Testing versus verification  Testing as a decision procedure  Test assumption, implementation relation, tests  Model-based testing with transition systems  Implementation relation ioco  Test generation for ioco  Test Tools  On-the-fly testing  Conclusions  The future, perspectives

9. © Jan Tretmans 9 formal world real world Formal Verification specification s implementation i model m of i sat m modcheck s m sat s     completesound model checker m modcheck s assumption: m is valid model of i sat

10. © Jan Tretmans 10 formal world real world Formal Testing specification s implementation i confto IUT IUT passes T s IUT confto s  correctness of test generation ? IUT passes T s IUT fails T s test execution test generation test suite T S

11. © Jan Tretmans 11 Correctness of Formal Testing IUT confto s  IUT passes T s ?       i IUT imp s  i IUT passes' T s  i IUT  MOD.  Implies completeness of testing as a decision procedure  Based on test assumption sound   exhaustive

12. © Jan Tretmans 12 Formal Testing : Test Assumption Test assumption :  IUT.  i IUT  MOD.  t  TEST. IUT passes t  i IUT passes t IUT i IUT test t

13. © Jan Tretmans 13 Testing Addition Test a function adding numbers of two dice: int dadd ( int x, y ) for x,y  [1…6] Is the following a complete test suite? (1,1) (1,2)..... (1,6) (2,1) (2,2)..... (2,6)... (6,1) (6,2)..... (6,6)

14. © Jan Tretmans 14 Testing Addition Test a function adding numbers of two dice: int dadd ( int x, y ) for x,y  [1…6] The test suite (1,1) (1,2)..... (1,6) (2,1) (2,2)..... (2,6)... (6,1) (6,2)..... (6,6) is sound & exhaustive if  imp is such that it requires correct addition for x,y  [1…6] only;  the test assumption is that implementations can be modelled as functions : MOD = int int  int

15. © Jan Tretmans 15 Formal Testing passes : TEST  MODS  {pass,fail} gen : SPEC   (TEST) T s  TEST s  SPEC imp Proof soundness and exhaustivess:  i  MOD. i passes T s.  i imp s Test hypothesis :  IUT  IMP.  i IUT  MOD.  t  TEST. IUT passes t  i IUT passes t pass / fail Then: IUT is confto-correct  IUT passes the test i IUT  MODS

16. © Jan Tretmans 16 Approaches to Formal Testing  Finite State Machine  Pre/post-conditions  Labelled Transition Systems  Programs as Functions  Abstract Data Type testing ....... Labelled Transition Systems Programs as Functions

17. © Jan Tretmans 17 Testing Properties of Sequential Input/Output Programs  Specification: property over x and y  property(x,y) = x  0  |y  y - x|    Implementation is function i :: X  Y  Test set T  X  Tools like G  ST and QuickCheck generate thousands of tests by systematic traversal of all values of type X  But still: what is a "good" set ? IUT i(x) =  x x: real pre: x  0 y: real post: |y  y - x|  

18. © Jan Tretmans 18 Approaches to Formal Testing  Finite State Machine  Pre/post-conditions  Labelled Transition Systems  Programs as Functions  Abstract Data Type testing ....... Labelled Transition Systems Programs as Functions

19. © Jan Tretmans 19 Model Based Testing Contents  Model-based testing  What and when  Formal model-based testing  Testing versus verification  Testing as a decision procedure  Test assumption, implementation relation, tests  Model-based testing with transition systems  Implementation relation ioco  Test generation for ioco  Test Tools  On-the-fly testing  Conclusions  The future, perspectives

20. © Jan Tretmans 20 Formal Testing with Transition Systems exec : TESTS  IMPS   (OBS) gen : LTS   (TTS) T s  TTS s  LTS IUT  IMPS ioco i IUT  IOTS passes : IOTS  TTS  {pass,fail} Proof soundness and exhaustiveness:  i  IOTS. (  t  gen(s). i passes t )  i ioco s Test hypothesis :  IUT  IMP.  i IUT  IOTS.  t  TTS. IUT passes t  i IUT passes t pass / fail

21. © Jan Tretmans 21 Specifications: Labelled Transition Systems Labelled Transition System  S, L, T, s 0  ?coin ?button !alarm ?button !coffee states actions transitions T  S  (L  {  })  S initial state s 0  S

22. © Jan Tretmans 22 Models of Implementations: Input-Output Transition Systems L I = { ?dub, ?kwart } L U = { !coffee, !tea } Input-Output Transition Systems IOTS (L I, L U )  LTS (L I  L U ) IOTS is LTS with Input-Output and always enabled inputs: for all states s, for all inputs ?a  L I : ?dub ?kwart ?dub ! coffee ?kwart !tea S ?a

23. © Jan Tretmans 23 i ioco s = def   Straces (s) : out (i after  )  out (s after  ) Implementation Relation ioco Correctness expressed by ioco  IOTS  LTS : p  p=  !x  L U  {  }. p !x out ( P )= { !x  L U | p !x, p  P }  {  | p  p, p  P } Straces ( s )= {   (L  {  })* | s  } p after  = { p’ | p  p’ }

24. © Jan Tretmans 24 i ioco s = def   Straces (s) : out (i after  )  out (s after  ) Implementation Relation ioco Correctness expressed by ioco  IOTS  LTS : Intuition: i ioco-conforms to s, iff if i produces output x after trace , then s can produce x after  if i cannot produce any output after trace , then s cannot produce any output after  ( quiescence  )

25. © Jan Tretmans 25 Genealogy of ioco Labelled Transition Systems IOTS (IOA, IOSM, IOLTS) Testing Equivalences (Preorders) Refusal Equivalence (Preorder) Canonical Tester conf Quiescent Trace Preorder Repetitive Quiescent Trace Preorder (Suspension Preorder) ioco ioconf

26. © Jan Tretmans 26   Implementation Relation ioco out ( i after  ) = out ( i after ?dub ) = out ( i after ?dub.?dub ) = out ( i after ?dub.!coffee) = out ( i after ?kwart ) = out ( i after !coffee ) = out ( i after ?dub.!tea ) = out ( i after  ) = ! coffee ?dub ?kwart ?dub ?kwart i { }{ } { !coffee } { }{ } {  }{  }   { }{ } i ioco s = def   Straces (s) : out (i after  )  out (s after  )

27. © Jan Tretmans 27 !coffee ?dub i !coffee ?dub s !tea out (i after ?dub) = { !coffee }out (s after ?dub) = { !coffee, !tea } ioco Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  )

28. © Jan Tretmans 28 !coffee ?dub s !coffee ?dub i !tea ?dub out (i after ?dub) = { !coffee, !tea }out (s after ?dub) = { !coffee}  ioco Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  )

29. © Jan Tretmans 29 ioco ?dub ?kwart !coffee ?kwart i !tea !coffee ?dub s out (i after ?dub) = { !coffee } out (i after ?kwart) = { !tea } out (s after ?dub) = { !coffee } out (s after ?kwart) =  But ?kwart  Straces ( s ) Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  )

30. © Jan Tretmans 30 !coffee ?dub i ?kwart ?dub ?kwart ?dub ?kwart out (s after ?kwart) = { !tea } out (i after ?kwart) = {  } ioco Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  ) s ?dub !coffee ?kwart !tea

31. © Jan Tretmans 31 out (s after ?dub) = { !coffee } out (i after ?dub) = { , !coffee } ioco ?dub !coffee ?dub i !coffee ?dub s Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  )

32. © Jan Tretmans 32 out (s after ?dub) = { , !coffee } out (i after ?dub) = { , !coffee } ioco !coffee ?dub  s Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  ) ?dub !coffee ?dub i

33. © Jan Tretmans 33 out (i after ?dub.?dub) = out (s after ?dub.?dub) = { !tea, !coffee } i ioco s Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  ) i ?dub !tea ?dub !coffee ?dub s !coffee ?dub !tea ?dub !tea s ioco i out (i after ?dub.   ?dub) = { !coffee }  out (s after ?dub. .?dub) = { !tea, !coffee }

34. © Jan Tretmans 34 Test Cases  labels in L  {  }  tree-structured  finite, deterministic  final states pass and fail  from each state  pass, fail :  either one input !a  or all outputs ?x and  ?coffee !dub !kwart ?tea ?coffee ?tea  !dub  pass fail pass Test case t  TTS TTS - Test Transition System :

35. © Jan Tretmans 35 ioco Test Generation i ioco s = def   Straces (s) : out (i after  )  out (s after  )  out (s after  ) = { !x, !y,  }  s !x !y  i !x  !z out (i after  ) = { !x, !z,  } out (test after  ) = L U  {  }  pass fail test ?x  ?y ?z pass

36. © Jan Tretmans 36 Algorithm To generate a test case from transition system specification s 0 compute T(S), with S a set of states, and initially S = s 0 after  ; 1end test case pass For T(S), apply the following recursively, non-deterministically: 2supply input !a T ( S after ?a   ) ioco Test Generation Algorithm allowed outputs or  : !x  out ( S ) forbidden outputs or  : !y  out ( S ) 3observe output fail T ( S after !x ) fail allowed outputsforbidden outputs ?y  ?x

37. © Jan Tretmans 37 ?coffee failpass  ?tea ?choc fail Test Generation Example test ?coffee fail  ?tea ?choc pass !dub s !tea ?dub !coffee

38. © Jan Tretmans 38 Test Execution Example Two test runs : t  i dub tea pass  i' fail  i'' t  i dub choc i fails t ! choc ?dub !tea i ?coffee failpass  ?tea ?choc fail test ?coffee fail  ?tea ?choc pass !dub i'i''

39. © Jan Tretmans 39 ?coffee  ?tea pass fail ?coffee pass fail  ?tea    Test Generation Example s ?dub !coffee ?dub test !dub

40. © Jan Tretmans 40 Test Execution Example Two test runs : t  i dub  pass  i' t  i dub coffee  i passes t ?coffee  ?tea pass fail ?coffee pass fail  ?tea test !dub i ?dub !coffee ?dub

41. © Jan Tretmans 41 Completeness of ioco Test Generation For every test t generated with algorithm we have:  Soundness : t will never fail with correct implementation i ioco s implies i passes t  Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s implies  t : i fails t

42. © Jan Tretmans 42 Formal Testing with Transition Systems exec : TESTS  IMPS   (OBS) gen : LTS   (TTS) T s  TTS s  LTS IUT  IMPS ioco i IUT  IOTS passes : IOTS  TTS  {pass,fail} Proof soundness and exhaustiveness:  i  IOTS. (  t  gen(s). i passes t )  i ioco s Test hypothesis :  IUT  IMP.  i IUT  IOTS.  t  TTS. IUT passes t  i IUT passes t pass / fail

43. © Jan Tretmans 43 Compositional Testing Component Based Testing i1i1 i2i2 s2s2 s1s1 ioco i 1 ioco s 1 i 2 ioco s 2 s 1 || s 2 i 1 || i 2 If s 1, s 2 input enabled - s 1, s 2  IOTS - then ioco is preserved !

44. © Jan Tretmans 44 Model Based Testing Contents  Model-based testing  What and when  Formal model-based testing  Testing versus verification  Testing as a decision procedure  Test assumption, implementation relation, tests  Model-based testing with transition systems  Implementation relation ioco  Test generation for ioco  Test Tools  On-the-fly testing  Conclusions  The future, perspectives

45. © Jan Tretmans 45 Some Model Based Testing Approaches and Tools  Agatha  Agedis  Autolink  Cooper  G  st  Gotcha  Leirios  Phact/The Kit  QuickCheck  RT-Tester  SaMsTaG  Spec#/SpecExplorer  Statemate MAGNUM ATG  STG  TestGen (Stirling)  TestGen (INT)  TestComposer  TGV  TorX  T-Uppaal  Tveda ...... TorX

46. © Jan Tretmans 46 A Tool for Transition Systems Testing: TorX  On-the-fly test generation and test execution  Implementation relation: ioco  Mainly applicable to reactive systems / state based systems;  specification languages: LOTOS, Promela, FSP, Automata TorX IUT observe output offer input next input specification check output pass fail inconclusive user: manual automatic

47. © Jan Tretmans 47 TorX Case Study Lessons  model construction  difficult: missing or bad specs  leads to detection of design errors  not supported by integrated environment yet  research on test based modelling  adapter/test environment  development is cumbersome  specific for each system  longer and more flexible tests  full automation : test generation + execution + analysis  no notion of test selection or specification coverage yet  only random coverage or user guided test purposes

48. © Jan Tretmans 48 ? money ? button1 ? button2 ! coffee ! tea test case fai l ! money ! button2 ? tea fai l ? coffee pass  n: int  [ n  35 ] -> [ n  50 ] -> with data model and time and hybrid c := 0 c < 10 c < 15 [ c  5 ] -> c := 0 d V t / dt = 3 d V c / dt = 2 V c := 0 [V c = 10 ] -> V t := 0 [V t = 15 ] -> ?coin1 ?coin3 ?coin2 and action refinement ? Testing Transition Systems:StatusExtensions

49. © Jan Tretmans 49 Concluding  Testing can be formal, too (M.-C. Gaudel, TACAS'95)  Testing shall be formal, too  A test generation algorithm is not just another algorithm :  Proof of soundness and exhaustiveness  Definition of test assumption and implementation relation  For labelled transition systems :  ioco for expressing conformance between imp and spec  a sound and exhaustive test generation algorithm  tools generating and executing tests: TGV, TestGen, Agedis, TorX,....

50. © Jan Tretmans 50 Model based formal testing can improve the testing process : model is precise and unambiguous basis for testing  design errors found during validation of model longer, cheaper, more flexible, and provably correct tests  easier test maintenance and regression testing Automatic test generation and execution  full automation : test generation + execution + analysis Extra effort of modelling compensated by better tests Perspectives