1. Test Generation for Input/Output Transition Systems Ed Brinksma Course 2004

2. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 2 Test Cases nlabels in L  {  } ntree-structured nfinite, deterministic nfinal states pass and fail nfrom each state  pass, fail : leither one input !a lor all outputs ?x and  ?coffee !dub !kwart ?tea ?coffee ?tea  !dub  pass fail pass Test case t  TTS TTS - Test Transition System :

3. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 3 Test Cases ?coffee !dub !kwart ?tea ?coffee ?tea  !dub  pass fail pass test case t !dub !kwart ; Start timer1 ?teafail ?timer1fail ?coffee !dub ; Start timer1 ?teapass ?timer2pass ?coffeefail

4. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 4 Observations and Verdicts OBS  ( L  {  } )* obs  obs : TTS  IOTS   ( ( L  {  } )* ) obs (t,i) = {   ( L  {  } )* | t  i  pass  i' or t  i  fail  i' } t  t :  ( ( L  {  } )* )  { fail, pass } t (O)=passif    O. t  pass =failotherwise

5. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 5 Test Generation Look for test derivation algorithm der : LTS   (TTS) such that der is sound and exhaustive:  i  IOTS. (  t  der ( s ). t ( obs ( t, i ) ) = pass )  i ioco s i ioco s = def   Straces (s) : out (i after  )  out (s after  ) obs (t,i) = {   (L  {  })* | t  i  pass  i' or t  i  fail  i' } t (O) =passif    O. t  pass =failotherwise

6. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 6 Test Generation i ioco s = def   Straces (s) : out (i after  )  out (s after  ) out (s after  ) = { !x, !y } s !x  !y i !x  !z out (i after  ) = { !x, !z } out (test after  ) = L U pass fail test ?x  ?y ?z pass

7. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 7 Test Generation i ioco s = def   Straces (s) : out (i after  )  out (s after  )  out (s after  ) = { !x, !y,  }  s !x !y  i !x  !z out (i after  ) = { !x, !z,  } out (test after  ) = L U  {  }  pass fail test ?x  ?y ?z pass

8. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 8 Algorithm To generate a test case t(S) from a transition system specification with S, with S set of states ( initially S = s 0 after  ) 1end test case pass Apply the following steps recursively, non-deterministically: 2supply input !a t ( S after ?a   ) Test Generation Algorithm allowed outputs or  : !x  out ( S ) forbidden outputs or  : !y  out ( S ) 3observe output fail t ( S after !x ) fail allowed outputsforbidden outputs ?y  ?x

9. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 9 ?coffee  ?tea pass fail ?coffee pass fail  ?tea    Test Generation Example s ?dub !coffee ?dub test !dub

10. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 10 Test Execution lt  i equal to t || i, with extra rule : ift t' and i thent  i t'  i    ?coffee ?tea ?coffee ?tea  !dub  pass fail pass t !coffee !tea ?dub i Two test runs : t  i  dub tea pass  i' fail  i' t  i  dub coffee i fails t  Test run : t  i  pass  i' or t  i  fail  i'

11. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 11 Test Execution Example Two test runs : t  i dub  pass  i' t  i dub coffee  i ?dub !coffee ?dub i passes t ?coffee  ?tea pass fail ?coffee pass fail  ?tea test !dub

12. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 12 test ?-2 ?2 pass otherwise fail To cope with non-deterministic behaviour, tests are not linear traces, but trees Test Generation Example Equation solver for y 2 =x specification ? x (x >= 0) !  x ? x (x < 0) ! -  x !4 pass ?-3 otherwise ?3 fail !9

13. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 13 ?-3 pass otherwise ?3 fail ?-2 ?2 pass otherwise fail ? x (x >= 0) !  x ? x (x < 0) ! -  x ? y implementationtest    !9 !4 Test Execution Example

14. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 14 Validity of Test Generation For every test t generated with algorithm we have: Soundness : t will never fail with correct implementation i ioco s implies i passes t Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s implies  t : i fails t

15. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 15 Validity of Test Generation Theorem: lSoundness : i ioco s implies  t  der ( s ). t ( obs ( t, i ) ) = pass lExhaustiveness :  t  der ( s ). t ( obs ( t, i ) ) = pass implies i ioco s

16. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 16 ?coffee  ?tea pass fail pass fail    s ?dub !coffee ?dub test Soundness and Exhaustiveness /// fail !dub Still sound ! Not sound anymore ! /// pass fail ?coffee  ?tea

17. © Ed Brinksma/Jan Tretmans TT 2004, Test Derivation 17 Formal Testing with Transition Systems t :  (traces)  {fail,pass} exec : TESTS  IMPS   (OBS) traces der : LTS   (TTS) T s  TTS s  LTS IUT  IMPS ioco i IUT  IOTS pass fail obs : TTS  IOTS   (traces) Soundness and exhaustivess proved:  i  IOTS. (  t  der(s). t (obs(t,i)) = pass )  i ioco s Test hypothesis :  IUT  IMPS.  i IUT  IOTS.  t  TTS. exec(t, IUT ) = obs(t,i IUT )