1. Model-Based Testing Ed Brinksma University of Twente Dept. of Computer Science Formal Methods & Tools group Enschede The Netherlands ARTIST2 Summer School Nässlingen

2. October 1, 2005ARTIST2 Summer School 2 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

3. October 1, 2005ARTIST2 Summer School 3 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

4. October 1, 2005ARTIST2 Summer School 4 Practical problems of testing Testing is: limportant lmuch practiced l30% - 50% of project effort lexpensive ltime critical lnot constructive (but sadistic?) But also: lad-hoc, manual, error-prone lhardly theory / research lno attention in curricula lnot cool : “if you’re a bad programmer you might be a tester” Attitude is changing: lmore awareness lmore professional Improvements possible with formal methods ! ?

5. October 1, 2005ARTIST2 Summer School 5 Types of Testing unit integration system performance robustness functional behaviour white boxblack box Level Accessibility Aspect usability reliability

6. October 1, 2005ARTIST2 Summer School 6 Test Automation Traditional test automation = tools to execute and manage test cases specification test tool implementation under test pass fail TTCN test cases Why not generate test automatically?!

7. October 1, 2005ARTIST2 Summer School 7 formal world concrete world Verification is only as good as the validity of the model on which it is based Verification and Testing Verification : lformal manipulation lprove properties lperformed on model Testing : lexperimentation lshow error lconcrete system Testing can only show the presence of errors, not their absence

8. October 1, 2005ARTIST2 Summer School 8 Testing with Formal Methods lTesting with respect to a formal specification lPrecise, formal definition of correctness : good and unambiguous basis for testing lFormal validation of tests lAlgorithmic derivation of tests : tools for automatic test generation lAllows to define measures expressing coverage and quality of testing

9. October 1, 2005ARTIST2 Summer School 9 Challenges of Testing Theory lInfinity of testing: ntoo many possible input combinations -- infinite breadth ntoo many possible input sequences -- infinite depth ntoo many invalid and unexpected inputs lExhaustive testing never possible: nwhen to stop testing ? nhow to invent effective and efficient test cases with high probability of detecting errors ? lOptimization problem of testing yield and invested effort nusually stop when time is over......

10. October 1, 2005ARTIST2 Summer School 10 test execution pass / fail Formal Testing test generation test suite T S specification S implementatio n i correctness criterion implementation relation imp i passes T s i imps     soundexhaustive

11. October 1, 2005ARTIST2 Summer School 11 Formal Testing : Conformance s  SPECS Specification IUT Implementation under Test IUT is concrete, physical object Model the physical world But IUT is black box ! ? Assume that model i IUT exists specification S implementatio n IUT correctness criterion IUT conforms-to s

12. October 1, 2005ARTIST2 Summer School 12 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

13. October 1, 2005ARTIST2 Summer School 13 Testing Preorders on Transition Systems implementation i specification s environment e    ? ? ?  i  s   e  Env. obs ( e, i )  obs (e, s ) For all environments e all observations of an implementation i in e should be explained by observations of the specification s in e.

14. October 1, 2005ARTIST2 Summer School 14 Classical Testing Preorder  LTS(L) Deadlocks(e||s) i  te s   e  E. obs ( e, i )  obs (e, s ) implementation i specification s environment e  te

15. October 1, 2005ARTIST2 Summer School 15 Classical Testing Preorder implementation i specification s environment e  te i  te s   e  LTS(L).    L*. e ||i deadlocks after   e ||s deadlocks after   FP (i)  FP (s) FP (p) = {  , A  | p afer  refuses A } i  te s   e  LTS(L).    L*. {  | e || i deadlocks after  }  {  | e||s deadlocks after  }

16. October 1, 2005ARTIST2 Summer School 16 Quirky Coffee Machine [Langerak] Can we distinguish between these machines? coin tea coffee bang coffee tea coin tea coffee bang coffee tea  te They are testing equivalent!

17. October 1, 2005ARTIST2 Summer School 17 Refusal Preorder i  rf s   e  E. obs ( e, i )  obs (e, s ) implementation i specification s environment e  rf  LTS ( L  {δ}) Deadlocks δ (e||i) e observes with δ deadlock on all alternative actions Deadlocks δ (e||i) = {  (L  {δ})* | e||i deadlocks after  }

18. October 1, 2005ARTIST2 Summer School 18 Quirky Coffee Machine Revisited coin tea coffee bang coffee tea coin tea coffee bang coffee tea  te  rf  coin coffee bang tester δ only enabled if coffee is not

19. October 1, 2005ARTIST2 Summer School 19 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

20. October 1, 2005ARTIST2 Summer School 20 I/O Transition Systems ltesting actions are usually directed, i.e. there are inputs and outputs L=L in  L out with L in  L out =  lsystems can always accept all inputs (input enabledness) for all states s, for all a  L in s  ltesters are I/O systems noutput (stimulus) is input for the SUT ninput (response) is output of the SUT a

21. October 1, 2005ARTIST2 Summer School 21 Quiescence lBecause of input enabledness S||T deadlocks iff T produces no stimuli and S no responses. This is known as quiescence lObserving quiescence leads to two implementation relations for I/O systems I and S: 1. I  iote S iff for all I/O testers T: nDeadlocks(I||T)  Deadlocks(S||T) (quiescence) 2. I  iorf S iff for all I/O testers T: nDeadlocks δ (I||T)  Deadlocks δ (S||T) (repetitive quiescence)

22. October 1, 2005ARTIST2 Summer School 22 Input-Output QCM coin? tea? coffee? bang? coffee! tea? coffee? tea ! coffee? tea? tea ! states must be saturated with input loops for input enabledness  iote  iorf coin! coffee! coffee?  bang! coffee ! coffee? coffee! quiescen t states

23. October 1, 2005ARTIST2 Summer School 23 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

24. October 1, 2005ARTIST2 Summer School 24 Implementation Relation ioco i  iorf s   I/O tests T : Deadlocks δ (i||T)  Deadlocks δ (s||T)    ( L  {  } )*: out ( i after  )  out ( s after  ) To allow under-specification we restrict the set of traces: i ioco s    Traces δ ( s ) : out ( i after  )  out ( s after  ) By adding a transition p  p to every quiescent state of a system we treat quiescence as an observable (synchronizable) action: δ

25. October 1, 2005ARTIST2 Summer School 25 i ioco s = def   Traces δ ( s ) : out (i after  )  out (s after  ) Implementation Relation ioco Correctness expressed by implementation relation ioco: Intuition: i ioco-conforms to s, iff if i produces output x after trace , then s can produce x after  if i cannot produce any output after trace , then s cannot produce any output after  ( quiescence  )

26. October 1, 2005ARTIST2 Summer School 26   Implementation Relation ioco out ( i after  ) = out ( i after ?dub ) = out ( i after ?dub.?dub ) = out ( i after ?dub.!coffee) = out ( i after ?kwart ) = out ( i after !coffee ) = out ( i after ?dub.!tea ) = out ( i after  ) = ! coffee ?dub ?kwart ?dub ?kwart i { }{ } { !coffee } { }{ } {  }   { }{ } i ioco s = def   Straces (s) : out (i after  )  out (s after  )

27. October 1, 2005ARTIST2 Summer School 27 !coffee ?dub i !coffee ?dub s !tea out (i after ?dub) = { !coffee }out (s after ?dub) = { !coffee, !tea } ioco Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  )

28. October 1, 2005ARTIST2 Summer School 28 !coffee ?dub s !coffee ?dub i !tea ?dub out (i after ?dub) = { !coffee, !tea }out (s after ?dub) = { !coffee}  ioco Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  )

29. October 1, 2005ARTIST2 Summer School 29 ioco ?dub ?kwart !coffee ?kwart i !tea !coffee ?dub s out (i after ?dub) = { !coffee } out (i after ?kwart) = { !tea } out (s after ?dub) = { !coffee } out (s after ?kwart) =  But ?kwart  Traces δ ( s ) Implementation Relation ioco i ioco s = def   Straces (s) : out (i after  )  out (s after  )

30. October 1, 2005ARTIST2 Summer School 30 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

31. October 1, 2005ARTIST2 Summer School 31 test execution pass / fail Formal Testing test generation test suite T S specification S implementatio n i correctness criterion implementation relation ioco i passes T s i iocos  ?

32. October 1, 2005ARTIST2 Summer School 32 Test Cases nlabels in L  {  } ntree-structured nfinite, deterministic nfinal states pass and fail nfrom each state  pass, fail l either one input i? l or all outputs o! and  coffee! coin? tea! coffee! tea!  coin?  pass fail pass Test case t  TTS TTS - Test Transition System :

33. October 1, 2005ARTIST2 Summer School 33 Test Cases test case t !coin !coin ; Start timer1 ?teafail ?timer1fail ?coffee !coin ; Start timer2 ?teapass ?timer2pass ?coffeefail coffee! coin? tea! coffee! tea!  coin?  pass fail pass

34. October 1, 2005ARTIST2 Summer School 34 Algorithm To generate a test case t(S) from a transition system specification with S set of states ( initially S = {s 0 } ) 1end test case PASS Apply the following steps recursively, non-deterministically 2supply input supply i? t(S after i?) Test Generation Algorithm 3observe output FAIL t(S after o!) FAIL allowed outputs o! forbidden outputs o! 

35. October 1, 2005ARTIST2 Summer School 35 test ?-2 ?2 PASS otherwise FAIL ?-3 PASS otherwise ?3 FAIL To cope with non-deterministic behaviour, tests are not linear traces, but trees Test Generation Example Equation solver for y 2 =x specification ? x (x >= 0) !  x ? x (x < 0) ! -  x !4 !9

36. October 1, 2005ARTIST2 Summer School 36 Validity of Test Generation For every test t generated with algorithm : Soundness : t will never fail with correct implementation i ioco s implies i passes t Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s implies  t : i fails t

37. October 1, 2005ARTIST2 Summer School 37 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

38. October 1, 2005ARTIST2 Summer School 38 Formal Testing with Transition Systems t :  (traces)  {fail,pass} traces der : LTS   (TTS) T s  TTS s  LTS ioco i IUT  IOTS pass fail obs : TTS  IOTS   (traces)

39. October 1, 2005ARTIST2 Summer School 39 Test Generation Tools for ioco lTVEDA (CNET - France Telecom) nderives TTCN tests from single process SDL specification ndeveloped from practical experiences nimplementation relation R1  ioco lTGV (IRISA - Rennes) nderives tests in TTCN from LOTOS or SDL nuses test purposes to guide test derivation nimplementation relation: unfair extension of ioco lTestComposer nCombination of TVEDA and TGV in ObjectGeode lTestGen (Stirling) nTest generation for hardware validation lTorX (Côte de Resyste)

40. October 1, 2005ARTIST2 Summer School 40 A Test Tool : TorX lOn-the-fly test generation and test execution lImplementation relation: ioco lSpecification languages: LOTOS, Promela, FSP, Automata TorX IUT observe output offer input next input specification check output pass fail inconclusive user: manual automatic

41. October 1, 2005ARTIST2 Summer School 41 TorX Tool Architecture explorerprimerdriveradapter IUT spec. states transitions abstract actions concrete actions specification text TorX IUT specification

42. October 1, 2005ARTIST2 Summer School 42 TTCN test taal batch test execution batch test generation TTCN test taal on the fly On-the-Fly  Batch Testing explorerprimerdriveradapter IUT spec.

43. October 1, 2005ARTIST2 Summer School 43 explorerprimerdriveradapterIUT bits bytes states transitions abstract actions transition ? x (x >= 0) !  x ? x (x < 0) ! -  x specification implementation ? x (x >= 0) !  x ? x (x < 0) ? x On-the-Fly Testing Concrete action ! 00001001 New menu ! x (x < 0) ! x (x >= 0) Abstract action ! 9 Abstract action ? 3 Choice ! 9 Concrete action ? 00000011 Action ? 3 Choice ! -1 New menu ! x (x < 0) ! x (x >= 0) Check ? 3 Abstract action ! -1 Concrete action ! 11111111 Concrete action ? (timeout) Abstract action ? (quiescence) Action ? (quiescence) Check ? (quiescence) New menu ! x (x < 0) ! x (x >= 0) spec

44. October 1, 2005ARTIST2 Summer School 44 TorX

45. October 1, 2005ARTIST2 Summer School 45 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

46. October 1, 2005ARTIST2 Summer School 46 TorX Case Studies lConference Protocol lEasyLink TV-VCR protocol lCell Broadcast Centre component lRoad Toll Payment Box protocol lV5.1 Access Network protocol lEasy Mail Melder lFTP Client l“Oosterschelde” storm surge barrier-control lTANGRAM: testing VLSI lithography machine academic Philips CMG Interpay Lucent CMG academic CMG ASML

47. October 1, 2005ARTIST2 Summer School 47 Interpay Highway Tolling System

48. October 1, 2005ARTIST2 Summer School 48 Highway Tolling Protocol Characteristics : lSimple protocol lParallellism : many cars at the same time lEncryption lSystem passed traditional testing phase

49. October 1, 2005ARTIST2 Summer School 49 Payment Box (PB) Road Side Equipment Onboard Unit UDP/IP Wireless Highway Tolling System

50. October 1, 2005ARTIST2 Summer School 50 Test Context ObuSim spec PB + ObuSim + TCP/IP + UDP/IP Payment Box TCP/IP TorX Highway Tolling: Test Architecture PCO SUT UDP/IP IAP

51. October 1, 2005ARTIST2 Summer School 51 Highway Tolling: Results lTest results : n1 error during validation (design error) n1 error during testing (coding error) lAutomated testing : nbeneficial: high volume and reliability nmany and long tests executed ( > 50,000 test events ) nvery flexible: adaptation and many configurations lReal-time : ninterference computation time on-the-fly testing ninterference quiescence and time-outs lStep ahead in formal testing of realistic systems

52. October 1, 2005ARTIST2 Summer School 52 Contents lintroduction & background ltesting pre-orders linput/output & quiescence lioco implementation relation ltest generation lTorX ltest case study lreal-time testing

53. October 1, 2005ARTIST2 Summer School 53 RT TorX Hacking Approaches 1.Ignore RT functionality: ltest pure functional behaviour lanalyse timing requirements using TorX log files & assumed timing constraints 2.Add timestamps to observations ladapter adds timestamps to observations when they are made and passed on to the driver ltimestamps are used to analyse TorX log files 3.Add timestamps to stimuli & observations ladapter add timestamps to observations when they are made and passed on to the driver ladapter adds timestamps to stimuli when they are applied and returned to the driver lanalysis: a.timing error logging: observed errors are written to TorX log file b.timing error failure: observed errors cause fail verdict of test case

54. October 1, 2005ARTIST2 Summer School 54 Real-time Testing and I/O Systems lcan the notion of repetitive quiescence be combined with real-time testing? lis there a well-defined and useful conformance relation that allows sound and (relative) complete test derivation? lcan the TorX test tool be adapted to support Real- timed conformance testing?

55. October 1, 2005ARTIST2 Summer School 55 Do We Still Need Quiescence? coin? tea? coffee? bang? coffee! tea? coffee? tea ! coffee? tea? tea ! Yes! the example processes should also be distinct in a real-time context coffee!

56. October 1, 2005ARTIST2 Summer School 56 Real-Time and Quiescence ls is quiescent iff: for no output action a and delay d: s  lspecial transitions: s  s for every quiescent system state s ltesters observing quiescence take time: Test M : set of test processes having only δ (M)- actions to observe quiescence lassume that implementations are M-quiescent: for all reachable states s and s’: if s  s’ then s’ is quiescent a(d) δ  (M)

57. October 1, 2005ARTIST2 Summer School 57 Real-Time and Quiescence i tioco M s    Traces δ(M) ( s ) : out M ( i after  )  out M ( s after  ) i  tiorf s   T  Test M : Deadlocks δ (i||T)  Deadlocks δ (s||T)    ( L  {  (M) } )*: out M ( i after  )  out M ( s after  ) M

58. October 1, 2005ARTIST2 Summer School 58 Properties 1.for all M 1  M 2 : i  tiorf s implies i  tiorf s 2.for all time-independent i,s and M 1,M 2  0 i  tiorf s iff i  tiorf s iff i  iorf s M1M1 M2M2 M1M1 M2M2

59. October 1, 2005ARTIST2 Summer School 59 A limitation states are saturated with input loops that reset the clocks x x x=k tea ! x=k coffee! xkxk coin? tea? coffee? tea? coffee? xkxk x=k coffee! x=k tea ! x<M bang? x<M bang? x  M bang? x  M bang? xkxkxkxk xx this process cannot be distinguished from the next this process cannot be distinguished from the previous

60. October 1, 2005ARTIST2 Summer School 60 Test Cases nlabels in L  {  }, G(d) ntree-structured nfinite, deterministic nfinal states pass and fail nfrom each state  pass, fail lchoose an input i? and a time k and wait for the time k accepting all outputs o! and after k time unit provide input i? lor wait for time M accepting all outputs o! and  off! x=5 x:=0 on? x:=0 off! x<5 off!  x=M fail pass Test case t  TTA TTA – Test Timed Automata : xMxM  xMxM xkxk x:= 0 off! fail

61. October 1, 2005ARTIST2 Summer School 61 To generate a test case t(S) from a timed transition system specification with S set of states ( initially S = {s 0 } ) 1. end test case PASS apply the following steps recursively, non-deterministically Timed test Generation Algorithm allowed outputs o j ! after d time-units 2. choose k  (0, M ) and input μ FAIL forbidden outputs o i ! after d’ time-units o1!o1! x=d n x=d 1 x=d’ n’ x=k x  k tμtμ t1t1 tntn x:=0 x=d’ 1 o n’ ! μ?μ? o1!o1! on!on! allowed outputs o j ! after d time-units 3. wait for observing possible output FAIL forbidden outputs o i ! after d’ time-units  x=d’ 1 x=d n x=d 1 x=d’ n’ x=M x  M tδtδ t1t1 tntn x:=0 o1!o1! o n’ ! o1!o1! on!on! can be calculated effectively only for subclasses of timed transition systems!

62. October 1, 2005ARTIST2 Summer School 62 Example fail m? x1x1 x=1 x:=0 x=M x:=0 c? x=1 x:=0 c? x=1 x:=0 b? x=1 x:=0 fail pass fail x1x1 xMxM x1x1 x1x1 xMxM m? t? c? b? c! t? c? t! c? t? t! c! spec: impl: M=k :test m? t? c? b? t! c? t! t? c! x<k δ c! fail t! δ pass t! x=M δ

63. October 1, 2005ARTIST2 Summer School 63 Soundness & Completeness lthe non-timed generation algorithm can be shown to generate only sound real-time test cases ltest generation is complete for every erroneous trace it can generate a test that exposes it ltest generation is not limit complete because of continuous time there are uncountably many timed error traces and only countably many test are generated by repeated runs ltest generation is almost limit complete repeated test geration runs will eventually generate a test case that will expose one of the non-spurious errors of a non-conforming implementation non-spurious errors = errors with a positive probability of occurring

64. October 1, 2005ARTIST2 Summer School 64 Current Work lExtension of the framework nM as a function of the specification state/output channel nintegration with symbolic data generation ntest action refinement nrobustness & tolerance in real-time testing lExtending TorX environment using CORBA IDL ngenerate abstract TorX actions ngenerate TTCN-3 signatures ngenerate adapter code lPractical application nTANGRAM project: testing control software for VLSI lithography machines (ASML) n smooth transition between timed & untimed testing

65. October 1, 2005ARTIST2 Summer School 65 Future Work lstochastic systems lquality of service lhybrid systems lcoverage measures lintegration white/black box spectrum l...

66. October 1, 2005ARTIST2 Summer School 66 For more information fmt.cs.utwente.nl/research/testing