Testing with Formal Methods Ed Brinksma course 2004 A Formal Framework

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 2 Software Testing Testing is: limportant lmuch practiced l30% - 50% of project effort lexpensive ltime critical lnot constructive (but sadistic?) But also: lad-hoc, manual, error-prone lhardly theory / research lno attention in curricula lnot cool : “if you’re a bad programmer you might be a tester” Attitude is changing: lmore awareness lmore professional Improvements possible with formal methods ! ?

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 3 Model Based Testing with Formal Methods Claims : lFormal specifications are nmore precise, nmore complete, nunambiguous lFormal models can be formally verified lFormal models allow automatic generation of tests

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 4 Types of Testing unit integration system performance robustness functional behaviour white boxblack box Level of detail Accessibility Characteristics usability reliability module security

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 5 Formal Methods lUse mathematics to model relevant parts of software lPrecise, formal semantics: no room for ambiguity or misinterpretation lAllow formal validation and reasoning about systems lAmenable to tools: automation lExamples: nZ nTemporal Logic nFirst order logic SDL Specification and Description Language nLOTOS nPromela nLabelled Transition Systems nFinite state machine nProcess algebra n......

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 6 Formal methods: lproving properties lresearch lsound theories l“clean” Testing : ltrial & error lpractice lheuristics l“dirty hands” Testing & Formal Methods A Perfect Couple ? “Testing is not necessary after formal verification” “Testing can only detect the presence of errors, not their absence” “Formal methods are toys for boys” “Formal methods have extreme potential - but not for my project”

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 7 Testing with Formal Methods lTesting with respect to a formal specification lPrecise, formal definition of correctness : good and unambiguous basis for testing lFormal validation of tests lAlgorithmic derivation of tests : tools for automatic test generation lAllows to define measures expressing coverage and quality of testing

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 8 Testing & Formal Methods Claims : lCombining the “mathematically clean” world of formal methods and the “dirty hands” world of testing lTesting and formal methods are both necessary in software development lFormal methods improve the testing process lFormal testing stimulates the use of formal methods

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 9 Goal: Testing functional behaviour of black-box implementation with respect to specification in formal language based on formal definition of conformance Specification Based Functional Testing with Formal Methods formal model system implementation formal testing

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 10 test tool implementation under test pass fail Test Automation Traditional test automation = tools to execute and manage test cases  Why not generate tests automatically ? specification TTCN test cases

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 11 test execution pass / fail Formal Testing test generation test suite T S specification S implementatio n i correctness criterion implementation relation i passes T s i ioco s     soundexhaustive

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 12 Formal relations lconforms-to  IMPS  SPECS niut conforms-to spec: implemetation under test obeys specification nhard: relates formal specification and real world system limp  MODS  SPECS ni IUT imp spec: model of the iut obeys specification nrelates formal objects ltest hypothesis: iut conforms-to spec  i IUT imp spec liut passes t = def v ( EXEC ( t, iut )) = pass nexecute a test: EXEC: TESTS  IMPS  P OBS nverdict function: v: P OBS  { pass, fail } niut passes T s   t  T s. iut passes t

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 13 Formal Testing : Conformance s  SPECS Specification IUT Implementation under Test IUT is concrete, physical object Model the physical world But IUT is black box ! ? Assume that model i IUT exists specification S implementatio n IUT correctness criterion IUT conforms-to s

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 14 Formal Testing : Conformance specification S implementatio n i IUT formal correctness criterion i IUT imp s s  SPECSSpecification i IUT  MODSmodel of IUT Test hypothesis : each concrete IUT can be modelled by some i IUT  MODS Conformance : i IUT imp s i IUT is not known ; testing to learn about i IUT

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 15 Conformance testing lconforms-to  IMPS  SPECS liut passes t = def v ( EXEC ( t, iut )) = pass niut passes T s   t  T s. iut passes t lconformance: ncomplete:iut conforms-to spec  iut passes T s nsound:iut conforms-to spec  iut passes T s nexhaustiveness:iut conforms-to spec  iut passes T s lioco: input output conformance

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 16 Formal Testing exec : TESTS  IMPS   (OBS) der : SPECS   (TESTS) T s  TESTS s  SPECS IUT  IMPS imp i IUT  MODS obs : TESTS  MODS   (OBS) t :  (OBS)  {fail,pass} OBS pass fail Proof soundness and exhaustivess:  i  MODS. (  t  der(s). t (obs(t,i)) = pass )  i imp s Test hypothesis :  IUT  IMPS.  i IUT  MODS.  t  TESTS. exec(t, IUT ) = obs(t,i IUT )

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 17 Formal Testing : Test Derivation test generation test suite T S specification S Test generation : der : SPECS   ( TESTS ) Test suite - set of test cases : T  TESTS Test case : t  TESTS

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 18 Formal Testing : Test Execution test execution OBS test suite T implementatio n IUT Test execution leads to a set of observations : exec : TESTS  IMPS   ( OBS ) Model of test execution : obs : TESTS  MODS   ( OBS ) i IUT

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 19 Formal Testing : Test Hypothesis OBS test suite T Observational framework : TESTS, OBS, exec, obs Test hypothesis :  IUT  IMPS.  i IUT  MODS.  t  TESTS. exec ( t, IUT ) = obs ( t, i IUT ) obs i IUT test execution IUT exec IUT exec

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 20 Formal Testing : Verdicts Observations are interpreted : t :  (OBS)  { fail, pass } test execution OBS t pass fail

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 21 Testing for Conformance IUT passes T s  def  t  T s. IUT passes t IUT passes t  def t ( exec ( t, IUT ) ) = pass Test hypothesis :  t  TESTS. exec ( t, IUT ) = obs ( t, i IUT ) Proof obligation :  i  MODS. (  t  T s. t ( obs ( t, i ) ) = pass )  i imp s IUT passes T s  IUT conforms-to s ? Definition : IUT conforms-to s IUT conforms-to s i IUT imp s   t  T s. t ( obs ( t, i IUT ) ) = pass   t  T s. t ( exec ( t, IUT ) ) = pass   t  T s. IUT passes t  IUT passes T s 

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 22 Testing for Conformance IUT passes T s Proof of completeness on models leads to completeness for tested systems : i conforms-to s exhaustive   sound  Proof obligation :  i  MODS. (  t  T s. t ( obs ( t, i ) ) = pass )  i imp s Test derivation: find such a T s, der : SPECS  P TESTS

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 23 Formal Testing exec : TESTS  IMPS   (OBS) der : SPECS   (TESTS) T s  TESTS s  SPECS IUT  IMPS imp i IUT  MODS obs : TESTS  MODS   (OBS) t :  (OBS)  {fail,pass} OBS pass fail Proof soundness and exhaustivess:  i  MODS. (  t  der(s). t (obs(t,i)) = pass )  i imp s Test hypothesis :  IUT  IMPS.  i IUT  MODS.  t  TESTS. exec(t, IUT ) = obs(t,i IUT )

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 24 Approaches to Formal Testing lFinite State Machine based approaches: nlanguage acceptance ntransition tours ndistinguishing sequences lLabelled Transition Systems nconcurrency theory nimplementation relations lAbstract Data Type testing

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 25 Labelled Transition Systems Labelled Transition System  S, L, T, s 0  ConReq ConConf Discon Data Discon states actions, labels transitions T  S  (L  {  })  S initial state s 0  S

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 26 Labelled Transition Systems S0 S1 dub transition dub coffee S0 S3 transition composition kwart tea S0 executable sequence non-executable sequence kwart soup S0 LTS(L) all transition systems over L L = { dub,kwart, coffee,tea,soup} dub coffee kwart tea S1 S2 S3 S0 s S4 S5 

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 27 Labelled Transition Systems dub coffee dub tea S1 S2 S3 S4 S0 traces (s) = {   L* | s  } s after  = { s’ | s  s’ } s Sequences of observable actions: Reachable states: traces(s) = { , dub, dub coffee, dub tea } s after dub = { S1, S2 } s after dub tea = { S4 }

© Ed Brinksma/Jan Tretmans TT 2004, Formal Framework 28 Formal Testing with Transition Systems t :  (traces)  {fail,pass} exec : TESTS  IMPS   (OBS) traces der : LTS   (TTS) T s  TTS s  LTS IUT  IMPS ioco i IUT  IOTS pass fail obs : TTS  IOTS   (traces) Proof soundness and exhaustivess:  i  IOTS. (  t  der(s). t (obs(t,i)) = pass )  i ioco s Test hypothesis :  IUT  IMPS.  i IUT  IOTS.  t  TTS. exec(t, IUT ) = obs(t,i IUT )