Formal methods & Tools Ed Brinksma University of Twente, Netherlands ISSTA/Wosp Rome, July 24 th, 2002 Qualitative and Quantitative Analysis of Software Systems: a Model-based View on Integrated Analysis

July 24, 2002ISSTA/Wosp, Rome2 MOTIVATION Can the qualitative and quantitative aspects of reactive systems be modelled and analysed within one compositional framework? Central Issue  increasing importance of quantitative behaviour  need for integrated design disciplines  cross-fertilization  theory of approximate correctness

July 24, 2002ISSTA/Wosp, Rome3 Contents  Analysis, validation, and modelling  Design, composition, and algebra  Model-based test generation  Integrated performance analysis  Conclusions & perspective

July 24, 2002ISSTA/Wosp, Rome4 Modelling  science vs. engineering  software programs as models  models of software systems  the role of experimentation

July 24, 2002ISSTA/Wosp, Rome5 The Empirical Cycle Formal world Physical world induction validation phenomenon experiment model analyse

July 24, 2002ISSTA/Wosp, Rome6 Methodology of Science  a normative procedure: when is a theory is acceptable?  models are refined by iteration  models play a descriptive role (Vienna Circle: Carnap et al.)

July 24, 2002ISSTA/Wosp, Rome7 Engineering  models (specs) are prescriptive  deals with artefacts

July 24, 2002ISSTA/Wosp, Rome8 The Design Cycle Formal world Physical world realization validation artefact experiment model analyse

July 24, 2002ISSTA/Wosp, Rome9 Methodology of Engineering  design cycle is  design cycle is normative, a posteriori is refined by  artefact is refined by iteration employ theory of  can employ theory of underlying cycles empirical cycles

July 24, 2002ISSTA/Wosp, Rome10 The formal nature of software Formal world Physical world implementation validation behaviour programspecification compilation verification

July 24, 2002ISSTA/Wosp, Rome11 Programming as mathematics  programs are formal models of their realizations.  program derivation and verification are mathematical activities, subject to proofs  validation can be achieved through reliable compilation/interpretation (Dijkstra, Hoare et al.)

July 24, 2002ISSTA/Wosp, Rome12 Limits of the Mathematical Paradigm Formal world Physical world implementation validation physical system software system specification realization verification incomplete combinatorial explosion large & incomplete unreliable

July 24, 2002ISSTA/Wosp, Rome13 System Modelling Formal world Physical world physical system software system specification system model verificationvalidation specification fragment validation

July 24, 2002ISSTA/Wosp, Rome14 System Modelling  Abstract models of software  Interpolation specs and programs  Basis for verification  Basis for validation like traditional engineering models

July 24, 2002ISSTA/Wosp, Rome15 System Model Validation  Top-down: specification refinement  Bottom-up: implementation abstraction  Formally: by proof or by construction  Empirically: by experimentation

July 24, 2002ISSTA/Wosp, Rome16 Empirical Validation Formal world Non-formal world validation system description query system model analyse induction system design physical system experimentation & modification

July 24, 2002ISSTA/Wosp, Rome17 Contents  Analysis, validation, and modelling  Design, composition, and algebra  Model-based test generation  Integrated performance analysis  Conclusions & perspective

July 24, 2002ISSTA/Wosp, Rome18 Design and Complexity  Divide and conquer: distribute functionality  Hierarchical design: different abstraction levels Modelling formalism must support compositionality and abstraction

July 24, 2002ISSTA/Wosp, Rome19 Navigating the Design Graph Abstraction level Alternatives Modelling formalism must support model transformation

July 24, 2002ISSTA/Wosp, Rome20 Process Algebra  abstract process modelling  compositionality  abstraction  transformation laws  operational interpretation

July 24, 2002ISSTA/Wosp, Rome21 Basic Process Algebraic Operators  inaction: 0  action-prefix: a.B, .B  choice: B + C,  I B i  composition: B || A C  hiding: B \ A  definition: p = def B  application: p

July 24, 2002ISSTA/Wosp, Rome22 mid |[mid]| hide mid in || {mid} A very basic example  A simple one place buffer Buf = def in. out. Buf  Two instances of this buffer outin Buf[out/mid] Buf[in/mid] Buf out in midout in out  Buf in Buf out

July 24, 2002ISSTA/Wosp, Rome23 A very basic example II  A two place buffer Buf2 = def in.Half Half = def in.Full + out.Buf2 Full = def out.Half out in Buf2 out in

July 24, 2002ISSTA/Wosp, Rome24 Equivalence two place Two ways to represent a two place buffer: out in  by enumerating the detailled behaviour Buf2 out in out in out  mid Buf in Buf out  by coupling two one place buffers equivalences examples for the need to study equivalences

July 24, 2002ISSTA/Wosp, Rome25 Equivalence Process algebraic equivalences are based on different answers to the question: What is an observable part of the behaviour of a process ?  Various notions have been studied (see [van Glabbeek])  trace equivalences  testing equivalences  bisimulation equivalences

July 24, 2002ISSTA/Wosp, Rome26 Algebraic Laws Equivalences (congruences) induce algebraic laws + = + B + C = C + B (+ ) + = + (+ ) (B + C) + D = B + (C + D) + 0 = B + 0 = B + = B + B = B || A = || A B || A C = C || A B (|| A ) || A = || A (|| A ) (B || A C) || A D = B || A (C || A D)

July 24, 2002ISSTA/Wosp, Rome27 Expansion Laws Parallelism can be removed step by step:  Let B =  k a k. B k and C =  l c l. C l B || A C =  {a k. (B k || A C ) | a k  A } +  {c l. (B || A C l ) | c l  A } +  {d. (B k || A C l ) | d = a k =c l  A } Example: a. 0 ||  c. 0 = a. c. 0 + c. a. 0 a c a c

July 24, 2002ISSTA/Wosp, Rome28 Expansion Laws  move between abstraction levels  (de)compose functionality  stepwise simulation These laws are crucial for the success of process algebraic modelling

July 24, 2002ISSTA/Wosp, Rome29 Contents  Analysis, validation, and modelling  Design, composition, and algebra  Model-based test generation  Integrated performance analysis  Conclusions & perspective

July 24, 2002ISSTA/Wosp, Rome30 Testing With Formal Methods Advantages formal approach:  algorithmic generation of sound tests  formal validation of tests tests specification implementation test generator test executor verdict (pass/fail) implements formal modelcorrectness relation

July 24, 2002ISSTA/Wosp, Rome31 Correctness Relations synchronous communication :  testing preorder (De Nicola & Hennessy, 1984)  conformance preorder (Brinksma 1987)  refusal testing (Phillips 1987, Langerak 1990) asynchronous communication :  queue-based testing (Brinksma & Tretmans 1992)  I/O-based testing (Phalippou,Tretmans,Segala 1993)  repetitive quiescence (Tretmans 1996)  multiple I/O (Heerink & Tretmans 1997)

July 24, 2002ISSTA/Wosp, Rome32 formal model: IUT ( || Env)/A Asynchronous Test Contexts implementation under test test engine test environment interaction input output  operating system  test method  communication buffers compositionality

July 24, 2002ISSTA/Wosp, Rome33 Formal Correctness I ioco S = def   Straces(s): out(I after  )  out(S after  ) ? x (x >= 0) !  x ? x (x < 0) ! -  x S ioco Example: equation solver for y 2 =x ? x (x >= 0) !  x ? x (x < 0) ? x I Intuition : I ioco-conforms to S iff 1.if I produces output x after trace , then S can produce x after  2.if I cannot produce any output after trace , then S cannot produce any output after  (quiescence)

July 24, 2002ISSTA/Wosp, Rome34 Nondeterministic algorithm Generate a test case t(S) from a transition system model with S a set of states (initially S = {s 0 }) Formal Test Generation 1. end test case PASS Apply the following steps recursively, nondeterministically 2. supply input supply ?a t(S after ?a)t(S after !x 1 ) 3. observe output FAIL t(S after !x 2 ) FAIL allowed outputs forbidden outputs !y1!y1 !x1!x1 !x2!x2 !x3!x3 t(S after !x 3 ) !y2!y2

July 24, 2002ISSTA/Wosp, Rome35 Test Generation Example Equation solver for y 2 =x Test ! 9 ! 4 ? -2 ? 2 PASS otherwise FAIL ? -3 PASS otherwise ? 3 FAIL Model ? x (x >= 0) !  x ? x (x < 0) ! -  x Note: to cope with non deterministic behaviour, tests are not linear traces, but trees

July 24, 2002ISSTA/Wosp, Rome36 Exhaustiveness for each ioco-incorrect implementation a test can be generated that detects it I ioco S implies  T such that I fails T Validity of Test Generation For every set of tests T generated with the algorithm: Soundness generated test will never fail with ioco-correct implementation I ioco S implies I passes T

July 24, 2002ISSTA/Wosp, Rome37 Test Generation Tools for ioco  TVEDA (CNET - France Telecom)  derives TTCN tests from SDL specification  developed from practical experiences  TGV (IRISA - Rennes)  derives tests in TTCN from LOTOS or SDL  uses test purposes  TorX (Côte de Resyste)  on-the-fly test generation and execution  uses LOTOS and Promela

July 24, 2002ISSTA/Wosp, Rome38 on the fly batch test generation TTCN batch test execution TTCN TorX Tool Architecture explorerprimerdriveradapterIUT bits bytes states transitions abstract actions transition concentrate on on-the-fly testing Expansion laws

July 24, 2002ISSTA/Wosp, Rome39 TorX Application Highway Tolling System

July 24, 2002ISSTA/Wosp, Rome40 Results  Test results : Errors found during model validation (design error) and during testing (coding error)  Automated testing :  beneficial: high volume and reliability  very flexible: adaptation and many configurations  Real-time :  How to cope with real time constraints ?  Efficient computation for on-the-fly testing ?  Lack of theory: quiescence vs. time-out

July 24, 2002ISSTA/Wosp, Rome41 Contents  Analysis, validation, and modelling  Design, composition, and algebra  Model-based test generation  Integrated performance analysis  Conclusions & perspective

July 24, 2002ISSTA/Wosp, Rome42 Markovian Process Algebra basic idea:  incorporate delays that follow exponential distributions into process algebra  MEMORYLESS Two distinct approaches:  associate delays to actions TIPP, PEPA, EMPA,...  introduce delays as orthogonal entities IMC (also MLOTOS)

July 24, 2002ISSTA/Wosp, Rome43  inaction: 0  prefix: a. B, . B  choice: B + C or  I B i  composition: B || A C  hiding: B \ A or hide A in B  definition: p = def B  application: p Interactive Markov Chains ( ). B,  inaction: 0  prefix: a. B, . B  choice: B + C or  I B i  composition: B || A C  hiding: B \ A or hide A in B  definition: p = def B  application: p supports phase type distributions  

July 24, 2002ISSTA/Wosp, Rome44 Algebraic Laws for IMC These are the algebraic laws for strong Markovian bisimulation, a straightforward combination of strong bisimulation and lumpability.  + = +  B + C = C + B  (+ ) + = + (+ )  (B + C) + D = B + (C + D)  + 0 =  B + 0 = B  + =  a. B + a. B = a. B  + = +  ( ). B + (  ). B = ( +  ). B

July 24, 2002ISSTA/Wosp, Rome45 Interleaving Law The interleaving law holds for IMC : ( ). B ||  (  ). C = ( ). (  ). (B ||  C) + (  ). ( ). (B ||  C) Example :   This does not hold for general distributions!

46 hide enter,serve in CUSTOMER |[enter]| QUEUE(0) |[serve]| SERVER  arriving customers  queue  service clerk Queuing Systems in IMC CUSTOMER = def ( ). enter. CUSTOMER QUEUE(i) = def [i enter. QUEUE(i+1) [i>0]-> serve. QUEUE(i-1) SERVER = def serve. (  ). SERVER        

47 hide enter,serve in CUSTOMER |[enter]| QUEUE(0) |[serve]| SERVER Queuing Systems in IMC          ?

48 hide enter,serve in CUSTOMER |[enter]| QUEUE(0) |[serve]| SERVER Queuing Systems in IMC          weak Markovian bisimulation        

July 24, 2002ISSTA/Wosp, Rome49 A telephony system  Original specification developed by P. Ernberg (SICS), further studied in the French/Canadian Eucalyptus project: more than 1500 lines of LOTOS.  Extensively verified using state-of-the-art techniques  model checking  equivalence checking         

July 24, 2002ISSTA/Wosp, Rome50 Performance analysis of the telephony system  Takes the original specification without changes.  Stochastic delays are incorporated  in a compositional way, i.e. as additional constraints imposed on the specification.  exponential, Erlang and phase-type distributions.  Weak bisimulation is used to factor out nondeterminism.  State space > 10 7 leads to a Markov Chain of 720 states with a highly irregular structure.          using a dedicated operator, time constraints

51  A particular phone:  The time it takes to pick up the phone:  The phone with time constraints:  Time constraints ringT_on pick_phone ringT_off      on ringT_on delay pick_phone in by ringT_off   ringT_on pick_phone ringT_off    

July 24, 2002ISSTA/Wosp, Rome52 Analysis results  14 different time constraints incorporated.  Compositional minimisation to avoid state space explosion.  Here: two subscribers phoning each other.

July 24, 2002ISSTA/Wosp, Rome53 Tools used  CAESAR/ALDEBARAN  original specification,  first minimisation steps.  TIPPtool  time constraints,  final minimisations,  numerical analysis.

July 24, 2002ISSTA/Wosp, Rome54 Contents  Analysis, validation, and modelling  Design, composition, and algebra  Model-based test generation  Integrated performance analysis  Conclusions & perspective

July 24, 2002ISSTA/Wosp, Rome55 Conclusions  software engineering & analysis need modelling, like in traditional engineering  modelling formalisms must support composition, abstraction & transformation  process algebra provides theoretical framework & tool support (expansion laws)  integrated quantitative & qualitative modelling & analysis by embedding models in specialized contexts (extending theory where needed)

July 24, 2002ISSTA/Wosp, Rome56 otherperformance Analytic Contexts testing model

July 24, 2002ISSTA/Wosp, Rome57 Perspectives  Extending specialized contexts  Existing: testing, soft & hard real-time  Future: dependability, security  Extending scope of analysis techniques  Model checking CTMCs (ETMCC tool)  Non-Markovian analysis  Hard real-time testing (STRESS = TorX + Uppaal)  Soft real-time testing  Integrated modelling and tool support  MoDest language and tools project