Holger Hermanns Formal Methods for Software Engineering Part III: Applications of Formal Methods Lecture 10: Applications?

Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 2 This Part We apply the concepts, methods and tools you learnt to love in contexts that are relatively close to what the people out there are facing. This week I show you what they are facing, and I round off the entire lecture series. Assumptions for today: Nothing particular.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 3 Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 4 What’s this?

Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 5 Nieuwe Waterweg Storm surge barrier North Sea   Rotterdam

Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 6 First planned in Completed in Some statistical data: Each barrier wall has the height of one Eifel Tour, and weighs twice as much. Decision are taken 24 hrs before actual closure, Reversible until 3 hrs before closure. Fully mechanised -software controlled - decision procedure. Nieuwe Waterweg Storm surge barrier FULLY (where ‘fully’ means FULLY’ ).

Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 7 Nieuwe Waterweg Storm surge barrier North Sea   Rotterdam  ‘BESW’ ‘BOS’  North Wall  South Wall 

Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 8 The Storm surge barrier System consists of distributed components: north wall, south wall, various hydraulic parts, engines, etc. BOS (‘beslissing & ondersteunend systeem’) knows the environmental conditions; takes decisions, based on the available data; BESW (‘besturingssysteem waterweg’) knows & controls the barrier; carries out commands of BOS; reports status information to BOS;

Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 9 The design philosophy taken from ‘Ministerie van Verkeer en Waterstaat’ ‘BBI’ (BOS-BESW Interface)

Holger Hermanns Formal Methods for Software Engineeiring - Lecture The Storm surge barrier Budget issues Total costs > 500 million €; Costs for software < 10 million € (< 2%) Control software (‘BBI’) developed mainly by CMG. Formal specification techniques used: Z Promela (academic SDL variant, nicer) Experience (in a nutshell): Difficult to learn, but pays off enormously.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture The Storm surge barrier BBI main components BOS is informed every 10 minute about water, wind and weather status and forecast computes anticipated water level; assesses the anticipation relative to the closing criteria; if needed takes a decision (to close or … to open!); instructs the BESW to implement the decision.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture The Storm surge barrier BBI main components BESW controls water levels in docks; opening/closing of dock gates; moving of barrier walls; sinking and refloating of barrier walls; … BESW implements the BOS instructions. BOS and BESW are about 300 mtrs apart, and interact via redundant bidirectional channel pairs. In particular, they exchange ‘heartbeat’ signals every 30 sec to indicate ‘I am alive’.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Some fragments of the BBI in SDL block BOS BOS [status,stop,close,…] [data] ENV [curr] BOS2BESW BESW2BOS system BBI BOS [status, stop, close,…] [data, emergency] [curr] ENV BOS2BESW BESW2BOS BESW SIGNAL status, stop, close, curr, …; [curr] block BESW BESW NORTH [closed,…] SOUTH [close,…] [closed,…] [status,stop,close,…] BOS2BESW BESW2BOS SIGNAL close, closed, …;

Holger Hermanns Formal Methods for Software Engineeiring - Lecture BESW process fragment in SDL process BOS S_active:=ff S_ready :=tt Closing closed FROM SOUTH curr(active,ready,stopped) - status closed FROM NORTH S_active - N_active:=ff N_ready :=tt N_active - active := S_active && N_active ready := S_ready && N_ready stopped:= S_stopped && N_stopped - stop S_active S_active := ff S_stopped:= tt N_active N_active := ff N_stopped:= tt … … tt ff DCL N_active, S_active, active Boolean; N_ready, S_ready, ready Boolean; N_stopped, S_stopped, stopped Boolean; …;

Holger Hermanns Formal Methods for Software Engineeiring - Lecture * BOS process fragments in SDL process BOS Checking status Waiting NONE curr(active,ready,stopped) ready || stopped tt Stable Idle close Checking data(…) … ff emergency stop DCL active Boolean; ready Boolean; stopped Boolean; …;

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Problems? Well, here is the intended behaviour. That’s how it should be. Good! Yahoo!

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Problem! The system may get stuck if a `stop’ message arrives at the BESW while the gates are closing.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Here is the (almost) original MSC, reported by Pim Kars in November It was found with the model checker SPIN.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Here is another MSC, reporting a timing violation problem discovered by Theo Ruys a little later. This design error has to do with the heartbeat signals and maximally anticipated delays when links fail. I cannot explain this problem, without adding too much detail. What I can tell you: The designers implemented a solution different from the one proposed. The implemented solution was never verified.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Storm surge barrier: Results Z was used for specifying the functions performed by processes; syntax- and type-checking was done with the ZTC tool; was found very useful to allow a too great deal of freedom and to offer little structure for the style in which it is to be used; was equipped with a common ‘style’ (comparable to a coding-standard) for using Z within the project, containing heuristics and pragmatic rules for its use.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Storm surge barrier: Results Promela: used for modelling the interaction between processes and to the “outside” world. limited to the verification of standard properties such as the absence of deadlock and successful because 1. it helped in reducing defects and 2. it helped in detecting defects early in the development process (reduces the effort and cost required in later stages of development.) This work was done by Pim Kars, Wil Janssen, Theo Ruys, Jan Tretmans, Job Zwiers and Ed Brinksma.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture The BBI experience in the literature [Tretmans,Wijbrans,Chaudron 2001]

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture System and software design cycle Requirements Engineering Conceptual Design Detailled Design Maintenance Coding Testing Design Validation

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Computer-assistance in design and validation Requirements Engineering Conceptual Design Detailled Design Maintenance Coding Testing Design Validation Early phases: (Graphical) Specification formalisms: SDL, FSP, Z, Promela, UML Later Phases: Implementation Code Generation Early phases: Verification Later Phases: Test Generation & Execution, Debugging

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Computer-assisted system validation Does a design D satisfy a requirement  ? Techniques: Theorem proving Strategy: generate a formal proof that D satisfies . Applicable if design D can be represented in some adequate mathematical theory. Model checking Strategy: check systematically and exhaustively whether each reachable state in D satisfies . Applicable if the behaviour of D can be finitely represented. Simulation, or Testing Strategy: Check whether  holds on some executions of D. Applicable if D is executable.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Testing Test whether it works ……

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Testing to check the quality of an object by performing experiments in a controlled way Software Testing: testing the quality of a software product

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Testing Testing is: important much practiced 30% - 50% of project effort - at least expensive time critical not constructive (but sadistic?) But also: ad-hoc, manual, error-prone hardly theory / research no attention in curricula not cool : “if you’re a bad programmer you might be a tester” Attitude is changing: more awareness more professionalism

Holger Hermanns Formal Methods for Software Engineeiring - Lecture construction: implementation process checking: testing process implementation specification Testing functional behaviour of black-box implementation with respect to a specification Specification Based Functional Testing

Holger Hermanns Formal Methods for Software Engineeiring - Lecture construction: implementation process checking: testing process implementation formal specification Testing functional behaviour of black-box implementation with respect to a specification in a formal language based on a formal definition of conformance Specification Based Functional Testing with formal methods

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Testing based on formal specification Testing with respect to a formal specification Precise, formal definition of correctness: good and unambiguous basis for testing Formal validation of tests Algorithmic derivation of tests: tools for automatic test generation Allows to define measures expressing coverage and quality of testing

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Labelled Transition System based Testing Testing theory based on LTS Specifications, implementations, and tests all modelled as LTS ConReq ConConf Discon Data Discon

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Labelled Transition System based Testing Uses a common variation of labelled transition system: Input-Output Transition System - IOTS distinction between outputs ! and always-enabled inputs ? IOTS with variables - equation solver for y 2 =x ? x (x >= 0) !  x ? x (x < 0) ? y ! -  x ? x (x >= 0) !  x ? x (x < 0) ? y You have seen such transition systems already in the SDL context, remember? (but their transitions were denoted slightly different there).  This is just another way of expressing non-persistent input a lá SDL.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Algorithm To generate a test case from an IOTS with initial state s. 1end test case PASS Apply the following steps recursively, non-deterministically 2supply input supply ?a t(S after ?a) Test Generation Algorithm 3observe output FAIL t(S after !x) FAIL allowed outputs !x forbidden outputs !y 

Holger Hermanns Formal Methods for Software Engineeiring - Lecture specificationtest ! 9 ! 4 ? -2 ? 2 PASS otherwise FAIL PASS otherwise ? 3 ? -3 FAIL ? x (x >= 0) !  x ? x (x < 0) ! -  x To cope with non-deterministic behaviour, tests are not linear traces, but trees Test Generation Example Equation solver for y 2 =x Let specification play against implementation, with input and output reversed

Holger Hermanns Formal Methods for Software Engineeiring - Lecture ? x (x >= 0) !  x ? x (x < 0) ! -  x ? y implementationtest ! 9 ! 4 ? -2 ? 2 PASS otherwise FAIL PASS otherwise ? 3 ? -3 FAIL Test Execution Example

Holger Hermanns Formal Methods for Software Engineeiring - Lecture A Test Tool: TorX On-the-fly test generation and test execution Correctness criterion: ‘ioco’ developed within the ‘Cote-de-Resyste’ project Specification languages: LOTOS, FSP Promela (SDL inspired, nicer) TorX next input specificationIUT observe output offer input check output pass fail inconclusive user: manual automatic specification  great stuff  pretty successful (FMT, Philips, Lucent,...)  Implementation under Test

Holger Hermanns Formal Methods for Software Engineeiring - Lecture explorerprimerdriveradapterIUT bits bytes states transitions abstract actions transition ? x (x >= 0) !  x ? x (x < 0) ! -  x specificationimplementation ? x (x >= 0) !  x ? x (x < 0) ? x On-The-Fly Testing Choose ! x (x < 0) ! x (x >= 0) Choice ! 9 Abstract action ! 9 Concrete action ! specification

Holger Hermanns Formal Methods for Software Engineeiring - Lecture explorerprimerdriveradapterIUT bits bytes states transitions abstract actions transition ? x (x >= 0) !  x ? x (x < 0) ! -  x specificationimplementation ? x (x >= 0) !  x ? x (x < 0) ? x On-The-Fly Testing Concrete action ? Abstract action ? 3 Action ?  9 Action ?  x

Holger Hermanns Formal Methods for Software Engineeiring - Lecture TorX

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Interpay ‘’Rekeningrijden’’ Highway Tolling System Appareantly simple protocol Parallellism: many cars at the same time Real-time constraints Encryption

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Payment Box RoadSide Equipment Onboard Unit UDP/IPWireless Highway Tolling System

Holger Hermanns Formal Methods for Software Engineeiring - Lecture specification Payment Box TorX Payment Box System under Test ‘’Rekeningrijden’’: Test Architecture + UDP/IP

Holger Hermanns Formal Methods for Software Engineeiring - Lecture ‘’Rekeningrijden’’: Results Results: errors detected during model checking (design error) errors detected during testing (coding error) Automated testing : beneficial: high volume and reliability very flexible: adaptation and many configurations Real-time: How to cope with real time constraints ? Efficient computation for on-the-fly testing ? Real life: How to cope with fast-moving managers? (Roel-Pieper-Effect) This work was done by Axel Belinfante, René De Vries, Jan Feenstra and Jan Tretmans.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in the entire set of lectures A third, very real application. Wrap up.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Model Construction What’s the obstacle?

Holger Hermanns Formal Methods for Software Engineeiring - Lecture I deleted some slides here, which I did not show due to time constraints, and because they are a bit off track. The slides were motivating ‘compositionality’ as the main principle to control state space sizes and verifcation issues. I keep here one technical slide on that issue (see next slide).

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Compositionality, in more formal terms Given an equivalence relation on processes P, Q, R,... (LTSs (S,L, ,s) ) and some operators to compose/manipulate processes, op (P,R), op’(P),... you want: if P and Q are equivalent, then op(P,R) and op(Q,R) are equivalent, op’(P,R) and op’(Q) are equivalent,... whatever R may be.  an example (observation equivalence, also trace equivalence but not trace-and-deadlock equivalence)

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Model construction features: What’ we haven’t told you process Controller (int y) { choose{ :: when (y > 0) { fail; Controller (y-1); } :: when (y<2) { throw sleep; } :: when (y=0){...} } loop{ try{ par { ::Gyro() ::Controller(6); } } catch sleep { prepare;launch; repair; } For instance: Exception Handling Real time (important) Randomness (cool) Priorities (tough)... Main criterion for a reasonable set up: compositionality

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Demo of UPPAAL goes here.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture Verification vs. Testing verification

Holger Hermanns Formal Methods for Software Engineeiring - Lecture formal world concrete world Verification is only as good as the validity of the model on which it is based Verification vs. Testing Verification: (model checking, theorem proving) formal manipulation prove properties performed on model Testing : experimentation show error concrete system Testing can only show the presence of errors, not their absence

Holger Hermanns Formal Methods for Software Engineeiring - Lecture What was this whole lecture all about Make you alert about concurrency and its difficulties Tell you about the principal model(s) of concurrency Give you a flavor of important specification languages Tell you about the principles behind such languages Let you play with Z, FSP, and SDL

Holger Hermanns Formal Methods for Software Engineeiring - Lecture What it was not about Deriving implementations from a formal specification  ‘protocol engineering’ (Luis Pires) Model checking a formal specification  ‘system validation’ (Joost-Pieter Katoen) Formal specification-based testing  ‘testing techniques’ (Ed Brinksma) Web page design But it prepared you for these lectures.

Holger Hermanns Formal Methods for Software Engineeiring - Lecture A few days into the mission, not long after Pathfinder started gathering meteorological data, the spacecraft began experiencing total system resets, each resulting in losses of data. Curious? Yet another example: Mars Pathfinder