2. Holger Hermanns Formal Methods for Software Engineering Part III: Applications of Formal Methods Lecture 10: Applications?

3. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 2 This Part We apply the concepts, methods and tools you learnt to love in contexts that are relatively close to what the people out there are facing. This week I show you what they are facing, and I round off the entire lecture series. Assumptions for today: Nothing particular.

4. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 3 Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

5. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 4 What’s this?

6. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 5 Nieuwe Waterweg Storm surge barrier North Sea   Rotterdam

7. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 6 First planned in 1953. Completed in 1999. Some statistical data: Each barrier wall has the height of one Eifel Tour, and weighs twice as much. Decision are taken 24 hrs before actual closure, Reversible until 3 hrs before closure. Fully mechanised -software controlled - decision procedure. Nieuwe Waterweg Storm surge barrier FULLY (where ‘fully’ means FULLY’ ).

8. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 7 Nieuwe Waterweg Storm surge barrier North Sea   Rotterdam  ‘BESW’ ‘BOS’  North Wall  South Wall 

9. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 8 The Storm surge barrier System consists of distributed components: north wall, south wall, various hydraulic parts, engines, etc. BOS (‘beslissing & ondersteunend systeem’) knows the environmental conditions; takes decisions, based on the available data; BESW (‘besturingssysteem waterweg’) knows & controls the barrier; carries out commands of BOS; reports status information to BOS;

10. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 9 The design philosophy taken from ‘Ministerie van Verkeer en Waterstaat’ ‘BBI’ (BOS-BESW Interface)

11. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 10 The Storm surge barrier Budget issues Total costs > 500 million €; Costs for software < 10 million € (< 2%) Control software (‘BBI’) developed mainly by CMG. Formal specification techniques used: Z Promela (academic SDL variant, nicer) Experience (in a nutshell): Difficult to learn, but pays off enormously.

12. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 11 The Storm surge barrier BBI main components BOS is informed every 10 minute about water, wind and weather status and forecast computes anticipated water level; assesses the anticipation relative to the closing criteria; if needed takes a decision (to close or … to open!); instructs the BESW to implement the decision.

13. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 12 The Storm surge barrier BBI main components BESW controls water levels in docks; opening/closing of dock gates; moving of barrier walls; sinking and refloating of barrier walls; … BESW implements the BOS instructions. BOS and BESW are about 300 mtrs apart, and interact via redundant bidirectional channel pairs. In particular, they exchange ‘heartbeat’ signals every 30 sec to indicate ‘I am alive’.

14. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 13 Some fragments of the BBI in SDL block BOS BOS [status,stop,close,…] [data] ENV [curr] BOS2BESW BESW2BOS system BBI BOS [status, stop, close,…] [data, emergency] [curr] ENV BOS2BESW BESW2BOS BESW SIGNAL status, stop, close, curr, …; [curr] block BESW BESW NORTH [closed,…] SOUTH [close,…] [closed,…] [status,stop,close,…] BOS2BESW BESW2BOS SIGNAL close, closed, …;

15. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 14 BESW process fragment in SDL process BOS S_active:=ff S_ready :=tt Closing closed FROM SOUTH curr(active,ready,stopped) - status closed FROM NORTH S_active - N_active:=ff N_ready :=tt N_active - active := S_active && N_active ready := S_ready && N_ready stopped:= S_stopped && N_stopped - stop S_active S_active := ff S_stopped:= tt N_active N_active := ff N_stopped:= tt … … tt ff DCL N_active, S_active, active Boolean; N_ready, S_ready, ready Boolean; N_stopped, S_stopped, stopped Boolean; …;

16. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 15 - * BOS process fragments in SDL process BOS Checking status Waiting NONE curr(active,ready,stopped) ready || stopped tt Stable Idle close Checking data(…) … ff emergency stop DCL active Boolean; ready Boolean; stopped Boolean; …;

17. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 16 Problems? Well, here is the intended behaviour. That’s how it should be. Good! Yahoo!

18. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 17 Problem! The system may get stuck if a `stop’ message arrives at the BESW while the gates are closing.

19. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 18 Here is the (almost) original MSC, reported by Pim Kars in November 1998. It was found with the model checker SPIN.

20. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 19 Here is another MSC, reporting a timing violation problem discovered by Theo Ruys a little later. This design error has to do with the heartbeat signals and maximally anticipated delays when links fail. I cannot explain this problem, without adding too much detail. What I can tell you: The designers implemented a solution different from the one proposed. The implemented solution was never verified.

21. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 20 Storm surge barrier: Results Z was used for specifying the functions performed by processes; syntax- and type-checking was done with the ZTC tool; was found very useful to allow a too great deal of freedom and to offer little structure for the style in which it is to be used; was equipped with a common ‘style’ (comparable to a coding-standard) for using Z within the project, containing heuristics and pragmatic rules for its use.

22. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 21 Storm surge barrier: Results Promela: used for modelling the interaction between processes and to the “outside” world. limited to the verification of standard properties such as the absence of deadlock and successful because 1. it helped in reducing defects and 2. it helped in detecting defects early in the development process (reduces the effort and cost required in later stages of development.) This work was done by Pim Kars, Wil Janssen, Theo Ruys, Jan Tretmans, Job Zwiers and Ed Brinksma.

23. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 22 The BBI experience in the literature [Tretmans,Wijbrans,Chaudron 2001]

24. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 23 Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

25. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 24 System and software design cycle Requirements Engineering Conceptual Design Detailled Design Maintenance Coding Testing Design Validation

26. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 25 Computer-assistance in design and validation Requirements Engineering Conceptual Design Detailled Design Maintenance Coding Testing Design Validation Early phases: (Graphical) Specification formalisms: SDL, FSP, Z, Promela, UML Later Phases: Implementation Code Generation Early phases: Verification Later Phases: Test Generation & Execution, Debugging

27. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 26 Computer-assisted system validation Does a design D satisfy a requirement  ? Techniques: Theorem proving Strategy: generate a formal proof that D satisfies . Applicable if design D can be represented in some adequate mathematical theory. Model checking Strategy: check systematically and exhaustively whether each reachable state in D satisfies . Applicable if the behaviour of D can be finitely represented. Simulation, or Testing Strategy: Check whether  holds on some executions of D. Applicable if D is executable.

28. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 27 Testing Test whether it works ……

29. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 28 Testing to check the quality of an object by performing experiments in a controlled way Software Testing: testing the quality of a software product

30. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 29 Testing Testing is: important much practiced 30% - 50% of project effort - at least expensive time critical not constructive (but sadistic?) But also: ad-hoc, manual, error-prone hardly theory / research no attention in curricula not cool : “if you’re a bad programmer you might be a tester” Attitude is changing: more awareness more professionalism

31. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 30 construction: implementation process checking: testing process implementation specification Testing functional behaviour of black-box implementation with respect to a specification Specification Based Functional Testing

32. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 31 construction: implementation process checking: testing process implementation formal specification Testing functional behaviour of black-box implementation with respect to a specification in a formal language based on a formal definition of conformance Specification Based Functional Testing with formal methods

33. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 32 Testing based on formal specification Testing with respect to a formal specification Precise, formal definition of correctness: good and unambiguous basis for testing Formal validation of tests Algorithmic derivation of tests: tools for automatic test generation Allows to define measures expressing coverage and quality of testing

34. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 33 Labelled Transition System based Testing Testing theory based on LTS Specifications, implementations, and tests all modelled as LTS ConReq ConConf Discon Data Discon

35. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 34 Labelled Transition System based Testing Uses a common variation of labelled transition system: Input-Output Transition System - IOTS distinction between outputs ! and always-enabled inputs ? IOTS with variables - equation solver for y 2 =x ? x (x >= 0) !  x ? x (x < 0) ? y ! -  x ? x (x >= 0) !  x ? x (x < 0) ? y You have seen such transition systems already in the SDL context, remember? (but their transitions were denoted slightly different there).  This is just another way of expressing non-persistent input a lá SDL.

36. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 35 Algorithm To generate a test case from an IOTS with initial state s. 1end test case PASS Apply the following steps recursively, non-deterministically 2supply input supply ?a t(S after ?a) Test Generation Algorithm 3observe output FAIL t(S after !x) FAIL allowed outputs !x forbidden outputs !y 

37. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 36 specificationtest ! 9 ! 4 ? -2 ? 2 PASS otherwise FAIL PASS otherwise ? 3 ? -3 FAIL ? x (x >= 0) !  x ? x (x < 0) ! -  x To cope with non-deterministic behaviour, tests are not linear traces, but trees Test Generation Example Equation solver for y 2 =x Let specification play against implementation, with input and output reversed

38. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 37 ? x (x >= 0) !  x ? x (x < 0) ! -  x ? y implementationtest ! 9 ! 4 ? -2 ? 2 PASS otherwise FAIL PASS otherwise ? 3 ? -3 FAIL Test Execution Example

39. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 38 A Test Tool: TorX On-the-fly test generation and test execution Correctness criterion: ‘ioco’ developed within the ‘Cote-de-Resyste’ project Specification languages: LOTOS, FSP Promela (SDL inspired, nicer) TorX next input specificationIUT observe output offer input check output pass fail inconclusive user: manual automatic specification  great stuff  pretty successful (FMT, Philips, Lucent,...)  Implementation under Test

40. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 39 explorerprimerdriveradapterIUT bits bytes states transitions abstract actions transition ? x (x >= 0) !  x ? x (x < 0) ! -  x specificationimplementation ? x (x >= 0) !  x ? x (x < 0) ? x On-The-Fly Testing Choose ! x (x < 0) ! x (x >= 0) Choice ! 9 Abstract action ! 9 Concrete action ! 00001001 specification

41. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 40 explorerprimerdriveradapterIUT bits bytes states transitions abstract actions transition ? x (x >= 0) !  x ? x (x < 0) ! -  x specificationimplementation ? x (x >= 0) !  x ? x (x < 0) ? x On-The-Fly Testing Concrete action ? 00000011 Abstract action ? 3 Action ?  9 Action ?  x

42. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 41 TorX

43. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 42 Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

44. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 43 Interpay ‘’Rekeningrijden’’ Highway Tolling System Appareantly simple protocol Parallellism: many cars at the same time Real-time constraints Encryption

45. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 44 Payment Box RoadSide Equipment Onboard Unit UDP/IPWireless Highway Tolling System

46. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 45 specification Payment Box TorX Payment Box System under Test ‘’Rekeningrijden’’: Test Architecture + UDP/IP

47. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 46 ‘’Rekeningrijden’’: Results Results: errors detected during model checking (design error) errors detected during testing (coding error) Automated testing : beneficial: high volume and reliability very flexible: adaptation and many configurations Real-time: How to cope with real time constraints ? Efficient computation for on-the-fly testing ? Real life: How to cope with fast-moving managers? (Roel-Pieper-Effect) This work was done by Axel Belinfante, René De Vries, Jan Feenstra and Jan Tretmans.

48. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 47 Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in the entire set of lectures A third, very real application. Wrap up.

49. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 48 Model Construction What’s the obstacle?

50. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 49 I deleted some slides here, which I did not show due to time constraints, and because they are a bit off track. The slides were motivating ‘compositionality’ as the main principle to control state space sizes and verifcation issues. I keep here one technical slide on that issue (see next slide).

51. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 50 Compositionality, in more formal terms Given an equivalence relation on processes P, Q, R,... (LTSs (S,L, ,s) ) and some operators to compose/manipulate processes, op (P,R), op’(P),... you want: if P and Q are equivalent, then op(P,R) and op(Q,R) are equivalent, op’(P,R) and op’(Q) are equivalent,... whatever R may be.  an example (observation equivalence, also trace equivalence but not trace-and-deadlock equivalence)

52. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 51 Model construction features: What’ we haven’t told you process Controller (int y) { choose{ :: when (y > 0) { fail; Controller (y-1); } :: when (y<2) { throw sleep; } :: when (y=0){...} } loop{ try{ par { ::Gyro() ::Controller(6); } } catch sleep { prepare;launch; repair; } For instance: Exception Handling Real time (important) Randomness (cool) Priorities (tough)... Main criterion for a reasonable set up: compositionality

53. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 52 Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

54. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 53 Demo of UPPAAL goes here.

55. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 54 Contents of this lecture A real application. Testing based on formal methods. Another real application. Model construction and model checking beyond what you have seen in this entire set of lectures. A third, very real application. Wrap up.

56. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 55 Verification vs. Testing verification

57. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 56 formal world concrete world Verification is only as good as the validity of the model on which it is based Verification vs. Testing Verification: (model checking, theorem proving) formal manipulation prove properties performed on model Testing : experimentation show error concrete system Testing can only show the presence of errors, not their absence

58. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 57 What was this whole lecture all about Make you alert about concurrency and its difficulties Tell you about the principal model(s) of concurrency Give you a flavor of important specification languages Tell you about the principles behind such languages Let you play with Z, FSP, and SDL

59. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 58 What it was not about Deriving implementations from a formal specification  ‘protocol engineering’ (Luis Pires) Model checking a formal specification  ‘system validation’ (Joost-Pieter Katoen) Formal specification-based testing  ‘testing techniques’ (Ed Brinksma) Web page design But it prepared you for these lectures.

60. Holger Hermanns Formal Methods for Software Engineeiring - Lecture 10 59 A few days into the mission, not long after Pathfinder started gathering meteorological data, the spacecraft began experiencing total system resets, each resulting in losses of data. Curious? Yet another example: Mars Pathfinder