1. Automated V&V for High Integrity Systems A Targeted Formal Methods Approach Simon Burton Research Associate Rolls-Royce University Technology Centre University of York

2. 2 Motivation V&V for High Integrity Systems is expensive (>50%) but important obviously Requirements change often High V&V costs are repeated Cost of fixing errors far greater the later they are discovered Detecting them sooner will reduce overall system costs Commercial pressure to reduce time to market Need to meet standard mandated metrics Technological conservatism “Radical” new approaches hard to get used “in the line” A rigorous but practical approach is needed

3. 3 Formal Methods - The Solution? Formal methods long been advocated as a cure-all for the problems in software engineering But still to gain widespread use in industry Industrial acceptance requires Development of powerful tools to support formal analysis Pragmatic approaches to using such tools within a process Industrially applicable examples of the successful use of formal methods For engineers with the domain knowledge to be able to perform effective V&V Need to “disguise” the formality Formal analysis under change Traditional formal methods geared towards “right first time” This isn’t practical in an industrial context

4. 4 Domain and Objectives Electronic Engine Controllers for aircraft engines Real-time Safety-critical Fault tolerant Embedded within complex engineering products Objectives of the work Develop efficient and effective software V&V techniques for such systems The techniques must be smoothly integrated into the engineering process Exploit the benefits of formal methods whilst retaining intuitive tabular and graphical notations Exploit the restricted domain of use to reduce the number of mathematical problems to be solved

5. 5 Process Overview Tabular/Graphical Specification Notations Formal Specification Requirements Analysis Test Design Semantics Proof and constraint solving Testing heuristics

6. 6 Summary of talk Background and objectives of work Translating domain specific notations into formal specifications Automatic analysis of “healthiness” properties Automated test design Results Conclusions

7. 7 Graphical/Tabular Notations Graphical/Tabular notations popular with engineers Semantics are often unclear Loose specification and inconsistency complicates specification verification and test case generation Notations will change across and even within projects Not cost effective to invest in automated V&V support for each notation Translate graphical/tabular notations into common formal notation Removes vagueness of original notation - making behaviour implied by requirements explicit for V&V purposes Re-use analysis and test techniques for several notations Restrict the structures used in the formal specification so they can be exploited when designing automated heuristics

8. 8 The PFS Notations Practical Formal Specifications (PFS) Invented at York as part of a MoD funded project Used to model the discrete aspects of engine controllers Consists of State-based components  Outputs depend on the set of current inputs and current state Reactive components  Outputs depend only on the set of current inputs Theory of combining the two Set of suggested “healthiness properties”  Guaranteeing context assumptions  Completeness  Determinism

9. 9 Example: Thrust Reverser

10. 10 Reactive Components This example transforms sensor inputs into conditions that can be used in the control mechanisms

11. 11 State-based component This example decides which actions to take based on input conditions and current state Each state can be refined to specify more detail

12. 12 Translation to Z Semantics of PFS notations formally specified in Z Formal specifications have fixed structure Auxiliary Definitions State Operations over state, inputs and outputs Proof obligations on healthiness properties For the state machines this translation has been automated Via an interface to STATEMATE Informal text and proof conditions for checking healthiness properties also generated (more later) For tabular requirements, translation currently manual Based on ‘templates’ No reason why this can’t also be automated

13. 13 Example Z Specification Formal specification of DoorDeployed Inputs Output Assumption Guards Definitions

14. 14 Summary of talk Background and objectives of work Translating domain specific notations into formal specifications Automatic analysis of “healthiness” properties Automated test design Results Conclusions

15. 15 Specifying Healthiness Properties Objective: Remove errors from the specification before they are propagated into the code or tests Specification validation can include review, animation and formal analysis Can the formal analysis be automated? Using same translation used to generate the formal specification, healthiness properties in the requirements are translated into proof obligations Example: Completeness of reactive component Example: State determinism

16. 16 Checking Completeness Conditions Completeness of DoorDeployed reactive component Proof fails

17. 17 Checking Determinism Conditions Determinism of Stowing state Proof fails

18. 18 Automating the analysis CADiZ (Computer Aided Design in Z) Interactive theorem proving assistant General purpose tactics can be written as ‘scripts’ Includes a number of constraint solvers Checking proof tactics Attempts to simplify the conjecture to  True - property maintained by the specification  False - property not maintained by the specification Counter-example proof tactics If False, use constraint solving to generate an illustrative counter-example  Model checkers  Simulated annealing

19. 19 Automating the Analysis For completeness and determinism checks a high level of automation has been achieved Same 2 proof tactics used for 136 conjectures Due to:  Restricted structures of the conjectures  Powerful constraint solvers Performing the analysis Using an API to CADiZ, “Healthy” tool automatically invokes the proof tactics Reports success or counter-example in an easy to understand format The actual proof steps can also be displayed for review

20. 20 ======================================================================= Healthiness condition:- Proof conjecture for the completeness of DoorStowed. Demonstrates the completeness of the following guards: Guard1 and Guard 2. -------------------------------------------------------------------------------- Goal Proven! ======================================================================= Healthiness condition:- Proof conjecture for the completeness of DoorDeployed. Demonstrates the completeness of the following guards: Guard1 and Guard 2. -------------------------------------------------------------------------------- Goal Disproved! Attempting to generate a counter-example... Counter example:- DeployedPosition=85 RamPosition=85 Hinge=80 DeployLock=Activated Requirements Validation Example

21. 21 Summary of talk Background and objectives of work Translating domain specific notations into formal specifications Automatic analysis of “healthiness” properties Automated test design Results Conclusions

22. 22 Category-Partitioning Objective: Generate test cases to verify the implementation based on an explicit specification of the functionality Z operations form the basis of the test specifications The specifications are partitioned into test cases based on well understood test heuristics Approach based on category-partitioning and equivalence classes Input space partitioned into domains that exhibit similar behaviour in the specification - Equivalence classes Only a selection of data from each partition is needed to show that the implementation satisfies the specification for all data in that class  Assuming equivalence class hypothesis holds

23. 23 Automating Test Case Generation Generic partitioning tactics are specified as lemmas ‘Cut’ into the specification using proof tactics Instantiated with parameters of operation to partition Left as a side condition to prove E.g. Disjunction partitioning Constraint solving techniques then used to generate test data Model checking, optimisation-based search techniques A proof tactic selects best method to apply AdaTEST test script generation Still need some revisions to reflect refinement between specification and code Test rationale can be added as comments at any stage By expanding the auto-generated specification description

24. 24 Pros and Cons Advantages of this method New heuristics can be added without adding any code Properties of the heuristics can be proven (completeness, disjointness of tests) Same proof tactic can then be used to prove same properties of resultant test cases = test generator qualification Relationship between specification level criteria and code level criteria can be investigated. E.g. MC/DC coverage Automation gives the potential for large empirical studies Disadvantages Invalid state space behaviour? Testing heuristics need to be formally specified what about intuition?

25. 25 Example Partitioning Partitioning tactic: Boundary Value Analysis

26. 26 Example Partitioning

27. 27 Example Partitioning

28. 28 Summary of talk Background and objectives of work Translating domain specific notations into formal specifications Automatic analysis of “healthiness” properties Automated test design Results Conclusions

29. 29 Results

30. 30 Results

31. 31 Results Requirements errors detected All caught by the determinism and completeness checks  I.e. focused on particular properties of the specification Two types of requirements errors found: Errors introduced by the engineer on modelling some higher level requirements  E.g. omitting a condition or defining a condition badly  Greatest proportion of errors found of this type Errors as a manifestation of shortcomings in the higher level requirements  E.g. Badly defined or omitted requirements  Fewer errors of this type found but were considered more important

32. 32 Results What’s been automated so far Translation of Statecharts to Z and associated proof obligations Proof and counter-example generation for completeness and determinism conditions (for Statecharts and Tables)  Intuitive front-end  Details of proof are recorded for review Partitioning of operations into test cases Test data generation and associated AdaTEST test scripts What can still be automated Translation of PFS tables to Z and associated proof obligations Automatic selection of partitioning strategies to generate tests to satisfy particular coverage criteria

33. 33 Summary of talk Background and objectives of work Translating domain specific notations into formal specifications Automatic analysis of “healthiness” properties Automated test design Results Conclusions

34. 34 Conclusions Formal methods well integrated into an existing modelling environment High level of automation achieved Requirements errors detected at phase in which they were introduced Success made possible by targeting the formal methods to a particular domain and subset of problems Completeness/Determinism checks Test case generation General-purpose formal analysis tools used Same process can be applied to different requirements notations given some translation to the formalism

35. 35 Conclusions Generic method for applying formal methods to V&V based on engineering notations Use intuitive domain applicable notations with fixed semantics Couch healthiness properties as mathematical constraints Formally specify the behaviour under test Exploit existing tool support Complement the formal specification, proof obligations and tests with informal text