1. www.QinetiQ.com © Copyright QinetiQ limited 2010 Formal Methods Tool Qualification for DO178B & DO178C Nick Tudor tel: +44 1684 894489 email: njtudor@qinetiq.com

2. www.QinetiQ.com © Copyright QinetiQ limited 2010 Agenda Tool Overview and Qualification Approach The Current Guidance - DO178B Qualification Considerations – DO178B Applicant’s Claims – DO178B Qualification Considerations – DO178C Applicant’s Claims – DO178C and Supplements Summary

3. www.QinetiQ.com © Copyright QinetiQ limited 2010 Tool Overview and Qualification Approach

4. www.QinetiQ.com © Copyright QinetiQ limited 2010 Overview of the tool The tool is a Formal Methods based software verification tool −Automates the verification of automatically generated code −Constraints include a subset of Simulink and a subset of the Ada programming language Exploits the Z formal specification language −Automatic generation of formal specification of the Simulink in Z −Gives required independence Uses Carroll Morgan’s Theory of Refinement, also automatic Built in automated use of an off-the-shelf Theorem Prover (ProofPower) Having used the tool manually for independent code verification, we aimed to: −Automatically prove automatically generated code for productivity reasons −Understand the tool qualification issues

5. www.QinetiQ.com © Copyright QinetiQ limited 2010 Overview of the CLawZ Process Z Discharge proof Ada Refinement Script Generator Supertac Z Producer Compliance Notation Tool ProofPower Verification Conditions B4S Simulink Development Verification User Interface

6. www.QinetiQ.com © Copyright QinetiQ limited 2010 The GUI and some functions (PID example) Importing Simulink & Ada files & defining units Creating Z specification, doing refinement and proof Creating Interface for each unit Defining variables Proof results

7. www.QinetiQ.com © Copyright QinetiQ limited 2010 The ‘pre-qualification’ approach As part of the final development steps, we took advice from CAA This is new because: −No-one has previously approached them to do any form of “pre-qualification” of a tool −NB This does not constitute certification authority pre-approval of this tool −It’s a formal methods tool −A formal methods tool which is intended to only automate code verification [and will be qualified as such] We needed to examine the guidance from DO178B to ensure that we knew what we were aiming at and to check likely impact of DO178C −Particularly in light of both the Formal Methods and Tool Qualification Supplements

8. www.QinetiQ.com © Copyright QinetiQ limited 2010 RTCA DO178C/EUROCAE ED12C Tool Qualification Supplement How high is the bar? RTCA DO178B/EUROCAE ED12B Minimum QinetiQ Internal Processes RTCA DO178B/EUROCAE ED12B “Plus”

9. www.QinetiQ.com © Copyright QinetiQ limited 2010 A virtual, parallel approach Tool Development Theoretical Applicant DO178B Theoretical DO178C Includes (theoretically): Core text, Tool Qualification, FM Supplement MBD Supplement Theoretical System

10. www.QinetiQ.com © Copyright QinetiQ limited 2010 The Current Guidance - DO178B

11. www.QinetiQ.com © Copyright QinetiQ limited 2010 Not Shooting for the Minimum RTCA DO178B states that for a Verification tool: −“The qualification criteria for software verification tools should be achieved by demonstration that the tool complies with its Tool Operational Requirements under normal operational conditions” −Production of Tool Operational Requirements, the Verification Plan, the Verification Results and the Tool Accomplishment Summary NB FAA-8110-49 says you could also have a TQP and TAS, but specifically: −“Tool Operational Requirements satisfies the same objectives as the Software Requirements Data of the airborne software”. …ie (from 12.2.3.2): −“Tool Operational Requirements describe the tool's operational functionality. This data should include: −a.A description of the tool's functions and technical features. [For software development tools, it includes the software development process activities performed by the tool]. −b.User information, such as installation guides and user manuals. −c.A description of the tool's operational environment. −d.[For software development tools, the expected responses of the tool under abnormal operating conditions.] ”

12. www.QinetiQ.com © Copyright QinetiQ limited 2010 Adapted from FAA-8110-49 Fig 9-2 DataApplicability Available/Submit/NoteRTCA/DO-178B Ref. Plan for Software Aspects of Certification (PSAC) Verification & DevelopmentSubmit12.2, 12.2.3.a, & 12.2.4 Tool Qualification PlanDevelopmentSubmit12.2.3.a(1), 12.2.3.1, & 12.2.4 Tool Operational Requirements Verification & DevelopmentAvailable12.2.3.c(2) & 12.2.3.2 Software Accomplishment Summary (SAS) Verification & DevelopmentSubmit12.2.4 Tool Accomplishment Summary DevelopmentSubmit12.2.3.c(3) & 12.2.4 Tool Verification Records (for example, test cases, procedures, and results) Verification & DevelopmentAvailable12.2.3 Tool Qualification Development data (for example, requirements, design, and code) DevelopmentAvailable12.2.3

13. www.QinetiQ.com © Copyright QinetiQ limited 2010 Qualification considerations – DO178B

14. www.QinetiQ.com © Copyright QinetiQ limited 2010 An Applicant’s perspective We needed to consider how an Applicant would use the tool and subsequently be able to claim credit −Despite confused guidance, what artefacts would need to be made available −Given that this is a tool based upon FM, what “language” to use (this is non-trivial!) We needed to be cognisant of context, ie the overall software development life cycle We have defined CLawZ Simulink Subset (CSS) used for Independent Verification and used a well known subset of code (Ada) −This constrains the possible input space to something that is well defined, but also very flexible −Also constrained the output space (code) By doing this we set the Tool Operational Requirements and hence the scope of the qualification

15. www.QinetiQ.com © Copyright QinetiQ limited 2010 Proposed approach Histor y Upgrade/ production of Plans Configuration Control Tool Accomplishment Summary Tool Qualification Plan Applicants PSAC Verification Plan Verification Results Tool Operational Requirements Applicants SAS Configuration Index Verification Cases and Procedures QA and Technical Audit Available information includes: Development Plan Configuration Management Plan QA Plan/records (not for 178B) Design Description Configuration Records Development Environment Index User Guide

16. www.QinetiQ.com © Copyright QinetiQ limited 2010 Applicant’s Claims – DO178B

17. www.QinetiQ.com © Copyright QinetiQ limited 2010 Verification Objectives – DO178B High-Level Requirements (A-2: 3, 4, 5) Executable Object Code Source Code (A-2: 7) (A-2: 6) System Requirements (A-2: 1, 2) A-6.1 Compliance A-6.2 Robustness A-5. 7 Complete & Correct A-3.1 Compliance A-3.6 Traceability A-3.2 Accuracy & Consistency A-3.3 HW Compatibility A-3.4 Verifiability A-3.5 Conformance A-3.7 Algorithm Accuracy A-4. 8 Architecture Compatibility A-4.1 Compliance A-4.6 Traceability A-4.9 Consistency A-4.10 HW Compatibility A-4.11 Verifiability A-4.12 Conformance A-4.13 Partition Integrity Software Architecture Low-Level Requirements A-4.2 Accuracy & Consistency A-4.3 HW Compatibility A-4.4 Verifiability A-4.5 Conformance A-4.7 Algorithm Accuracy A-5.2 Compliance A-5.1 Compliance A-5.5 Traceability A-5.3 Verifiability A-5.4 Conformance A-5.6 Accuracy & Consistency A-6.3 Compliance A-6.4 Robustness A-6.5 Compatible With Target Compliance: with requirements Conformance: with standards NB Verification of the Verification, QA, DER liaison and other documentation also required

18. www.QinetiQ.com © Copyright QinetiQ limited 2010 Claims relating to source code verification Table A-5 ObjectiveApplicability by Software Level DescriptionRef.ABCD 1Source Code complies with low-level requirements. 6.3.4a  2Source Code complies with software architecture. 6.3.4b  3Source Code is verifiable.6.3.4c  4Source Code conforms to standards.6.3.4d  5Source Code is traceable to low-level requirements. 6.3.4e  6Source Code is accurate and consistent. 6.3.4f  7Output of software integration process is complete and correct. 6.3.5  Partial We can show that eg there are no uninitialised or unused variables or constants. There are no claims re stack usage, resource contention, WCET, etc

19. www.QinetiQ.com © Copyright QinetiQ limited 2010 Verification Objectives System Requirements High-Level Requirements Source Code Executable Object Code (A-2: 3, 4, 5) (A-2: 7) (A-2: 6) (A-2: 1, 2) Simulink [Continuous] Simulink [Discrete] Simulink for CLawZ Autocode, proof Major Contribution Software Architecture Low-Level Requirements A-5.3 Verifiability A-5.4 Conformance A-5.6 Accuracy & Consistency A-5.1 Compliance A-5.5 Traceability A-5.2 Compliance A-5. 7 Complete & Correct

20. www.QinetiQ.com © Copyright QinetiQ limited 2010 Qualification considerations – DO178C

21. www.QinetiQ.com © Copyright QinetiQ limited 2010 DO178C introduces 3 categories of tools (see section12.2.2) Development toolCriteria 1 Criteria 3Verification tool DO178BDO178C DALCriteria 1Criteria 2Criteria 3 ATQL1TQL4TQL5 BTQL2TQL4TQL5 CTQL3TQL5 DTQL4TQL5 Criteria 2 ? Criteria 1: A tool whose output is part of the airborne software and thus could insert an error. Criteria 3: A tool that, within the scope of its intended use, could fail to detect an error. Criteria 2: A tool that automates verification process(es) and thus could fail to detect an error, and whose output is used to justify the elimination or reduction of: 1. Verification process(es) other than that automated by the tool, or 2. Development process(es) that could have an impact on the airborne software.

22. www.QinetiQ.com © Copyright QinetiQ limited 2010 DO178C Tool Qualification Supplement We wanted to know what we might be under ED12C/DO178C we we’re either −A verification tool (Criteria 3) and hence TQL5 would be required, or −A super verification tool (Criteria 2) and hence TQL5 or even TQL4 We believe CLawZ would be a TQL5 tool, but this depends largely on what the Applicant will wish claim as a result of using the tool And whether this is acceptable for the certifier More on this at the end… We will also examine TQL4 …just in case!

23. www.QinetiQ.com © Copyright QinetiQ limited 2010 We needed to keep and eye on what TQL4 required at the same time Referring to Table T-1 and T2, there is considerable extra material required over DO178B No requirement for Tool Operational Requirements, but is implicit Interesting that QA Plan appears to be required, but not QA records Also, that Tool Plans do not have to comply with the Supplement (T-1-6) DO178C TQL 4 – Tables T-1 & T-2 Document178BTQL4Note The Tool Qualification Plan (TQP)YY Tool Development Plan (TDP)NYNot strictly needed, but certifier will look for it Tool Verification Plan (TVP)YYNot entirely clear if this is needed, but… Tool Configuration Management Plan (TCMP) NYNot entirely clear if this is needed, but the Configuration Index is required Tool Quality Assurance Plan (TQAP)NYNot needed, but needed for eg TickIT?

24. www.QinetiQ.com © Copyright QinetiQ limited 2010 TQL4 – DO178C – Tables T-3, T-4 and T-5 There is large concentration of effort in Table T-3 (requirements) which is not “needed” under DO178B −It is implied that the Tool Operational Requirements are the “requirements”. Suspect that in practice, for all but trivial tools under qualified under DO178B, there is a more clearly defined set of requirements contained within the TOR It was not immediately clear why the only requirement for Table T4 is T- 4-10 (protection mechanisms). Sought guidance from FAQ #2 Tool Qualification Supplement : −Seeks to show evidence for separation of functions, essentially through architecture – fine, but… −There are no other architecture requirements for TQL 4 (ie compatible with requirements, consistency, standards), so does this objective help safety? No objectives for Table T-5 verification of the coding process…

25. www.QinetiQ.com © Copyright QinetiQ limited 2010 TQL4 - Tables T-4 and T-5 - A closer examination From DP #2: Rationale for Tool Qualification Criteria Definition, para 2.2.3.2 Rationale for introducing Criteria 2 “These additional uses of tools raise some concerns about the rigor required for qualification. For example: −From a safety perspective, the impact of these tools on the final software can vary. −The required confidence may not be able to be assessed by considering only the Tool Operational Requirements. −The maturity and soundness of a mathematical theory and its implementation may be necessary to provide the required confidence The major concern would appear to be the implementation, so no examination of low level requirements or the source code? Reliance is therefore left to Table T-6… NB No objectives are required to be met for TQL5

26. www.QinetiQ.com © Copyright QinetiQ limited 2010 TQL4 – ED12C/DO178C – Tables T-6, T-7, T-8 and T-9 Focus here is on how the tool functions when actually undertaking verification Examines object code with respect to the Tool Requirements −Checks compliance and robustness Also looks at verification results −Checks that results were correct/discrepancies explained and −Coverage of Tool Requirements Ensures that the configuration was identified

27. www.QinetiQ.com © Copyright QinetiQ limited 2010 Applicant’s Claims – DO178C and Supplements

28. www.QinetiQ.com © Copyright QinetiQ limited 2010 Possible claims relating to source code verification – DO178C FM Supplement Specific claims for formal verification could be made in accordance with the Formal Methods Supplement (see table FM A-5) −Specific claims can be made FM A5-1 to 6 (FM7 refers to core text as above) − Also need to examine possible to claims to Extra Objectives FM8-11 ObjectiveApplicability by SW Level DescriptionRef.ABCD FM 8Formal analysis cases and procedures are correct. FM.6.3.6a FM.6.3.6b  FM 9Formal analysis results are correct and discrepancies explained. FM.6.3.6c  FM 10Requirements formalization is correct. FM.6.3i  FM 11Formal method is correctly defined and sound. FM.6.2 

29. www.QinetiQ.com © Copyright QinetiQ limited 2010 Perspective of OTS tools for DO178C vs in-house tool development for DO178B Number of views from Tool Qualification Supplement −Re-use – see section 11.2 −OTS - see section 11.3 −Service History – see section 11.4 (not applicable) −Alternative methods – see section 11.5 (not applicable) Section 11.3.2 shows activities required by the tool developer −Which are to produce: TOR, TQP, TCI, TAS −Also outlines which objectives are modified Section 11.3.3 shows activities for the tool user −To assess the TOR, TQP, TCI and TAS and plan extra activities such as division of responsibilities

30. www.QinetiQ.com © Copyright QinetiQ limited 2010 Summary We believe we can support an Applicant in system certification to DO178B We have gone further than was required because this tool was “pre- developed” and is a formal methods tool We believe that it would be a TQL5 tool under DO178C Tool Qualification Supplement, but… Could be TQL4, depending on development processes of the Applicant Interesting that our approach has mirrored the COTS advice to tool qualification So does TQL4 achieve anything extra in terms of system safety?