Considerations in the Preference for and Application of RTCA/DO-178B in the Australian Military Avionics Context SQNLDR Derek Reinhardt Systems Certification and Integrity (SCI) Directorate of Aircraft Engineering (DAIRENG)

Overview Introduction to criticisms of RTCA/DO-178B Background and structure of DO-178B Criticisms that RTCA/DO-178B is insufficient Criticisms that RTCA/DO-178B is too onerous ADF’s preference for RTCA/DO-178B Application issues of RTCA/DO-178B in the Australian Military Avionics Context

Introduction RTCA/DO-178B is the centre of much debate or criticism – insufficient, too onerous, etc Avoidable software failures have already been responsible for aircraft mishaps cockpit displays go blank engines throttle back during takeoff contradictory airspeed readings This presentation/paper examines these criticisms how do they influence the ADF’s preference for and application of RTCA/DO-178B

ADF Preference for RTCA/DO-178B RTCA/DO-178B is the ADF’s preferred software assurance standard for safety critical and safety related airborne software development Refer AAP7001.054 Airworthiness Design Requirements Manual ADF recognises the FAA processes and standards widely used and accepted by many international airworthiness authorities Many military aviation systems have a civil heritage with software developed under RTCA/DO-178B AEW&C B737, MRTT A330, etc

Background of RTCA/DO-178B 14 CFR 25.1309 often referred to as FAR 25.1309 SAE ARP 4754 and SAE ARP 4761 aircraft level FHA, FTA system level FMEA, FTA, etc. common cause analysis – PRA, ZSA, CMA RTCA/DO-178B for software design assurance RTCA/DO-254 for hardware design assurance Five failure condition severities are assigned design assurance levels (DALs) Catastrophic (A), Hazardous / Severe Major (B), Major (C), Minor (D), No Effect (E)

Structure of RTCA/DO-178B 66 objectives in 10 tables + several implicit objectives satisfaction tailored by software level several prescriptive objectives statement coverage, decisions coverage, modified condition decision coverage Lifecycle phases planning, development, requirements, design, coding and integration, verification Integral processes configuration management, quality assurance Certification Authority liaison RTCA/DO-248B, FAA Order 8110.49, CAST5

Intricacies of DO-178B Common misconception that RTCA/DO-178B completely specifies the process and activities Yes – objectives are coupled to the software lifecycle Yes – they don’t distinguish between requirements functional, software safety, etc Fidelity of the objectives presents challenges for COTS and PDS With exception of 3 objectives – flexibility is permitted in how the objectives are satisfied Objectives can be broadly classified as contributing to requirements validity, requirements satisfaction, and requirements traceability

Criticisms of RTCA/DO-178B Divided into several positions those that believe RTCA/DO-178B is insufficient academics, researchers, or consultants those that believe RTCA/DO-178B is too onerous development contractors with cost and schedule driven imperative Criticisms are at odds with each other central to the criticisms, then is it about right? Widely accepted by FAA, EASA NTSB reports that it is effective in those contexts How does the ADF address the criticisms?

RTCA/DO-178B is Insufficient Absence of mandatory formal methods Absence of mandatory static code analysis Ineffectiveness of MC/DC testing Absence of specific requirements or objectives relating to software safety analysis and software safety requirements Assumptions underlying the design assurance level definition are questionable

Absence of Mandatory Formal Methods Does not prohibit formal methods acceptable method to satisfy objectives Application to problem domain not universally applicable to problem domains and technologies used in critical systems only partially applicable to problem scope are closed languages, no inherent real world meaning, natural language is still required Formal methods and safety does not address significant sources of error WRT the safety of systems little evidence that it would prevent a number of mishaps attributable to software Complementary to testing testing is required to demonstrate real world behaviours, on real hardware, in the target environment Formal methods is not the ‘silver bullet’ for safety software

Absence of Mandatory Static Code Analysis Does not prohibit static code analysis it is an acceptable method used to satisfy objectives Static analysis won’t find all the faults (requirements and implementation) most related to the safety of the software Permits greater focus on those activities related to identifying requirements validity and satisfaction problems affecting safety prevents developers being overwhelmed in code reviews and testing identifying inadvertent implementation problems, which static code analysis tools readily detect Limitations to applying static analysis retrospectively

Ineffectiveness of MC/DC Testing Exercise each data flow that directly affects a control flow to identify as many faults as possible Widely debated objective studies confirm that MCDC does find faults that other DO-178B testing approaches do not find other studies found “no significant difference” RCDC address some of these limitations MCDC not finding additional faults is not the concern if normal and robustness testing has been comprehensive, then it is possible that the gap in MCDC will be small, and NOT safety related adequacy of normal and robustness testing

Absence of Specific Requirements or Objectives Relating to Software Safety Analysis and Software Safety Requirements Is not explicit in objectives relating to software safety analysis and software safety requirements but they are not absent! number of objectives contribute to requirements validity, including that of safety requirements However, systematic software safety analysis are not always proposed to show that the identified and allocated set of software safety requirements is complete and correct DGTA recommends an IEEE1228 Software Safety Plan be used to document the planned software safety activities – and their outcomes or the RTCA/DO-178B PSAC can be used

Assumptions Underlying the Design Assurance Level Definition are Questionable Issues with integrity/assurance levels little evidence that software of different integrity levels does have failure rates of integrity level order debate regarding the philosophy and rules for allocating integrity levels significant differences in the processes recommended by standards Inconsistent application misunderstanding of the differences between reviews versus analyses some objectives being presumed to be satisfied solely by reviews intent is combination of analysis and reviews of the outputs variable normal and robustness testing software architecture is appropriate avoids design constructs that would not comply with system safety objectives software safety requirements are identified/allocated Apportionment and adequacy of objectives objectives fundamental to the validity and satisfaction – from Level C Level A and B provide additional evidence - trustworthiness

RTCA/DO-178B is Too Onerous Excessive requirements specification and traceability fidelity requirements Excessive verification requirements

Excessive Requirements Specification and Traceability Fidelity Requirements RTCA/DO-178B motivation for fidelity each behaviour that constitutes the requirements at level of abstraction is systematically accounted for design tool to assist developer produce a good design Why should all the behaviours be accounted for? evidence that all behaviours of the software are acceptable do not lead to unacceptable failure conditions all the behaviours of the software should be disclosed permits reasoning about their suitability for the safety of the system arguing non-interference with the behaviours that are important to the safety of the system Questionable disagreements Intellectual Property constraints software development is a creative process, not itself compatible with such rigour requirements

Excessive Verification Requirements Testing will always be required to gain confidence in the behaviour of software on the target hardware in the intended operating environment Completion of testing defensible engineering argument as to why testing is complete with evidence to support it Not based on the following factors: cost and schedule consensus of program managers broad consensus of the programmers and testers any other non-engineering based arguments RTCA/DO-178B provides a defensible argument as to when testing is complete by specifying: requirements completeness criteria requirements coverage criteria extent of normal and robustness tests extensiveness of requirements based low level testing coupled with additional implementation related coverage criteria (structural coverage) to elicit certain properties

Application of RTCA/DO-178B RTCA/DO-178B is not applied in isolation Test coverage objectives Use of RTCA/DO-178B as a benchmark for assessing software assurance practices COTS with RTCA/DO-178B Migrating to RTCA/DO-178B

Not Applied in Isolation System Safety Program - MIL-STD-882C or FAR 25.1309 Key Issues A Software Safety Program (SwSP) should be established to coordinate hazard identification and mitigation efforts for hazards with software-related causal factors. IEEE1228 Software Safety Plan Software Safety Analysis, Generic Software Safety Requirements A Software Assurance standard is required for the development of all software that is safety related Software process standards should be applied to the development of software for airborne and related systems An assessment of the applicant’s software development capability, including domain knowledge, should be conducted as part of the tender evaluation and contract negotiation process examines the safety culture determine if non-systematic (experience) activities can be trusted

Test coverage objectives Some organisations believe that DGTA does not require structural code analysis – by default – THIS IS NOT TRUE! In cases where DGTA has not required it additional activities to ensure that the coverage is comprehensive fidelity of requirements explicitness of traceability extent of normal and robustness testing additional activity to identify dead and deactivated code often exception related code mission systems, with no series hazards extenuating circumstances – legacy constraints Negotiate through the PSAC –expect to have a compelling argument if you are going to proposed not doing it

Software Assurance Benchmark Assess software products and their development practices software agencies that do not use RTCA/DO-178B confused that DGTA is applying DO-178B where not contracted RTCA/DO-178B objectives help with the assessment of requirements validity, satisfaction, and traceability RTCA/DO-178B provide clear measures of fidelity in the specification and traceability of requirements no gap between required behaviours and executable code all software behaviours are appropriate with respect to safety extent of test based verification of requirements and implementation on the target computer in the intended environment development and review rigour independence and oversight assures that the evidence presented contains an acceptably small number of faults

COTS with RTCA/DO-178B COTS fall under the scope of PDS PDS criteria are demanding, but not excessive but often the evidence is just not available Alternate proposals for use of COTS Provide evidence to demonstrate the following types of properties Partitioning (containment and/or mediation) Non-Interference Acceptable Behaviours

Migrating to DO-178B Acquired where no software assurance standard applied DOD-STD-2167A, MIL-STD-498, IEEE12207 Two options – transition to RTCA/DO-178B or develop a Software Assurance Task Matrix FAA guidelines Notice 8110.49 Chapter 10 Intended for systems developed to RTCA/DO-178 and RTCA/DO-178A Careful management of the scope of retrospective production of software assurance evidence is required DGTA has assessed the approach as acceptable

Summary Introduction to criticisms of RTCA/DO-178B Background and Structure of DO-178B Criticisms that RTCA/DO-178B is insufficient Criticisms that RTCA/DO-178B is too onerous ADF’s preference for RTCA/DO-178B Application of RTCA/DO-178B in the Australian Military Avionics Context RTCA/DO-178B applied within the ADF framework addresses relevant criteria for producing safety software systems

Questions ?