1. Considerations in the Preference for and Application of RTCA/DO-178B in the Australian Military Avionics Context
SQNLDR Derek Reinhardt
Systems Certification and Integrity (SCI)
Directorate of Aircraft Engineering (DAIRENG)

2. Overview Introduction to criticisms of RTCA/DO-178B
Background and structure of DO-178B
Criticisms that RTCA/DO-178B is insufficient
Criticisms that RTCA/DO-178B is too onerous
ADF’s preference for RTCA/DO-178B
Application issues of RTCA/DO-178B in the Australian Military Avionics Context

3. Introduction
RTCA/DO-178B is the centre of much debate or criticism – insufficient, too onerous, etc
Avoidable software failures have already been responsible for aircraft mishaps
cockpit displays go blank
engines throttle back during takeoff
contradictory airspeed readings
This presentation/paper
examines these criticisms
how do they influence the ADF’s preference for and application of RTCA/DO-178B

4. ADF Preference for RTCA/DO-178B
RTCA/DO-178B is the ADF’s preferred software assurance standard for safety critical and safety related airborne software development
Refer AAP Airworthiness Design Requirements Manual
ADF recognises the FAA processes and standards
widely used and accepted by many international airworthiness authorities
Many military aviation systems have a civil heritage with software developed under RTCA/DO-178B
AEW&C B737, MRTT A330, etc

5. Background of RTCA/DO-178B
14 CFR
often referred to as FAR
SAE ARP 4754 and SAE ARP 4761
aircraft level FHA, FTA
system level FMEA, FTA, etc.
common cause analysis – PRA, ZSA, CMA
RTCA/DO-178B for software design assurance
RTCA/DO-254 for hardware design assurance
Five failure condition severities are assigned design assurance levels (DALs)
Catastrophic (A), Hazardous / Severe Major (B), Major (C), Minor (D), No Effect (E)

6. Structure of RTCA/DO-178B
66 objectives in 10 tables
+ several implicit objectives
satisfaction tailored by software level
several prescriptive objectives
statement coverage, decisions coverage, modified condition decision coverage
Lifecycle phases
planning, development, requirements, design, coding and integration, verification
Integral processes
configuration management, quality assurance
Certification Authority liaison
RTCA/DO-248B, FAA Order , CAST5

7. Intricacies of DO-178B
Common misconception that RTCA/DO-178B completely specifies the process and activities
Yes – objectives are coupled to the software lifecycle
Yes – they don’t distinguish between requirements
functional, software safety, etc
Fidelity of the objectives presents challenges for COTS and PDS
With exception of 3 objectives – flexibility is permitted in how the objectives are satisfied
Objectives can be broadly classified as contributing to requirements validity, requirements satisfaction, and requirements traceability

8. Criticisms of RTCA/DO-178B
Divided into several positions
those that believe RTCA/DO-178B is insufficient
academics, researchers, or consultants
those that believe RTCA/DO-178B is too onerous
development contractors with cost and schedule driven imperative
Criticisms are at odds with each other
central to the criticisms, then is it about right?
Widely accepted by FAA, EASA
NTSB reports that it is effective in those contexts
How does the ADF address the criticisms?

9. RTCA/DO-178B is Insufficient
Absence of mandatory formal methods
Absence of mandatory static code analysis
Ineffectiveness of MC/DC testing
Absence of specific requirements or objectives relating to software safety analysis and software safety requirements
Assumptions underlying the design assurance level definition are questionable

10. Absence of Mandatory Formal Methods
Does not prohibit formal methods
acceptable method to satisfy objectives
Application to problem domain
not universally applicable to problem domains and technologies used in critical systems
only partially applicable to problem scope
are closed languages, no inherent real world meaning, natural language is still required
Formal methods and safety
does not address significant sources of error WRT the safety of systems
little evidence that it would prevent a number of mishaps attributable to software
Complementary to testing
testing is required to demonstrate real world behaviours, on real hardware, in the target environment
Formal methods is not the ‘silver bullet’ for safety software

11. Absence of Mandatory Static Code Analysis
Does not prohibit static code analysis
it is an acceptable method used to satisfy objectives
Static analysis won’t find all the faults (requirements and implementation) most related to the safety of the software
Permits greater focus on those activities related to identifying requirements validity and satisfaction problems affecting safety
prevents developers being overwhelmed in code reviews and testing identifying inadvertent implementation problems, which static code analysis tools readily detect
Limitations to applying static analysis retrospectively

12. Ineffectiveness of MC/DC Testing
Exercise each data flow that directly affects a control flow to identify as many faults as possible
Widely debated objective
studies confirm that MCDC does find faults that other DO-178B testing approaches do not find
other studies found “no significant difference”
RCDC address some of these limitations
MCDC not finding additional faults is not the concern
if normal and robustness testing has been comprehensive, then it is possible that the gap in MCDC will be small, and NOT safety related
adequacy of normal and robustness testing

13. Absence of Specific Requirements or Objectives Relating to Software Safety Analysis and Software Safety Requirements
Is not explicit in objectives relating to software safety analysis and software safety requirements
but they are not absent!
number of objectives contribute to requirements validity, including that of safety requirements
However, systematic software safety analysis are not always proposed to show that the identified and allocated set of software safety requirements is complete and correct
DGTA recommends an IEEE1228 Software Safety Plan be used to document the planned software safety activities – and their outcomes
or the RTCA/DO-178B PSAC can be used

14. Assumptions Underlying the Design Assurance Level Definition are Questionable
Issues with integrity/assurance levels
little evidence that software of different integrity levels does have failure rates of integrity level order
debate regarding the philosophy and rules for allocating integrity levels
significant differences in the processes recommended by standards
Inconsistent application
misunderstanding of the differences between reviews versus analyses
some objectives being presumed to be satisfied solely by reviews
intent is combination of analysis and reviews of the outputs
variable
normal and robustness testing
software architecture is appropriate
avoids design constructs that would not comply with system safety objectives
software safety requirements are identified/allocated
Apportionment and adequacy of objectives
objectives fundamental to the validity and satisfaction – from Level C
Level A and B provide additional evidence - trustworthiness

15. RTCA/DO-178B is Too Onerous
Excessive requirements specification and traceability fidelity requirements
Excessive verification requirements

16. Excessive Requirements Specification and Traceability Fidelity Requirements
RTCA/DO-178B motivation for fidelity
each behaviour that constitutes the requirements at level of abstraction is systematically accounted for
design tool to assist developer produce a good design
Why should all the behaviours be accounted for?
evidence that all behaviours of the software are acceptable
do not lead to unacceptable failure conditions
all the behaviours of the software should be disclosed
permits reasoning about their suitability for the safety of the system
arguing non-interference with the behaviours that are important to the safety of the system
Questionable disagreements
Intellectual Property constraints
software development is a creative process, not itself compatible with such rigour requirements

17. Excessive Verification Requirements
Testing will always be required to gain confidence in the behaviour of software on the target hardware in the intended operating environment
Completion of testing
defensible engineering argument as to why testing is complete
with evidence to support it
Not based on the following factors:
cost and schedule
consensus of program managers
broad consensus of the programmers and testers
any other non-engineering based arguments
RTCA/DO-178B provides a defensible argument as to when testing is complete by specifying:
requirements completeness criteria
requirements coverage criteria
extent of normal and robustness tests
extensiveness of requirements based low level testing
coupled with additional implementation related coverage criteria (structural coverage) to elicit certain properties

18. Application of RTCA/DO-178B
RTCA/DO-178B is not applied in isolation
Test coverage objectives
Use of RTCA/DO-178B as a benchmark for assessing software assurance practices
COTS with RTCA/DO-178B
Migrating to RTCA/DO-178B

19. Not Applied in Isolation
System Safety Program - MIL-STD-882C or FAR
Key Issues
A Software Safety Program (SwSP) should be established to coordinate hazard identification and mitigation efforts for hazards with software-related causal factors.
IEEE1228 Software Safety Plan
Software Safety Analysis, Generic Software Safety Requirements
A Software Assurance standard is required for the development of all software that is safety related
Software process standards should be applied to the development of software for airborne and related systems
An assessment of the applicant’s software development capability, including domain knowledge, should be conducted as part of the tender evaluation and contract negotiation process
examines the safety culture
determine if non-systematic (experience) activities can be trusted

20. Test coverage objectives
Some organisations believe that DGTA does not require structural code analysis – by default – THIS IS NOT TRUE!
In cases where DGTA has not required it
additional activities to ensure that the coverage is comprehensive
fidelity of requirements
explicitness of traceability
extent of normal and robustness testing
additional activity to identify dead and deactivated code
often exception related code
mission systems, with no series hazards
extenuating circumstances – legacy constraints
Negotiate through the PSAC –expect to have a compelling argument if you are going to proposed not doing it

21. Software Assurance Benchmark
Assess software products and their development practices
software agencies that do not use RTCA/DO-178B
confused that DGTA is applying DO-178B where not contracted
RTCA/DO-178B objectives help with the assessment of
requirements validity, satisfaction, and traceability
RTCA/DO-178B provide clear measures of
fidelity in the specification and traceability of requirements
no gap between required behaviours and executable code
all software behaviours are appropriate with respect to safety
extent of test based verification of requirements and implementation on the target computer in the intended environment
development and review rigour
independence and oversight
assures that the evidence presented contains an acceptably small number of faults

22. COTS with RTCA/DO-178B COTS fall under the scope of PDS
PDS criteria are demanding, but not excessive
but often the evidence is just not available
Alternate proposals for use of COTS
Provide evidence to demonstrate the following types of properties
Partitioning (containment and/or mediation)
Non-Interference
Acceptable Behaviours

23. Migrating to DO-178B
Acquired where no software assurance standard applied
DOD-STD-2167A, MIL-STD-498, IEEE12207
Two options – transition to RTCA/DO-178B or develop a Software Assurance Task Matrix
FAA guidelines
Notice Chapter 10
Intended for systems developed to RTCA/DO-178 and RTCA/DO-178A
Careful management of the scope of retrospective production of software assurance evidence is required
DGTA has assessed the approach as acceptable

24. Summary Introduction to criticisms of RTCA/DO-178B
Background and Structure of DO-178B
Criticisms that RTCA/DO-178B is insufficient
Criticisms that RTCA/DO-178B is too onerous
ADF’s preference for RTCA/DO-178B
Application of RTCA/DO-178B in the Australian Military Avionics Context
RTCA/DO-178B applied within the ADF framework addresses relevant criteria for producing safety software systems

25. Questions ?