1 Reducing the Software Impact to System Safety Paul Mayo – SafeEng Limited

4 Contents System safety engineering Software’s contribution to system safety Software safety techniques

5 Safety –Ability of a system to avoid causing an accident Accident –Unintended event or sequence leading to harm Hazard –Situation that can lead to an accident Risk –Product of severity and likelihood of an undesired event Some Definitions

6

7 Accidents Highly Publicised System Failures Chernobyl Challenger Zeebrugge Piper Alpha Clapham / Southall / Ladbroke Grove Highly Publicised Software Failures Ariane 5 USS Yorktown V22 Osprey Patriot

8 Standards and Guidance Fundamental rethink Why comply with the standards? –To gain certification –Protection against personal liability –Protection against product liability –Protection against criminal law –To produce a safe system

9 Defence Standard History First standard in series was (begun in 1987) Developed because of concerns over emergence of safety-critical software Formal methods were receiving much support at the time So 00-55/1 was “technology forcing” FM standard

10 But when is software safety-critical? developed to answer this But safety is a SYSTEM property –so covered whole system and hardware as well and were first published as a pair of interim standards in 1991 Defence Standard History

11 The new standard is goal based That is, it says what you must do, not how you must do it The top-level goals are: 1. Identify the safety requirements 2. Show that the safety requirements are met Defence Standard - Current

12 Safety Requirements

13 So What’s Different About Software? Software doesn’t kill…. –Causes other devices to do it instead!

14 So What’s Different About Software?

15 So What’s Different About Software? Software doesn’t kill…. –Causes other devices to do it instead!

16 So What’s Different About Software? Software doesn’t kill…. –Causes other devices to do it instead!

17 Scope Safety related software –Concentrates on safety related software –Equal applicability to other critical software: Mission Security Business –Sometimes referred to as High Integrity Software –Safety should have the lead, but mission critical software may be the downfall of a major project

18 Scope All software of all provenance: –Bespoke –Legacy –SOUP –COTS –ASICs

19 So What’s Different About Software? Difficult to predict and quantify failure rates –Hardware - random failure –Software - systematic errors

20 Time Failures Random Failures

21 Time Failures Systematic Failures

22 Overview - key requirements All credible hazards and accidents are identified, the associated accident sequences are defined and the risks associated with them are determined Defect/failure reports and incident/accident/near-miss reports are monitored and remedial actions identified and implemented

23 Platform Systems Units Requirements Design & Decomposition Implementation Integration and Test Delivered Product Preliminary Hazard Analysis System Hazard Analysis System Safety Assessment Safety Case System Hazard Analysis V Life Cycle

24

25 Errors, Faults and Failures We want predictable software In designing and developing software, humans make errors These errors result in faults in the software If the faulty code is executed, this results in a failure

26 Software Assurance Primary causes of control system failure

27 What’s Different About Software? Why use it? –Flexibility –Functionality boom – complex systems –It’s (almost) free to reproduce Software is… –Inclusive of the design –A non tangible artefact

28 Software Assurance So how do you know the software is correct? –How do you stop faults manifesting themselves? Either –Don’t put the faults in (process)

29 Process Assurance The more rigorous the development, the less likely errors are Design reviews throughout ensure correct transition between phases Includes the use of formal methods to ensure correctness Extends to choice of language

30 Process Assurance Various ‘schemes’ attempt to provide an assurance of process –ISO , certification of QA process for software –Capability maturity model (CMM) –ARPs and DO178b, DO254...etc Aim to give a level of confidence about ability to develop a product –Not a guarantee about a specific product Need to back these up with involvement in product reviews

31 Software Assurance So how do you know the software is correct? –How do you stop faults manifesting themselves? Either –Don’t put the faults in (process) –Stop faults causing problem (fault tolerance)

32 Fault Tolerance Design system such that faults do not result in critical failures Use hardware to guard against software failure Design the software to tolerate faults –N-version programming –Recovery blocks –Exception handling –Defensive programming –Graceful Degradation –Fail-safe Design

33 Critical Failure

34 Software Assurance So how do you know the software is correct? –How do you stop faults manifesting themselves? Either –Don’t put the faults in (process) –Stop faults causing problem (fault tolerance) –Take them out at the end (test/analysis)

35 Software Testing/analysis Can not test every possible state of complex software Satisfactory operation for 1000 hrs gives –50% confidence of a further 1000 hrs operation –90% confidence of 100 hrs satisfactory operation To test complex software to 90% confidence of 1x10-6 per year failure rate would take approximately yrs

36 Software Testing/analysis Design out hazards through iterative analysis process at the earliest stages Testing/analysis of modules & integration Formal proof Static code analysis Black box / white box testing Accept that 100% testing of software is infeasible

37 Software Assurance So how do you know the software is correct? –How do you stop faults manifesting themselves? Either –Don’t put the faults in (process) –Stop faults causing problem (fault tolerance) –Take them out at the end (test/analysis) –Or all of the above...

38 What if you get it WRONG?

39 Safety Analysis is required to: –Underpin the safe development of a safe product –Provide process guidelines Variety of tools and techniques but be aware… Conclusion

40 Conclusion results of testing may not always be as they seem…

41 Paul Mayo Safety Consultant SafeEng Limited Questions?