BS3909 Week 8 1 Self-Study: Safety-critical systems l Wide range of equipment now computer-controlled »Machine could injure operator if certain faults happen »Sewage must not be discharged in rising tide »Traffic lights should never go green all-round »Drug-delivery drip mustn’t give over- or under-dose »Nuclear reactor needs accurate and timely control »Radiotherapy equipment must give right dose »Fly-by-wire aircraft mustn’t be crash-by-silicon l Require far higher standards of reliability that we are used to in the rest of the industry l Emphasis on proving correctness rather than testing

BS3909 Week 8 2 Risk Assessment l Not a precise (or even a logical) science »We take bigger risks when we’re “in control” l In reality, we need to combine two factors: »Probability that an undesired event will occur »Severity of impact if it does l Examples of risk/impact/precaution »Steel toe-caps protect carriers against fairly likely event of dropping object on their feet (rarely fatal) »Head restraints protect us against somewhat less likely event of being shunted (sometimes fatal) »Duplicated systems guard against very unlikely event of aircraft crash (often fatal)

BS3909 Week 8 3 Regulation & Standards See for more details l Underlying problem is that systems are designed with more concern about what they should do than about what the must not do »We tend to test what we programmed in »Not what we did inadvertently l Regulation can be based on: »Responsibility (professional and legal) »Observance of standards »Certification or licensing

BS3909 Week 8 4 Two Approaches to Standards l Mandate design technology »For example, insist on formal specifications »Or use of strongly-typed language such as Ada »Or even Formal Methods (mathematically-provable software design) OK until new and better technology arises l Define performance standards »Say what must and must not happen »Doesn’t limit design to current technology But very hard to test or prove performance

BS3909 Week 8 5 Achieving Standards l Certification of practitioners »Ensure that people designing safety-critical systems know what they are doing, and that »they don’t exceed the limitations of their expertise »Use an uncertified practitioner at your own peril l Certification of systems »To ensure compliance with Codes of Practice »Involves audit that procedures have been followed »That testing has been thorough l Licensing practitioners »Goes beyond certification by outlawing unlicensed practitioners (so if you lose licence you lose job)

BS3909 Week 8 6 Legal Issues l Engineers have normal HASAW responsibilities, and l Liable for Negligence if proved »Deviation from standards or codes may provide proof »Engineers expected to warrant expected result l + (probably) responsibility under Consumer Protection »No need to prove negligence »If a product hurts someone through malfunction, there is a case to answer »Software may be held to be an inherent component of the product l Best to insure against these liabilities if you can

BS3909 Week 8 7 Building Safe Systems l Safety must be key objective throughout development cycle »Specifications should consider safety implications –Should be provably consistent –Or design may ignore inconsistent system states »Design should focus on safety issues –Easier to design safety in than to bolt on later –“Impossible” states must still be handled »Provable mapping from design to coding »Testing to include consideration of malfunction l Hazard analysis needed to home in on safety Hoffnung

BS3909 Week 8 8 Hazard analysis l Need to recognize areas of risk to home in on safety l Most disasters are concatenation of minor faults »Operator misunderstands interface »Events are missed when they are simultaneous »System operates prematurely; or late; or doesn’t stop l Fault Tree Analysis »Defines possible undesired events »Then looks at how they might arise »Tracks down subsystems to see if they could occur l Alternative is by design inspections

BS3909 Week 8 9 Specifications l There is an inherent problem with specifications »Plain English is too imprecise »Requirement is often owned by non-specialist (remember last week’s issues?) »So jargon may be misunderstood »Rigorous mathematical specification even harder for non- specialist to evaluate (but at least the engineer knows that it’s consistent) »Producing safety specification concentrates the mind »Ensures that possible failure modes are considered »That fail-safe actions are specified for them »Can prescribe external safeguards

BS3909 Week 8 10 Safety Design l Need to prevent hazards when possible »e.g. by interlocks and mutually-exclusive processing l Minimize likelihood or scale of hazard »e.g. by automatic control, negative feedback »Comparable with safety-stop on lifts l Detect hazards if they occur »e.g. by monitoring, warning devices l Minimize impact of hazard when it is detected »e.g. by emergency stop, recovery process »and training staff to react correctly

BS3909 Week 8 11 Practical Approaches l Rigorous specification »Mathematical definition of intended behaviour l Standards and procedures for development cycle »Enforces code of practice »Avoids risk from uncontrolled “no impact” changes l Minimal interaction between components »If components have simple functions, their correctness can be proved »Then assembly of components can also be proved l Redundancy »Guard against single point of failure