Software Safety CS3300 Fall 2015

Failures are costly ● Bhopal 1984 – 3000 dead and injured ● Therac – 6 dead ● Chernobyl / Three Mile Island ● Challenger Shuttle

Culture ● The general attitude and approach to safety reflected by those who participate in that industry. ● Accidents come from: ● Overconfidence and complacency – Three mile island – faith in equipment – Shuttle – routine operations ● Disregard or low priority for safety – Bhopal – training cuts for staff ● Flawed resolution of conflicting goals – Challenger – pressure to lauch vs. safety

System Safety ● Build in safety, not add it on later ● Consider system as a whole, not subsystems ● Take larger view of hazards than just failures ● Analysis over experience and standards ● Qualitative over quantitative approach ● Tradeoffs and conflicts are important in design ● System Safety is more than systems engineering

Definitions ● Reliability: probability that a piece of equipment will perform its intended function satisfactorily for a prescribed time under stipulated conditions ● Failure: nonperformance or inability of the system to meet its intended function for a specified time under specified conditions ● Error: design flaw or deviation from intended state ● Accident: Undesired and unplanned (but maybe not unexpected) event that results in loss

More definitions ● Hazard: set of conditions that together with environment will lead to an accident ● Risk: hazard level combined with likelihood of accident (danger) and hazard duration (latency) ● Safety: Freedom from accident or loss

Models of Accident Analysis ● Domino Theory ● Environment > Person > Act > Accident > Injury ● National Safety Council ● background factors > initiating factors > intermediate factors > immediate factors > measurable results ● unsafe condition > agent of accident > increase in potential > unsafe act > injury ● Chain of Events – single event sets off path to accident

Fault Tree Analysis Wrong Treatment for patient Vital Signs show Incorrect state Correct state but Untimely reaction OR Measurement Frequency Too low Computer fails To raise alarm Vital signs Not reported Nurse does Not respond OR Computer does Not read within Time limit Human sets Frequency too low OR Sensor Failure Human error On input AND

Event Tree Pressure Too High Relief Valve 1 Relief Valve 2 Opens Fails Pressure Decrease Pressure Decrease Explosion

Software Safety ● Requirements Complete/Consistent ● Behavior deterministic ● Robustness ● Every state must have a transition for every possible event ● Transitions out of a state must form tautology ● Behavior specified for timeouts (no input for some period) ● HCI Criteria ● Safety critical outputs checked for reasonableness

● All specified states must be reachable from start state ● No paths to unplanned hazardous states ● Every hazardous state has a path to a safe state

Safety Design ● Hazard elimination (Substitution, Simplification, Decoupling, Human error, materials) ● Hazard reduction (Controllability, safety factors, redundancy, recovery) ● Hazard control (limit exposure, isolation, protection system) ● Damage minimization

New Model ● STAMP : Systems Theoretic Accident Model and Process ● STPA : System-Theoretic Process Analysis

STPA Step 1 ● Identify potential for inadequate control of the system ● A control action required for safety not provided or followed ● An unsafe control action is provided ● A potentially safe action is provided at wrong time ● A control action required for safety is stopped too soon or applied too long

STPA Step 2 ● Determine how each action in step 1 could occur ● Examine parts of control loop. Design controls and measures. For multiple controllers, identify conflicts and coordination problems ● Consider how controls could degrade over time and build in protection – Management of procedures – Audits – Accident and incident analysis

NASA 10 Rules of Development Language Choice == Not one of the rules but very important. Need mature compilers with extensive code checking. ● 1. Restrict all code to very simple control structures. No goto, setjmp/longjmp, direct or indirect recursion. ● 2. All loops must have a fixed upper bound. Must be able to prove a loop terminates. Exception are for loops not meant to terminate, then inverse of above must be true.

NASA 10 rules 3. No dynamic memory allocation after initialization. Garbage collectors and malloc can cause unpredictable issues. 4. no function longer than can be printed on a single sheet of paper. 5. code should average 2 assertions per function. Assertions must be side-effect free. 6. Data items must be declared at the smallest possible scope. 7. Return value of non-void functions must be checked.

NASA 10 rules 8. Use of preprocessor must be limited to header inclusion and simple macros 9. Use of pointers should be limited. No more than one level of dereferencing is allowed. Function pointers are not permitted. 10. Code must be compiled with all compiler warnings enabled at most pedantic setting. Code must not have any warnings. Code must be checked by 2 different code analyzers and pass with no warnings.