Safety Analysis of Software-intensive Systems Tor Stålhane IDI / NTNU

2 What is safety A system is safe if it behaves in such a way that it does not harms people, equipment or the environment. Safety is a relationship between a system and its environment Safety is not an add-on to a system but an integrated part that needs to be considered from day one of a development project.

3 What is safety analysis - 1 Safety analysis is the totality of activities that are used to identify Hazards that may rise when a system is put into operation. Ways to remove these hazards or reduce their consequences to an acceptable level. Actions needed throughout the system’s development to ensure that all safety requirements are implemented.

4 What is safety analysis - 2 The soft side of safety analysis: Collecting and analyzing info. The problems are human related. Collecting info from all stakeholders Organize it in such a way that it can be used to create –Safety requirements for development –Safety tests –Safety routines and procedures for the operation and maintenance of the system

5 What is safety analysis - 3 The hard side of safety analysis: Defining barriers. The problems are related to both humans, software and hardware: How can we construct barriers against hazards in the software? How can we define operating procedures for handling crises?

6 Collecting info - 1 All stakeholders must be involved in the safety analysis since they all possess vital info. Safety analysis is thus a people intensive process – critically dependent on The participants’ experience and knowledge. Our ability to elicit relevant info

7 Collecting info - 2 We need to identify All potentially dangerous events - hazards. The events’ consequences. The events’ probability or frequency – at least in qualitative terms. Important scenarios. The quality of the info from a person increases when the questions are related to a scenario.

8 Tools and methods - 1 The methods that we use in safety analysis – especially in the early phases – must be able to involve all stakeholders. We need methods that are easy to Learn and understand Use on real-life problems Apply to software, hardware, people and routines and procedures.

9 Tools and methods - 2 Which tools and methods to use depend on who participate in the process, the info available and how it is represented. The info available will depend on where in the development process we are. The way the info is represented is, at least partly, something that we can influence. We have good experience with using UML diagrams in all phases.

10 Tools and methods - 3 As we move from a concept to a high level design and then on to detailed design and implementation, more and more Information will be available Decisions will be made and thus leave us with less freedom when making new decisions. Thus, we will need different analysis methods in different phases of the system’s life cycle.

11 Project time and decisions Time TDTD Knowledge Freedom of decisions Experience ConceptHLDLLDImplementation

12 The concept phase Most systems start as a concept, e.g.: Automatic shut-down of production when we discover a gas leakage. All patient info kept in a central database and be available for all that need it through a data network. Complete overview of all our trains – where they are, their speed and so on.

13 Electronic patient journal – Concept Primary Physician Nurse Physician Lab system Patient journal system Top level view – system and stakeholders

14 ExperienceKnowledge ExperienceKnowledge System concept Tools and methods Hazards and barriers Operational environment Stakeholder

15 Preliminary Hazard Analysis - 1 The preliminary hazard analysis is used early in the process. This is reflected in the level of details required in the PHA table. We can include both hazards and the corresponding preventive actions – barriers. Barrier descriptions are converted to system requirements.

16 Preliminary Hazard Analysis - 2 HazardCauseMain effectPreventive action Somebody retrieve wrong info Wrong info inserted Kill or hurt patient Double check all patient info inserted Stored info corrupted Double store and check Wrong patient id used at insertion or retrieval Redundant patient info required for retrieval ::::

17 Requirements Once we have decided to go ahead with the project, we need to elicit and document the requirements. These consist of two components: The functions used to fulfil the customer’s needs Barriers against hazards identified in the PHA

18 Use Case for Electronic patient journal Nurse Medication Documents Diagnosis Orders and responses Treatment plan Physician Primary Physician Lab system

19 NeedsExpectations Customer Requirements Hazards and barriers System concept Operational environment Methods and tools New hazards and barriers ExperienceKnowledge Stakeholder ExperienceKnowledge Stakeholder

20 Safety in the requirements phase Functional requirements – which services should the system offer to its users? Use case diagrams and textual use cases have turned out to be two efficient ways of documenting this. They Are easy to understand for all stakeholders. Can be used as input to several safety analysis methods.

21 Use case for medication

22 Functional FMEA Component IdTreatment plan FunctionFailure mode Local effect Sys effectActionsSeriousness Check current treatment Return wrong data Wrong decision Patient can get hurt or killed Check against other data available on this patient H Return data for another patient H Return no data None Suspend decision Alternative data source L Update treatment plan Wrong update Wrong data in journal Wrong treatment Implement update receipt H No updateH

23 Misuse case Review treatment plan Review drug data Review documents Review diagnosis Network is down Wrong update Delete data Data is lost Unlucky doctor Faulty system Doctor >

24 High level design When we enter high level design, all identified hazards and barriers have been converted to requirements. The high level design can be documented for instance as Package diagrams High level class diagrams High level sequence diagrams

25 Part of electronic patient journal Patient diagnosis Patient drug data General patient info Patient documents Treatment plan

26 ExperienceKnowledge ExperienceKnowledge System concept Tools and methods New hazards Operational environment Extended requirements Barriers and tests Stakeholders

27 Safety and design Packages and classes can be viewed as components and we can thus make our safety analysis much more detailed. Important methods that can be used at this stage are for instance: HazOp, for architectural design. Component FMEA

28 HazOp - 1 HazOp uses study nodes as units of investigation and guide words to help in the hazard identification process. This makes the method quite efficient for identifying hazards On the other hand, HazOp also requires more information – the system’s architecture – to define the study nodes.

29 HazOp - 2 This is a simple version – more elaborate versions gives more info and requires more work. GuidewordStudy node ConsequencesCausesPossible solutions LessGeneral patient info Incomplete info which can lead to wrong treatment Missing updates Lost updates Incomplete updates Check and sign-off for updates Mirror database

30 Failure Mode Effect Analysis - 1 FMEA will systematically check each system component How can this component fail? What are the consequences for the component? What are the consequences for the system? How can we handle the hazard?

31 Failure Mode Effect Analysis - 2 Component Failure mode EffectHandling or barrier Seriousness Patient drug data Give wrong data Wrong medication description. e.g. dosage Check dosage against medication rules database. Prevent too high dosage High Incorrect or missing update Outdated medication description, e.g. dosage High

32 Failure Mode Effect Analysis - 3 The failure Mode Effect Analysis: Offers a systematic walk-through of one or more system components. Focuses on preventions – barriers - rather than cures and fixes. Produces an easy-to-use list of hazards and ideas on how they can be removed or handled.

33 Detailed design Just as high level design, the detailed design can be documented for instance as packages, class diagrams and sequence diagrams. We have more info than we had during high level design and we can thus make a more detailed safety analysis.

34 Patient info Patient documents Drug DBPatient drug data Treatment plan Test results Current treatment Drug description If changes necessary Update drug data

35 ExperienceKnowledge ExperienceKnowledge Tools and methods New hazards Operational environment High level design Barriers and tests Barriers Detailed design Stakeholder

36 Component FMEA Component Id Treatment plan Failure modeLocal effectSys effectActionsSeriousness Return wrong treatment Wrong info to doctor Wrong decision Sanity checkH Update wrong drug data Wrong info in patient’s journal Wrong medication Update receipt H Update drug data for wrong patient H :::::

37 Implementing barriers All hazard analyses must lead to barriers that have one of the following effects: Prevent a hazard from leading to a problem. Prevent a problem from causing a dangerous event. Reduce the effect of a dangerous event if it cannot be prevented.

38 Barrier 1 Barrier 2Barrier 3 Barrier 4 Barrier 5 Barrier 6 RiskProb. Event Prevention Prevent risk from becoming a problem Handling Prevent event from having bad consequences Reduction Reduce effect of event Barrier roles

39 Risk RMRM Minimum achievable risk RARA Acceptable risk Unmitigated risk from EUC RuRu All barriers work as planned Barr. 1Barr. 2Barr. 3Barr. n Barrier failure Barrier reliability

40 Realizing barriers Barriers in software can be realized in several ways. It is important that they do not lead to a large increase in complexity. One way to realize barriers is to use patterns such as: Façades or wrapper façades Protected single channel Sanity checks on values Monitor - actuator

41 Façade pattern

42 Façade pattern sequence diagram

43 Wrapper façade pattern

44

45 Sanity check pattern

46 Monitor - actuator

47 Safety analysis research - 1 Research on safety analysis are concerned with some broad problem areas: How to Implement barriers to prevent or reduce the effect of dangerous events? Create safety analysis patterns? Elicit the necessary information from all stakeholders?

48 Safety analysis research - 2 Our current research in the area of software safety has focused on: Which methods are the easier to understand, learn and use? What is the relationship between method and system representation – is it e.g. easier to base an analysis on scenarios than on a requirements list?

49 Safety analysis research - 3 How can we Improve the safety analysis by making earlier experiences on similar systems available to all stakeholders? Most efficiently move from identified hazards to –Prevention, e.g. barriers –Tests – do the barriers work as intended?

50 Last but not least It is possible to be too safe. A chainsaw with a fully protected blade is Absolutely safe Absolutely useless It is not possible to be absolutely safe. Whatever you do or don’t do, the probability of dying during the next hour is more than Make sure you have a nice day.