1. Software Engineering for Safety: a Roadmap
Ahmad Alsawi
Dapeng Xie
Vladimir Jakobac

2. Motivation
Software become an integral part of many safety-critical systems
“The Nation depends on fragile software”… more work needed
More and more safety-critical systems are built
Motivation for software engineering for safty
The demand is increasing as the number of embedded software in safety-critical system increases
4/18/2019

3. You Can’t Compromise
4/18/2019

4. Some Issues in SE4Safty Hazard analysis
Safety requirement specifications
Designing for safety
Testing
Certifications & standards
4/18/2019

5. Hazard Analysis Core of safe systems development Analysis
severity of effects
Likelihood of occurrence
Decide which hazard to avoid or handled
Identify s/w component that can contribute or prevent hazards
Generate a set of safety constraints and requirements
Hazards: states that can lead to an accident
Accident: unplanned events
Forward analysis methods to identify the possibly hazardous effects of failure
Backward analysis mehtods to investigate if the hypothesized failure is credible
4/18/2019

6. Req. Specification and Analysis
Check safety properties are preserved
Formal specification (allow automation)
Req. are internally consistent
All data are used, all states are reachable
Interactive theorem provers, model checkers
[wrong] 1st identify critical s/w component then analyze if hazard level acceptable
4/18/2019

7. Designing for Safety Design trade offs:
Time to market, features, budget, …
Will fault tolerant feature causes another hazardous condition?
Vulnerability to simple design errors
Tend to neglect small errors
“small errors have small consequences” is not true in s/w
Limited use of known design techniques
Practitioners are not disciplined!
Small error e.g. Mars climate orbiter spacecraft
Do not follow the rules: Aegis missiles cruiser, avoid bad data by operator manual intervention
4/18/2019

8. Current State Testing: It is very important in both:
. Development of safe system
. Certification of safe system
Assumption:
. About environment
. About users
. About operation
A new approach
4/18/2019

9. Current State (cont.) Certification and Standards: Certification:
. More complicated
. Less well-defined
Standards:
. Issue: what standards are appropriate for large, safety-critical
systems composed of subsystem from different
domains.
. Problems:
a.) lack of guidance in existing standards
b.) poor integration of software issues with system safety
c.) heavy burden of making a safety case for certification
. Recommendation:
a.) classifying and evaluating standards according to products,
process and resources
b.) constructing domain specific standards for products.
4/18/2019

10. Current State (cont.) Resources: . Book a.) Safeware by N. Leveson
b.) Software Safety and Reliability
by D.S. Hermann
. Website
a.) Bowen’s website “Safety-Critical Systems”
b.) a recent IEEE video on the subject:
“Developing software for safety critical
systems”
4/18/2019

11. Directions for future work
Integration of informal and formal methods
. Three important working area
a.) automatic translation of informal
notation into formal models.
b.) lightweight formal methods.
c.) integration of previously distinct
formal methods.
4/18/2019

12. Directions… (cont.)
Constraints on safe product families and safe reuse
. Two research areas.
a.) Safety analysis of product families.
. A major goal.
b.) Safe reuse of COTS software
. Two problems.
4/18/2019

13. Directions… (cont.) Testing and evaluation Runtime monitoring
use of requirements-based testing
evaluation from multiple sources
model consistency testing
virtual environment simulations
Runtime monitoring
Improve the testing and evaluation through:
the use of reqs. Often, in practice, additional safety requirements are discovered during design or integration testing, especially from testing of prototypes
Include Domain Experts, Independent Verification and Validation. Not only that software does what it is supposed to do, but that it cannot do what it is not supposed to do.
Use runtime monitoring - Detection of hazardous states
Well suited to monitoring for known, hazardous conditions
Remote agent software can diagnose broader mismatches between expected and actual behavior and recommend recovery action (figure)
4/18/2019

14. Directions… (cont.) Education Related areas
more scientific university courses
textbooks
Related areas
safety – a subset of survivability, security?
software architecture
human factors engineering
safety - freedom from accidents or losses; threats to life or property; focuses on well intended actions; preventing more general malicious activities;
security - threats to privacy or national security; focuses on malicious actions; preventing unauthorized access.
survivability - is the ability to satisfy certain specified critical requirements (for example, security, reliability, real-time responsiveness, and correctness), in the face of adverse conditions. In some cases, survivability may require reconfigurability, interoperability, etc.
Software architecture: safety consequences of product lines
Human factors: formal specification of mental models in order to have more accurate safety requirements…
4/18/2019

15. Software Fault Tree Analysis
hazard events represented by nodes
AND/OR gates
domino effect
errors in the requirements phase
automated analysis with human interaction
example taken from:
4/18/2019

16. The Way Forward
Placing too much reliance on probabilistic risk assessment is unwise
Building safety into a system instead of adding protection devices
Safety is a system problem
Automate the process of safety analysis
Tools able to evolve dynamically over time
Building safety into a system will be much more effective than adding protection devices onto a completed design. The earlier safety is considered in the development process, the better will be the results.
Safety is a system problem and can only be solved by experts in different disciplines working together. Software engineers must understand SYSTEM safety concepts and techniques.
4/18/2019