1. STAMP A new accident causation model using Systems Theory (vs
STAMP A new accident causation model using Systems Theory (vs. Reliability Theory)

2. Introduction to Systems Theory
Ways to cope with complexity
Analytic Reduction
Statistics

3. Analytic Reduction Divide system into distinct parts for analysis
Physical aspects  Separate physical components
Behavior  Events over time
Examine parts separately
Assumes such separation possible:
The division into parts will not distort the phenomenon
Each component or subsystem operates independently
Analysis results not distorted when consider components separately

4. Analytic Reduction (2) Called Organized Simplicity
2. Components act the same when examined singly as when playing their part in the whole
Components or events not subject to feedback loops and non-linear interactions
3. Principles governing the assembling of components into the whole are themselves straightforward
Interactions among subsystems simple enough that can be considered separate from behavior of subsystems themselves
Precise nature of interactions is known
Interactions can be examined pairwise
Called Organized Simplicity

5. Statistics
Treat system as a structureless mass with interchangeable parts
Use Law of Large Numbers to describe behavior in terms of averages
Assumes components are sufficiently regular and random in their behavior that they can be studied statistically
Called Unorganized Complexity

6. Complex, Software-Intensive Systems
Too complex for complete analysis
Separation into (interacting) subsystems distorts the results
The most important properties are emergent
Too organized for statistics
Too much underlying structure that distorts the statistics
Called Organized Complexity

8. Systems Theory
Developed for biology (von Bertalanffly) and engineering (Norbert Weiner)
Basis of system engineering and system safety
ICBM systems of the 1950s
Developed to handle systems with “organized complexity”

9. Systems Theory (2)
Focuses on systems taken as a whole, not on parts taken separately
Some properties can only be treated adequately in their entirety, taking into account all social and technical aspects
These properties derive from relationships among the parts of the system
How they interact and fit together
Two pairs of ideas
Hierarchy and emergence
Communication and control

10. Hierarchy and Emergence
Complex systems can be modeled as a hierarchy of organizational levels
Each level more complex than one below
Levels characterized by emergent properties
Irreducible
Represent constraints on the degree of freedom of components at lower level
Safety is an emergent system property
It is NOT a component property
It can only be analyzed in the context of the whole

11. Example
Control
Structure

12. Communication and Control
Hierarchies characterized by control processes working at the interfaces between levels
A control action imposes constraints upon the activity at a lower level of the hierarchy
Open systems are viewed as interrelated components kept in a state of dynamic equilibrium by feedback loops of information and control
Control in open systems implies need for communication

13. Requirements for Effective Control
To effect control over a system requires four conditions:
Goal Condition: The controller must have a goal or
goals (e.g., to maintain a setpoint)
Action Condition: The controller must be able to
affect the system state
Observability Condition: The controller must be
able to ascertain the state of the system.
Model Condition: The controller must be (or contain)
a model of the system

14. Control processes operate between levels
Controller
Model of
Process
Process models must contain:
- Required relationship among
process variables
- Current state (values of
- The ways the process can
change state
Control
Actions
Feedback
Controlled Process

15. Process models must contain:
- Required relationship among
process variables
- Current state (values of
- The ways the process can
change state

17. Relationship Between Safety and Process Models
Accidents occur when models do not match process and
Incorrect control commands given
Correct ones not given
Control commands given at wrong time (too early, too late)
Control stops too soon
(Note the relationship to system accidents)

18. Relationship Between Safety and Process Models (2)
How do they become inconsistent?
Wrong from beginning
Missing or incorrect feedback
Not updated correctly
Time lags not accounted for
Resulting in
Uncontrolled disturbances
Unhandled process states
Inadvertently commanding system into a hazardous state
Unhandled or incorrectly handled system component failures

19. Relationship Between Safety and Human Mental Models
Explains most human/computer interaction problems
Pilots and others are not understanding the automation
Or don’t get feedback to update mental models or disbelieve it
What did it just do? Why won’t it let us do that?
Why did it do that? What caused the failure?
What will it do next? What can we do so it does
How did it get us into this state? not happen again?
How do I get it to do what I want?

20. Mental Models

21. Relationship Between Safety and Human Mental Models (2)
Also explains developer errors. May have incorrect model of
Required system or software behavior for safety
Development process
Physical laws
Etc.

22. STAMP
Accidents arise from unsafe interactions among humans, machines, and the environment (not just component failures)
“prevent failures” ↓
“enforce safety constraints on system behavior”
Losses are the result of complex dynamic processes, not simply chains of failure events
Most major accidents arise from a slow migration of the entire system toward a state of high-risk
Need to control and detect this migration

23. STAMP (2) View accidents as a control problem
O-ring did not control propellant gas release by sealing gap in field joint
Software did not adequately control descent speed of Mars Polar Lander
Events are the result of the inadequate control
Result from lack of enforcement of safety constraints by design and operations

24. © Copyright Nancy Leveson, Aug. 2006
STAMP (3)
Safety is an emergent property that arises when system components interact with each other within a larger environment
A set of constraints related to behavior of system components enforces that property
Accidents occur when interactions violate those constraints (a lack of appropriate constraints on the interactions)
“Controllers” embody or enforce those constraints
© Copyright Nancy Leveson, Aug. 2006

25. Example Safety Constraints
Build safety in by enforcing safety constraints on behavior
Controllers contribute to accidents not by “failing” but by:
Not enforcing safety-related constraints on behavior
Commanding behavior that violates safety constraints
System Safety Constraint:
Water must be flowing into reflux condenser whenever catalyst
is added to reactor
Software Safety Constraint:
Software must always open water valve before catalyst valve

26. Class Exercise: Identify the primary physical system safety constraint violated at Bhopal and then a constraint on one of the system components that was violated

27. The Nature of Controls (1)
Component failures and unsafe interactions may be “controlled” through design
(e.g., redundancy, interlocks, fail-safe design)
or through process
Manufacturing processes and procedures
Maintenance processes
Operations
or through social controls

28. The Nature of Controls (2)
Social controls need not necessarily be governmental or regulatory
Controls may also be cultural, policy, individual (self-interest)
Example: When investment banks went public, individual controls to reduce personal risk and long-term profits were eliminated and risk shifted to shareholders and others who had few and weak controls over those taking the risks.
Controls must be designed and implemented throughout the whole system, not just externally
We done this (or at least started it) for corporate fraud. As an example,
we modeled Sarbanes-Oxley and the factors that would need to exist
for it to be successful in reducing risk to acceptable levels.

29. Summary: Accident Causality
Accidents occur when
Control structure or control actions do not enforce safety constraints
Unhandled environmental disturbances or conditions
Unhandled or uncontrolled component failures
Dysfunctional (unsafe) interactions among components
Control structure degrades over time (asynchronous evolution)
Control actions inadequately coordinated among multiple controllers

31. Dysfunctional Controller Interactions
Boundary areas
Overlap areas (side effects of decisions and control actions)
Controller 1
Process 1
Process 2
Controller 2
Process
Controller 1
Controller 2
© Copyright Nancy Leveson, Aug. 2006

32. Uncoordinated “Control Agents”
“UNSAFE STATE”
BOTH TCAS and ATC provide uncoordinated & independent instructions
“SAFE STATE”
TCAS provides coordinated instructions to both planes
“SAFE STATE”
ATC provides coordinated instructions to both planes
Control Agent
(TCAS)
Instructions
No Coordination
Control Agent
(ATC)
Instructions
Control Agent
(ATC)
Instructions

33. Uses for STAMP
Basis for new, more powerful hazard analysis techniques (STPA)
Safety-driven design (physical, operational, organizational)
More comprehensive accident/incident investigation and root cause analysis
Organizational and cultural risk analysis
Identifying physical and project risks
Defining safety metrics and performance audits
Designing and evaluating potential policy and structural improvements
Identifying leading indicators of increasing risk (“canary in the coal mine”)
New holistic approaches to security

34. Applying STAMP
To understand accidents, need to examine safety control structure itself to determine why inadequate to maintain safety constraints and why events occurred
To prevent accidents, need to create an effective safety control structure to enforce the system safety constraints
Not a “blame” model but a “why” model