1. EEC 688/788 Secure and Dependable Computing
Lecture 10
Wenbing Zhao
Department of Electrical and Computer Engineering
Cleveland State University

2. EEC688/788: Secure & Dependable Computing
Outline
Dependability concepts
Fault, error and failure
Fault/failure detection in distributed systems
Consensus in asynchronous distributed systems
11/22/2018
EEC688/788: Secure & Dependable Computing

3. Dependability and its Attributes
Two alternative definitions (each focus on a different aspects)
Def#1 of Dependability: ability to deliver service that can justifiably be trusted
Aimed at generalizing availability, reliability, safety, confidentiality, integrity, maintainability, that are then attributes of dependability
Focus on trust, i.e. accepted dependence
=> Dependence of system A on system B is the extent to which system A’s dependability is (or would be) affected by that of system B
11/22/2018
EEC688/788: Secure & Dependable Computing

4. Dependability and its Attributes
Def#2 of dependability: ability to avoid service failures that are more frequent or more severe than is acceptable
A system can, and usually does, fail. Is it however still dependable? When does it become undependable?
This def defines the criterion for deciding whether or not, in spite of service failures, a system is still to be regarded as dependable
Dependability failure  fault(s)
11/22/2018
EEC688/788: Secure & Dependable Computing

5. Dependability Related Terminology
A system is an entity that interacts with other entities, i.e., other systems, including hardware, software, humans, and the physical world with its natural phenomena
These other systems are the environment of the given system
The system boundary is the common frontier between the system and its environment
System
System Boundary
Environment
11/22/2018
EEC688/788: Secure & Dependable Computing

6. Dependability Related Terminology
Service delivered by a system: work done that benefits its users
User: another system that interacts with the former
Function of a system: what the system is intended to do
(Functional) Specification: description of the system function
Correct service: when the delivered service implements the system function
Use a calculator as an example. Service: calculation service. User: whoever wants to perform a calculation using the service. Function: calculation. Spec: add, minus, multiply, division, etc.. Correct service: add: 1+1=2, etc.
11/22/2018
EEC688/788: Secure & Dependable Computing

7. Attributes of a Dependable System
For a system to be dependable, it must be
Available - e.g., ready for use when we need it
Reliable - e.g., able to provide continuity of service while we are using it
Safe - e.g., does not have a catastrophic consequence on the environment
Secure - e.g., able to preserve confidentiality
11/22/2018
EEC688/788: Secure & Dependable Computing

8. Quantitative Dependability Measures
Reliability - a measure of continuous delivery of proper service - or, equivalently, of the time to failure
It is the probability of surviving (potentially despite failures) over an interval of time
For example, the reliability requirement might be stated as a availability for a 10-hour mission. In other words, the probability of failure during the mission may be at most 10-6
Hard real-time systems such as flight control and process control demand high reliability, in which a failure could mean loss of life
11/22/2018
EEC688/788: Secure & Dependable Computing

9. Quantitative Dependability Measures
Availability - a measure of the delivery of correct service with respect to the alternation of correct service and out-of-service
It is the probability of being operational at a given instant of time
A availability means that the system is not operational at most one hour in a million hours
A system with high availability may in fact fail. However, failure frequency and recovery time should be small enough to achieve the desired availability
Soft real-time systems such as telephone switching and airline reservation require high availability
11/22/2018
EEC688/788: Secure & Dependable Computing

10. EEC688/788: Secure & Dependable Computing
11/22/2018
EEC688/788: Secure & Dependable Computing

11. Fault, Error, and Failure
The adjudged or hypothesized cause of an error is called a fault
An error is a manifestation of a fault in a system, in which the logical state of an element differs from its intended value
A service failure occurs if the error propagates to the service interface and causes the service delivered by the system to deviate from correct service
The failure of a component causes a permanent or transient fault in the system that contains the component
Service failure of a system causes a permanent or transient external fault for the other system(s) that receive service from the given system
Circular definition? A fault is really a failure in a smaller scope
11/22/2018
EEC688/788: Secure & Dependable Computing

12. EEC688/788: Secure & Dependable Computing
Fault
Faults can arise during all stages in a computer system's evolution - specification, design, development, manufacturing, assembly, and installation - and throughout its operational life
Most faults that occur before full system deployment are discovered through testing and eliminated
Faults that are not removed can reduce a system's dependability when it is in the field
A fault can be classified by its duration, nature of output, and correlation to other faults
11/22/2018
EEC688/788: Secure & Dependable Computing

13. Fault Types - Based on Duration
Permanent faults are caused by irreversible device/software failures within a component due to damage, fatigue, or improper manufacturing, or bad design and implementation
Permanent software faults are also called Bohrbugs
Easier to detect
Transient/intermittent faults are triggered by environmental disturbances or incorrect design
Transient software faults are also referred to as Heisenbugs
Study shows that Heisenbugs are the majority software faults
Harder to detect
Once a permanent fault has occurred, the faulty component can be restored by replacement or repair
environmental disturbances such as voltage fluctuations, electro-magnetic interference, or radiation
These events typically have a short duration, returning the affected circuitry to a normal operating state without causing any lasting damage
Example Heisenbugs: temperary memory corruption? => lead to software aging
11/22/2018
EEC688/788: Secure & Dependable Computing

14. Fault Types - Based on Nature of Output
Malicious fault: The fault that causes a unit to behave arbitrarily or malicious. Also referred to as Byzantine fault
A sensor sending conflicting outputs to different processors
Compromised software system that attempts to cause service failure
Non-malicious faults: the opposite of malicious faults
Faults that are not caused with malicious intention
Faults that exhibit themselves consistently to all observers, e.g., fail-stop
Malicious faults are much harder to detect than non-malicious faults
If a sensor does not read correctly consistently to everyone, we say there is non-malicious fault
11/22/2018
EEC688/788: Secure & Dependable Computing

15. EEC688/788: Secure & Dependable Computing
Fail-Stop System
A system is said to be fail-stop if it responds to up to a certain maximum number of faults by simply stopping, rather than producing incorrect output
A fail-stop system typically has many processors running the same tasks and comparing the outputs. If the outputs do not agree, the whole unit turns itself off
A system is said to be fail-safe if one or more safe states can be identified, that can be accessed in case of a system failure, in order to avoid catastrophe
11/22/2018
EEC688/788: Secure & Dependable Computing

16. Fault Types - Based on Correlation
Components fault may be independent of one another or correlated
A fault is said to be independent if it does not directly or indirectly cause another fault
Faults are said to be correlated if they are related. Faults could be correlated due to physical or electrical coupling of components
Correlated faults are more difficult to detect than independent faults
What can you say about running three replicated processes on the same node?
11/22/2018
EEC688/788: Secure & Dependable Computing

17. Fail Fast to Reduce Heisenbugs
The bugs that software developers hate most:
The ones that show up only after hours of successful operation, under unusual circumstances
The stack trace usually does not provide useful information
This kind of bugs might be caused by many reasons, such as
Not checking the boundary of an array
Invalid defensive programming <= what fail fast addresses
Reference
11/22/2018
EEC688/788: Secure & Dependable Computing

18. Fail Fast to Reduce Heisenbugs
Invalid defensive programming
Making your software robust by working around problems automatically
This results in the software “failing slowly”
That is, it facilitates error propagation - the program continues working right after an error but fails in strange ways later on
Example:
public int maxConnections() {
string property = getProperty(“maxConnections”);
if (property == null) {
return 10; }
else {
return property.toInt();
11/22/2018
EEC688/788: Secure & Dependable Computing

19. Fail Fast to Reduce Heisenbugs
Fail fast programming
When a problem occurs, it fails immediately & visibly
It may sound like it would make your software more fragile, but it actually makes it more robust
Bugs are easier to find and fix, so fewer go into production
Example:
public int maxConnections() {
string property = getProperty(“maxConnections”);
if (property == null) {
throw new NullReferenceException(“maxConnections property not
found in “ + this.configFilePath); }
else { return property.toInt(); }
Fail fast => early detection
11/22/2018
EEC688/788: Secure & Dependable Computing

20. Approaches to Achieving Dependability
Fault Avoidance - how to prevent, by construction, the fault occurrence or introduction
Fault Removal - how to minimize, by verification, the presence of faults
Fault Tolerance - how to provide, by redundancy, a service complying with the specification in spite of faults
Fault Forecasting - how to estimate, by evaluation, the presence, the creation, and the consequence of faults
11/22/2018
EEC688/788: Secure & Dependable Computing

21. EEC688/788: Secure & Dependable Computing
Graceful Degradation
If a specified fault scenario develops, the system must still provide a specified level of service. Ideally, the performance of the system degrades gracefully
The system must not suddenly collapse when a fault occur, or as the size of the faults increases
Rather it should continue to execute part of the work load correctly
11/22/2018
EEC688/788: Secure & Dependable Computing

22. Failure Detection in Distributed Systems
Consider the failure detection problem in an asynchronous distributed system, where
No upper bound on process time
No upper bound on clock drift rate
No upper bound in networking delay
In an asynchronous distributed system, you cannot tell a crashed process from a slow one, even if you can assume that messages are sequenced and retransmitted (arbitrary numbers of times), so they eventually get through
This leads to Fischer, Lynch and Paterson to proof that it is impossible to reach a consensus in a fully asynchronous distributed system
11/22/2018
EEC688/788: Secure & Dependable Computing

23. EEC688/788: Secure & Dependable Computing
Consensus Problem
Safety:
Only a value that has been proposed may be chosen
Only a single value is chosen, and
A process never learns that a value has been chosen unless it actually has been
Liveness:
Some proposed value is eventually chosen and, if a value has been chosen, then a process can eventually learn the value
11/22/2018
EEC688/788: Secure & Dependable Computing

24. Impossibility Results
FLP Impossibility of Consensus
A single faulty process can prevent consensus
Because a slow process is indistinguishable from a crashed one
Chandra/Toueg Showed that FLP Impossibility applies to many problems, not just consensus
In particular, they show that FLP applies to group membership, reliable multicast
So these practical problems are impossible in asynchronous systems
They also look at the weakest condition under which consensus can be solved
Ways to bypass the impossibility result
Use unreliable failure detector
Use a randomized consensus algorithm
11/22/2018
EEC688/788: Secure & Dependable Computing

25. The Paxos Algorithm – Consensus for Asynchronous Systems
Contribution: separately consider safety and liveness issues. Safety can be guaranteed and liveness is ensured during period of synchrony
Participants of the algorithm are divided into three categories
Proposers: those who propose values
Accepters: those who decide which value to choose
Learners: those who are interested in learning the value chosen
11/22/2018
EEC688/788: Secure & Dependable Computing

26. EEC688/788: Secure & Dependable Computing
The Paxos Algorithm
How to choose a value
Use a single acceptor: straightforward but not fault tolerant
Use a number of acceptors: a value is chosen if the majority of the acceptors have accepted it
Accepted does not mean chosen. However, if a value has been chosen, it must have been accepted first
11/22/2018
EEC688/788: Secure & Dependable Computing

27. EEC688/788: Secure & Dependable Computing
The Paxos Algorithm
Requirements for choosing a value
P1. An acceptor must accept the first proposal that it receives
P2. If a proposal with value v is chosen, then every higher-numbered proposal that is chosen has value v
Since the proposal numbers are totally ordered, P2 guarantees the safety property
11/22/2018
EEC688/788: Secure & Dependable Computing

28. EEC688/788: Secure & Dependable Computing
The Paxos Algorithm
How to guarantee P2?
P2a: If a proposal with value v is chosen, then every higher-numbered proposal accepted by any acceptor has value v
But what if an acceptor that has never accepted v accepted a proposal with v’?
P2b: if a proposal with value v is chosen, then every higher-numbered proposal issued by any proposer has value v
P2b implies P2a, which implies P2
11/22/2018
EEC688/788: Secure & Dependable Computing

29. EEC688/788: Secure & Dependable Computing
The Paxos Algorithm
How to ensure P2b?
P2c: For any v and n, if a proposal with value v and number n is issued, then there is a set S consisting of a majority of acceptors such that either
(a) no acceptor in S has accepted any proposal numbered less than n, or
(b) v is the value of the highest-numbered proposal among all proposals numbered less than n accepted by the acceptors in S
11/22/2018
EEC688/788: Secure & Dependable Computing

30. EEC688/788: Secure & Dependable Computing
The Paxos Algorithm
To ensure P2c, an acceptor must promise:
It will not accept any more proposals numbered less than n, once it has accepted a proposal n
11/22/2018
EEC688/788: Secure & Dependable Computing

31. EEC688/788: Secure & Dependable Computing
The Paxos Algorithm
Phase 1.
(a) A proposer selects a proposal number n and sends a prepare request with number n to a majority of acceptors.
(b) If an acceptor receives a prepare request with number n greater than that of any prepare request to which it has already responded, then it responds to the request with a promise not to accept any more proposals numbered less than n and with the highest-numbered proposal (if any) that it has accepted.
11/22/2018
EEC688/788: Secure & Dependable Computing

32. EEC688/788: Secure & Dependable Computing
The Paxos Algorithm
Phase 2.
(a) If the proposer receives a response to its prepare requests (numbered n) from a majority of acceptors, then it sends an accept request to each of those acceptors for a proposal numbered n with a value v, where v is the value of the highest-numbered proposal among the responses, or is any value if the responses reported no proposals.
(b) If an acceptor receives an accept request for a proposal numbered n, it accepts the proposal unless it has already responded to a prepare request having a number greater than n.
11/22/2018
EEC688/788: Secure & Dependable Computing

33. EEC688/788: Secure & Dependable Computing
The Paxos Algorithm
11/22/2018
EEC688/788: Secure & Dependable Computing

34. EEC688/788: Secure & Dependable Computing
Paxos Examples
11/22/2018
EEC688/788: Secure & Dependable Computing

35. EEC688/788: Secure & Dependable Computing
Paxos Examples
11/22/2018
EEC688/788: Secure & Dependable Computing

36. EEC688/788: Secure & Dependable Computing
Paxos Examples
11/22/2018
EEC688/788: Secure & Dependable Computing