Chapter 7 Fault Tolerance Basic Concepts Failure Models Process Design Issues Flat vs hierarchical group Group Membership Reliable Client Server Communication Point to point Communication RPC Reliable Group Communication Atomic Multicast Two Phase Commit

Introduction Partial failure in distributed system may happen when one component is fails. May affect the operation in certain component Leaving another component totally unaffected The design goal in DS is Build a system that automatically recover from a partial failure Without seriously affecting the overall performance

Basic Concepts Dependability Includes Availability Reliability Safety Maintainability

Availability The system is ready to be used immediately In general, the system is operating correctly at any given moment and is available to performs its functions. Percentage of availability = (total elapsed time – sum of downtime)/total elapsed time 99.9% 61.32 hours

Reliability System can run continuously without failure. High reliable system is one that will most likely continue to work without interruption during a relative long period of time One measure used to define a component or system's reliability is mean time between failures (MTBF) MTBF = (total elapsed time – sum of downtime)/number of failures A related measurement is mean time to repair (MTTR). MTTR is the average time interval (usually expressed in hours) that it takes to repair a failed component.

Safety Nothing catastrophic will happen if a system temporary fails to operate correctly.

Maintainability Refers to how easy a failed system can be repaired

Terminology Failure: When a component is not living up to its specifications, a failure occurs Error: That part of a component's state that can lead to a failure Fault: The cause of an error Fault prevention: prevent the occurrence of a fault Fault tolerance: build a component in such a way that it can meet its specifications in the presence of faults

Different types of failures. Failure Models Different types of failures. Type of failure Description Crash failure A server halts, but is working correctly until it halts Omission failure Receive omission Send omission A server fails to respond to incoming requests A server fails to receive incoming messages A server fails to send messages Timing failure A server's response lies outside the specified time interval Response failure Value failure State transition failure The server's response is incorrect The value of the response is wrong The server deviates from the correct flow of control Arbitrary failure A server may produce arbitrary responses at arbitrary times

Failure Models(cont) Timing failures: The output of a component is correct, but lies outside a specified real-time interval - (performance failures: too slow) Response failures: The output of a component is incorrect Value failure: The wrong value is produced State transition failure: Execution of the component's service brings it into a wrong state

Failure Models(cont) Crash failures: A component halts but behaves correctly before halting Omission failures: A component fails to respond Receive omission: A server fails to receive incoming messages Send omission: A server fails to send messages Arbitrary failures: A component may produce arbitrary output and be subject to arbitrary timing failures

Process Resilience Protect against faulty processes by replicating and distributing computations in a group: Flat groups: Good for fault tolerance as information exchange immediately occurs with all group members; however, may impose more overhead as control is completely distributed (hard to implement). Hierarchical groups: All communication through a single coordinator  not really fault tolerant and scalable, but relatively easy to implement.

Flat Groups versus Hierarchical Groups Communication in a flat group. Communication in a simple hierarchical group

Group Membership Method for creating and deleting groups is needed Allowing process to join and leave the group Group Server Centralized Single point of failure Send a message to all groups members Want to join the group Send goodbye to leave the group

RPC Semantic in the presence of failure What can go wrong?: 1: Client can’t locate server 2: Client request is lost 3: Server crashes 4: Server response is lost 5: Client crashes

RPC Semantics The client unable to locate the server The server might be down Relatively simple – just report back to client Lost Request message Just resend message Use timer in waiting for acknowledgement msg, If the ack is not received within certain time limit, the msg will be transmit again

Server crashes are harder - you don't know what was done: RPC Semantics Server crashes are harder - you don't know what was done: A server in client-server communication Normal case Crash after execution Crash before execution

RPC Semantics(cont) Lost Reply message Client Crashes Use the timer again, resend the request Keep track of the request by having a sequence number to differentiate between the original req and a retransmission. Or by having a bit in the message header to distinguish initial request from retransmission. Client Crashes After send a request to the server Server will do the computation while there is no client waiting for the result – orphan.

Basic Reliable-Multicasting Schemes A simple solution to reliable multicasting when all receivers are known and are assumed not to fail Message transmission Reporting feedback

Atomic Multicast Definition msg is delivered to either all processes or none at all all messages are delivered in the same order to all processes Atomic Multicast is required for many applications example? Replicated Database, Bank Transactions Example: replicated database each replica is represented by a process each process needs to get the updates in the same order when a process dies and re-starts, it needs to perform the updates in order For atomic multicast, all the processes need to agree on group membership eg: how does a crashed process re-join the group

Reliable Stock Exchange Reliable Multicast Receivers Object Group External Data Feed (ORB) Bridge

Virtually Synchronous Group View: the view on the set of process contained in the group. The message multicast to group view G is delivered to each non-faulty process in G. If the sender crashes, the message may either be delivered to all the remaining process, or ignored by each of them.

Virtual Synchrony (1) The logical organization of a distributed system to distinguish between message receipt and message delivery

The principle of virtual synchronous multicast. Virtual Synchrony (2) The principle of virtual synchronous multicast.

Message Ordering Unordered multicast FIFO-ordered multicast Causally-ordered multicast Totally-ordered multicast

Message Ordering (1) Process P1 Process P2 Process P3 sends m1 receives m1 receives m2 sends m2 Three communicating processes in the same group. The ordering of events per process is shown along the vertical axis.

Message Ordering (2) Process P1 Process P2 Process P3 Process P4 sends m1 receives m1 receives m3 sends m3 sends m2 sends m4 receives m2 receives m4 Four processes in the same group with two different senders, and a possible delivery order of messages under FIFO-ordered multicasting

Implementing Virtual Synchrony (1) Multicast message m to a group of process. No guarantee that all member in G will receive m (sender may fail before transmit m to all its member) Because the sender has crashed; another process should get m from somewhere else Let every process in G keep m until all member in G have received it. m is said to be stable, if all member received it Only a stable message are allowed to be delivered

Implementing Virtual Synchrony (2) Process 4 notices that process 7 has crashed, sends a view change Process 6 sends out all its unstable messages, followed by a flush message Process 6 installs the new view when it has received a flush message from everyone else

Consist of coordinator and participants(voting phase&decision phase) Two-Phase Commit (1) Consist of coordinator and participants(voting phase&decision phase) The coordinator sends a VOTE_REQUEST message to all participants Participant may either send back the VOTE_COMMIT or VOTE_ABORT Coordinator will send GLOBAL_COMMIT if received VOTE_COMMIT from all participant; GLOBAL_ABORT if only one send VOTE_ABORT Each participant wait for message from coordinator; GLOBAL_COMMIT- Locally commit the transaction GLOBAL_ABORT - Locally abort the transaction

Two-Phase Commit (2) The finite state machine for the coordinator in 2PC. The finite state machine for a participant.

Two-Phase Commit (3) The protocol can easily fail when a process crashes; other processes may indefinitely waiting for a message from that process. Time out mechanism are use to solved the problems 1. Participant may waiting for VOTE_REQUEST in their INIT state; decide to locally abort the transaction 2. Coordinator may waiting for a message from participants in the WAIT state; send GLOBAL_ABORT 3. Participant block in READY state waiting for global vote from coordinator; participant cannot simply decide to abort the transaction; - wait for coordinator to up again; -Contact another participant to check for the current state

Two-Phase Commit (4) Coordinator Q State Coordinator Msg COMMIT GLOBAL_COMMIT ABORT GLOBAL_ABORT INIT VOTE_REQUEST READY Wait for coordinator recover READY P Q State of Q Action by P COMMIT Make transition to COMMIT ABORT Make transition to ABORT INIT READY Contact another participant Actions taken by a participant P when residing in state READY and having contacted another participant Q.