Fault Tolerance Chap 7

Index Introduction to Fault Tolerance Process Resilience Reliable Client-Server communication Reliable group communication Distributed commit Recovery Summary

Basic Concepts Dependable systems: Availability: property that a system to be used immediately Reliability: the property that a system can run continuously without failure Safety: if a system temporarily fails to operate correctly, nothing catastrophic happens Maintainability: refers to how easy a failed system can be repaired

Different types of failures. Failure Models Type of failure Description Crash failure A server halts, but is working correctly until it halts Omission failure Receive omission Send omission A server fails to respond to incoming requests A server fails to receive incoming messages A server fails to send messages Timing failure A server's response lies outside the specified time interval Response failure Value failure State transition failure The server's response is incorrect The value of the response is wrong The server deviates from the correct flow of control Arbitrary failure A server may produce arbitrary responses at arbitrary times Different types of failures.

Failure Masking by Redundancy Three kind for masking faults: information redundancy, time redundancy, and physical redundancy. Triple modular redundancy.

Flat groups versus Hierarchical groups Communication in a flat group. Communication in a simple hierarchical group

Failure Masking and Replication Failure masking  having a group of identical processes A group of process is organized in a hierarchical fashion with one or more primary backups An important issue with using process groups to tolerate faults is how much replication is needed

Agreement in Faulty Systems (1) 5000 Red Troop Attack Attack 3000 3000 Blue Troop Command by Napoleon Blue Troop Command by Alexander  it is easy to show that Alexander and Napoleon will never reach agreement, no matter how many acknowledgements they send. (due to unreliable communication).

Agreement in Faulty Systems (2) The Byzantine generals problem for 3 loyal generals and1 traitor. The generals announce their troop strengths (in units of 1 kilosoldiers). The vectors that each general assembles based on (a) The vectors that each general receives in step 3.

Agreement in Faulty Systems (3) The same as in previous slide, except now with 2 loyal generals and one traitor. Lamport proved that in a system with m faulty processes, agreement can be achieved only if 2m+1 correctly functioning processes are present, for a total of 3m+1.

RPC Semantics in the Presence of Failures The client is unable to locate the server The request message from the client to the server is lost The server crashes after receiving a request The reply message from the server to the client is lost The client crashes after sending a request

RPC Semantics in the Presence of Failures The client is unable to locate the server  (solution)raise an exception (like divide by 0)  destroys the transparency The request messages from the client to the server is lost  using a timer for sending the request Timer expired  request message is sent a gain

Sever Crashes (1) Three philosophy exist on what to do here: A server in client-server communication Normal case Crash after execution Crash before execution Three philosophy exist on what to do here: At least once semantics At most once semantics Do nothing

Sever Crashes (2) Client Server Strategy M  P Strategy P  M Reissue strategy MPC MC(P) C(MP) PMC PC(M) C(PM) Always DUP OK Never ZERO Only when ACKed Only when not ACKed Different combinations of client and server strategies in the presence of server crashes.

Lost Reply Messages Client Crashes No reply  send the request (idempotent request) once more Assign each request a sequence number Client Crashes Appearance of orphan Extermination: check log and kill orphan Reincarnation: based on broadcasting message to all machines declaring the state of a new epoch (when client reboots) Gentle reincarnation: like reincarnation, orphan is killed only if owner cannot be found Expiration: each RPC is given a standard amount of time

Basic Reliable-Multicasting A simple solution to reliable multicasting when all receivers are known and are assumed not to fail Message transmission Reporting feedback

Nonhierarchical Feedback Control Several receivers have scheduled a request for retransmission, but the first retransmission request leads to the suppression of others.

Hierarchical Feedback Control The essence of hierarchical reliable multicasting. Each local coordinator forwards the message to its children. A local coordinator handles retransmission requests.

Virtual Synchrony (1) The logical organization of a distributed system to distinguish between message receipt and message delivery

The principle of virtual synchronous multicast. Virtual Synchrony (2) The principle of virtual synchronous multicast.

Message Ordering (1) Unordered multicast FIFO-ordered multicast Process P1 Process P2 Process P3 sends m1 receives m1 receives m2 sends m2 Process P1 Process P2 Process P3 Process P4 sends m1 receives m1 receives m3 sends m3 sends m2 sends m4 receives m2 receives m4

Basic Message Ordering Total-ordered Delivery? Reliable causally-ordered multicast delivers messages so that potential causality between different messages is preserved Total-ordered delivery Multicast Basic Message Ordering Total-ordered Delivery? Reliable multicast None No FIFO multicast FIFO-ordered delivery Causal multicast Causal-ordered delivery Atomic multicast Yes FIFO atomic multicast Causal atomic multicast

Implementing Virtual Synchrony Isis (point-to-point communication is reliable, using TCP) Process 4 notices that process 7 has crashed, sends a view change Process 6 sends out all its unstable messages, followed by a flush message Process 6 installs the new view when it has received a flush message from everyone else

Two-phase Commit (1) The finite state machine for the coordinator in 2PC. The finite state machine for a participant. Process crashes  other processes may be indefinite waiting for a message  This protocol can easily fail  timeout mechanisms are used

Two-phase Commit (2) actions by coordinator: while START _2PC to local log; multicast VOTE_REQUEST to all participants; while not all votes have been collected { wait for any incoming vote; if timeout { while GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; exit; } record vote; } if all participants sent VOTE_COMMIT and coordinator votes COMMIT{ write GLOBAL_COMMIT to local log; multicast GLOBAL_COMMIT to all participants; } else { write GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; } Outline of the steps taken by the coordinator in a two phase commit protocol

Three-phase Commit Finite state machine for the coordinator in 3PC Finite state machine for a participant

Recovery-Introduction Goal: replace an erroneous state with an error-free state Backward recovery: to restore such a recorded state when things go wrong  Combining checkpoints and message logging Forward recovery: an attempt is made to bring the system in a correct new state from which it can continue to execute

Stable Storage Stable Storage Crash after drive 1 is updated Bad spot Stable storage is well suited to applications that require a high degree of fault tolerance

Checkpointing A recovery line.

Independent Checkpointing The domino effect.

Message Logging Incorrect replay of messages after recovery, leading to an orphan process.

Summarization (1) Fault tolerance is defined as the characteristic by which a system cam mask the occurrence and recovery from failures Redundancy is the key technique needed to achieve fault tolerance Reliable group communication is suitable for small groups Atomic multicasting can be precisely formulated in terms of a virtual synchronous execution model

Summarization (2) Group membership change  agreement on the same list of members  using commit protocol Recovery in fault-tolerant systems is invariably achieved by checkpointing with message logging Problem: in RPC failures, they only mention about how to kill an orphan  why don’t use it again

Thank you for joining us! End of chapter 7 Thank you for joining us!