1. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN Chapter 8/B Fault Tolerance Modified by Dr. Gheith Abandah

2. Overview Introduction to Fault Tolerance Basic Concepts Failure Models Failure Masking by Redundancy Process Resilience Design Issues Failure Masking and Replication Agreement in Faulty Systems Failure Detection Reliable Client-server Communication Point-to-point Communication RPC Semantics in The Presence of Failures Reliable Group Communication Basic Reliable-multicasting Schemes Scalability in Reliable Multicasting Atomic Multicast Distributed Commit Two-phase Commit Three-phase Commit Recovery Check-pointing Message Logging Recovery-oriented Computing Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5

3. Reliable Client-Server Communication Error detection Framing of packets to allow for bit error detection Use of frame numbering to detect packet loss Error correction Add so much redundancy that corrupted packets can be automatically corrected Request retransmission of lost, or last N packets

4. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Reliable RPC Five different classes of failures that can occur in RPC systems: 1.The client is unable to locate the server: Report back to client. 2.The request message from the client to the server is lost: Resend message (some number of retries). 3.The server crashes after receiving a request: Difficult, see next slide. 4.The reply message from the server to the client is lost: Client re-requests with same sequence no. 5.The client crashes after sending a request: New epoch no. after client reboots.

5. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Server Crashes (1) A server in client-server communication. (a) The normal case. (b) Crash after execution. (c) Crash before execution.

6. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Server Crashes (2) We need to decide on what we expect from the server At-least-once-semantics: The server guarantees it will carry out an operation at least once, no matter what. At-most-once-semantics: The server guarantees it will carry out an operation at most once. Three events that can happen at the server: Send the completion message (M), Print the text (P), Crash (C).

7. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Server Crashes (3) These events can occur in six different orderings: 1.M →P →C: A crash occurs after sending the completion message and printing the text. 2.M →C (→P): A crash happens after sending the completion message, but before the text could be printed. 3.P →M →C: A crash occurs after sending the completion message and printing the text. 4.P→C(→M): The text printed, after which a crash occurs before the completion message could be sent. 5.C (→P →M): A crash happens before the server could do anything. 6.C (→M →P): A crash happens before the server could do anything.

8. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Server Crashes (4) Different combinations of client and server strategies in the presence of server crashes.

9. Overview Introduction to Fault Tolerance Basic Concepts Failure Models Failure Masking by Redundancy Process Resilience Design Issues Failure Masking and Replication Agreement in Faulty Systems Failure Detection Reliable Client-server Communication Point-to-point Communication RPC Semantics in The Presence of Failures Reliable Group Communication Basic Reliable-multicasting Schemes Scalability in Reliable Multicasting Atomic Multicast Distributed Commit Two-phase Commit Three-phase Commit Recovery Check-pointing Message Logging Recovery-oriented Computing Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5

10. Reliable Multicasting Observation If we can stick to a local-area network, reliable multicasting is “easy” Principle Let the sender log messages submitted to channel c: If P sends message m, m is stored in a history buffer Each receiver acknowledges the receipt of m, or requests retransmission at P when noticing message lost Sender P removes m from history buffer when everyone has acknowledged receipt

11. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Scalable Reliable Multicasting: Feedback Suppression Basic idea Let a process P suppress its own feedback when it notices another process Q is already asking for a retransmission Assumptions All receivers listen to a common feedback channel to which feedback messages are submitted Process P schedules its own feedback message randomly, and suppresses it when observing another feedback message

12. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Basic Reliable-Multicasting Schemes A simple solution to reliable multicasting when all receivers are known and are assumed not to fail. (a) Message transmission. (b) Reporting feedback.

13. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Scalable Reliable Multicasting: Feedback Suppression Several receivers have scheduled a request for retransmission, but the first retransmission request leads to the suppression of others.

14. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Scalable Reliable Multicasting: Hierarchical Solutions The essence of hierarchical reliable multicasting. Each local coordinator forwards the message to its children and later handles retransmission requests.

15. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Atomic Multicast (1) Basic model We have a multicast channel c with two (possibly overlapping) groups: The sender group SND(c) of processes that submit messages to channel c The receiver group RCV(c) of processes that can receive messages from channel c Simple reliability: If process P  RCV(c) at the time message m was submitted to c, and P does not leave RCV(c), m should be delivered to P Atomic multicast: How can we ensure that a message m submitted to channel c is delivered to process P  RCV(c) only if m is delivered to all members of RCV(c)

16. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Atomic Multicast (2) Reliable multicasting in the presence of process failures in terms of process groups and changes to group membership. A message is delivered only to the non-faulty members of the current group. All members should agree on the current group membership →Virtually synchronous multicast.

17. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Message Ordering (1) Four different orderings are distinguished: Unordered multicasts FIFO-ordered multicasts Causally-ordered multicasts Totally-ordered multicasts

18. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Message Ordering (2) Unordered: Three communicating processes in the same group. The ordering of events per process is shown along the vertical axis.

19. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Message Ordering (3) FIFO ordered from one process: Four processes in the same group with two different senders, and a possible delivery order of messages under FIFO-ordered multicasting

20. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Message Ordering (4) Six different versions of virtually synchronous reliable multicasting.

21. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Implementing Virtual Synchrony (1) Example from Isis (uses TCP for reliable p2p): (a) Process 4 notices that process 7 has crashed and sends a view change.

22. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Implementing Virtual Synchrony (2) (b) Process 6 sends out all its unstable messages (to insure that all have received all messages from the previous view), followed by a flush message to enforce the new view.

23. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Implementing Virtual Synchrony (3) (c) Process 6 installs the new view when it has received a flush message from everyone else.

24. Overview Introduction to Fault Tolerance Basic Concepts Failure Models Failure Masking by Redundancy Process Resilience Design Issues Failure Masking and Replication Agreement in Faulty Systems Failure Detection Reliable Client-server Communication Point-to-point Communication RPC Semantics in The Presence of Failures Reliable Group Communication Basic Reliable-multicasting Schemes Scalability in Reliable Multicasting Atomic Multicast Distributed Commit Two-phase Commit Three-phase Commit Recovery Check-pointing Message Logging Recovery-oriented Computing Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5

25. Distributed Commit Essential issue Given a computation distributed across a process group, how can we ensure that either all processes commit to the final result, or none of them do (atomicity)? One-phase commit is not sufficient Two-phase commit Three-phase commit

26. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Two-Phase Commit (1) Model The client who initiated the computation acts as coordinator; processes required to commit are the participants Phase 1a: Coordinator sends vote-request to participants (also called a pre-write) Phase 1b: When participant receives vote-request it returns either vote-commit or vote-abort to coordinator. If it sends vote-abort, it aborts its local computation Phase 2a: Coordinator collects all votes; if all are vote- commit, it sends global-commit to all participants, otherwise it sends global-abort Phase 2b: Each participant waits for global-commit or global- abort and handles accordingly.

27. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Two-Phase Commit (2) (a) The finite state machine for the coordinator in 2PC. (b) The finite state machine for a participant.

28. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Two-Phase Commit (3) Failing Participant Participant crashes in state S, and recovers to S Initial state: No problem: participant was unaware of protocol Ready state: Participant is waiting to either commit or abort. After recovery, participant needs to know which state transition it should make → log the coordinator’s decision Abort state: Merely make entry into abort state idempotent, e.g., removing the workspace of results Commit state: Also make entry into commit state idempotent, e.g., copying workspace to storage. Observation When distributed commit is required, having participants use temporary workspaces to keep their results allows for simple recovery in the presence of failures.

29. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Two-Phase Commit (4) Result: If all participants are in the READY state, the protocol blocks. Apparently, the coordinator is failing. Note: The protocol prescribes that we need the decision from the coordinator. Alternative: When P recovers in the READY state, can check the state of participant Q → no need to log coordinator’s decision.

30. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Three-Phase Commit (1) The states of the coordinator and each participant satisfy the following two conditions: 1.There is no single state from which it is possible to make a transition directly to either a COMMIT or an ABORT state. 2.There is no state in which it is not possible to make a final decision, and from which a transition to a COMMIT state can be made.

31. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Three-Phase Commit (2) The client acts as coordinator Phase 1a: Coordinator sends vote-request to participants Phase 1b: When participant receives vote-request it returns either vote-commit or vote-abort to coordinator. If it sends vote-abort, it aborts its local computation Phase 2a: Coordinator collects all votes; if all are vote- commit, it sends prepare-commit to all participants, otherwise it sends global-abort, and halts Phase 2b: Each participant waits for prepare-commit, or waits for global-abort after which it halts Phase 3a: (Prepare to commit) Coordinator waits until all participants have sent ready-commit, and then sends global- commit to all Phase 3b: (Prepare to commit) Participant waits for global- commit

32. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Three-Phase Commit (3) (a) The finite state machine for the coordinator in 3PC. (b) The finite state machine for a participant.

33. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Three-Phase Commit (4) Can P find out what it should it do after crashing in the ready or pre-commit state, even if other participants or the coordinator failed? Reasoning Essence: Coordinator and participants on their way to commit, never differ by more than one state transition Consequence: If a participant timeouts in ready state, it can find out at the coordinator or other participants whether it should abort, or enter pre-commit state Observation: If a participant already made it to the pre-commit state, it can always safely commit (but is not allowed to do so for the sake of failing other processes) Observation: We may need to elect another coordinator to send off the final COMMIT

34. Overview Introduction to Fault Tolerance Basic Concepts Failure Models Failure Masking by Redundancy Process Resilience Design Issues Failure Masking and Replication Agreement in Faulty Systems Failure Detection Reliable Client-server Communication Point-to-point Communication RPC Semantics in The Presence of Failures Reliable Group Communication Basic Reliable-multicasting Schemes Scalability in Reliable Multicasting Atomic Multicast Distributed Commit Two-phase Commit Three-phase Commit Recovery Check-pointing Message Logging Recovery-oriented Computing Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5

35. Recovery: Background Essence When a failure occurs, we need to bring the system into an error- free state: Forward error recovery: Find a new state from which the system can continue operation Backward error recovery: Bring the system back into a previous error-free state Practice Use backward error recovery, requiring that we establish recovery points Observation Recovery in distributed systems is complicated by the fact that processes need to cooperate in identifying a consistent state from where to recover

36. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Recovery – Stable Storage Two disks after recovery: (a) Stable storage. (b) Crash after drive 1 is updated (copy from the primary). (c) Bad spot (copy the good d).

37. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Consistent Recovery State Requirement: Every message that has been received is also shown to have been sent in the state of the sender. Recovery line: Assuming processes regularly checkpoint their state, the most recent consistent global checkpoint.

38. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Independent Checkpointing The domino effect: If checkpointing is done at the “wrong” instants, the recovery line may lie at system startup time → cascaded rollback

39. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Characterizing Message-Logging Schemes Incorrect replay of messages after recovery, leading to an orphan process. Alternative: Instead of taking an (expensive) checkpoint, try to replay your (communication) behavior from the most recent checkpoint → store messages in a log.