1. Reliable Distributed Systems Logical Clocks

2. Time and Ordering We tend to casually use temporal concepts Example: “membership changes dynamically” Implies a notion of time: first membership was X, later membership was Y Challenge: relating local notion of time in a single process to a global notion of time Will discuss this issue before developing multicast delivery ordering options in more detail

3. Time in Distributed Systems Three notions of time: Time seen by external observer. A global clock of perfect accuracy Time seen on clocks of individual processes. Each has its own clock, and clocks may drift out of sync. Logical notion of time: event a occurs before event b and this is detectable because information about a may have reached b.

4. External Time The “gold standard” against which many protocols are defined Not implementable: no system can avoid uncertain details that limit temporal precision! Use of external time is also risky: many protocols that seek to provide properties defined by external observers are extremely costly and, sometimes, are unable to cope with failures

5. Time seen on internal clocks Most workstations have reasonable clocks Clock synchronization is the big problem (will visit topic later in course): clocks can drift apart and resynchronization, in software, is inaccurate Unpredictable speeds a feature of all computing systems, hence can’t predict how long events will take (e.g. how long it will take to send a message and be sure it was delivered to the destination)

6. Logical notion of time Has no clock in the sense of “real-time” Focus is on definition of the “happens before” relationship: “a happens before b” if: both occur at same place and a finished before b started, or a is the send of message m, b is the delivery of m, or a and b are linked by a chain of such events

7. Logical time as a time-space picture a b d c p0p1p2p3p0p1p2p3 a, b are concurrent c happens after a, b d happens after a, b, c

8. Notation Use “arrow” to represent happens- before relation For previous slide: a  c, b  c, c  d hence, a  d, b  d a, b are concurrent Also called the “potential causality” relation

9. Logical clocks Proposed by Lamport to represent causal order Write: LT(e) to denote logical timestamp of an event e, LT(m) for a timestamp on a message, LT(p) for the timestamp associated with process p Algorithm ensures that if a  b, then LT(a) < LT(b)

10. Algorithm Each process maintains a counter, LT(p) For each event other than message delivery: set LT(p) = LT(p)+1 When sending message m, LT(m) = LT(p) When delivering message m to process q, set LT(q) = max(LT(m), LT(q))+1

11. Illustration of logical timestamps LT(a)=1 LT(b)=1 g LT(e)=3 LT(f)=4 p0p1p2p3p0p1p2p3 LT(d)=2 LT(c)=1

12. Concurrent events If a, b are concurrent, LT(a) and LT(b) may have arbitrary values! Thus, logical time lets us determine that a potentially happened before b, but not that a definitely did so! Example: processes p and q never communicate. Both will have events 1, 2,... but even if LT(e)<LT(e’) e may not have happened before e’

13. What about “real-time” clocks? Accuracy of clock synchronization is ultimately limited by uncertainty in communication latencies These latencies are “large” compared with speed of modern processors (typical latency may be 35us to 500us, time for thousands of instructions) Limits use of real-time clocks to “coarse- grained” applications

14. Interpretations of temporal terms Understand now that “a happens before b” means that information can flow from a to b Understand that “a is concurrent with b” means that there is no information flow between a and b What about the notion of an “instant in time”, over a set of processes?

15. Chandy and Lamport: Consistent cuts Draw a line across a set of processes Line cuts each execution Consistent cut has property that the set of included events is closed under happens- before relation: If the cut “includes” event b, and event a happens before b, then the cut also includes event a In practice, this means that every “delivered” message was sent within the cut

16. Illustration of consistent cuts a b g e p0p1p2p3p0p1p2p3 d c f green cuts are consistent red cut is inconsistent

17. Intuition into consistent cuts A consistent cut is a state that could have arisen during execution, depending on how processes were scheduled An inconsistent cut could not have arisen during execution One way to see this: think of process timelines as rubber bands. Scheduler stretches or compresses time but can’t deliver message before it was sent

18. Illustration of consistent cuts a b g e p0p1p2p3p0p1p2p3 d c f green cuts are consistent stretch shrink

19. There may be many consistent cuts through any point in the execution a b g e p0p1p2p3p0p1p2p3 d c f possible cuts define a range of potentially instantaneous system states

20. Illustration of consistent cuts a b g e p0p1p2p3p0p1p2p3 d c f to make the red cut “straight” message f has to travel backwards in time!

21. Reliable Distributed Systems Quorums

22. Quorum replication We developed a whole architecture based on our four-step recipe But there is a second major approach that also yields a complete group communication framework and solutions Based on “quorum” read and write operations Omits notion of process group views

23. Today’s topic Quorum methods from a mile high Don’t have time to be equally detailed We’ll explore How the basic read/update protocol works Failure considerations State machine replication (a form of lock- step replication for deterministic objects) Performance issues

24. A peek at the conclusion These methods are Widely known and closely tied to consensus Perhaps, easier to implement But they have serious drawbacks: Need deterministic components Are drastically slower (10s-100s of events/second) Big win? Recent systems combine quorums with Byzantine Agreement for ultra-sensitive databases

25. Static membership Subsets of a known set of processes E.g. a cluster of five machines, each running replica of a database server Machines can crash or recover but don’t depart permanently and new ones don’t join “out of the blue” In practice the dynamic membership systems can easily be made to work this way… but usually aren’t

26. Static membership example p q r s t client Q read = 2, Q write = 4 readwritereadWrite fails To do a read, this client (or even a group member) must access at least 2 replicas To do a write, must update at least 4 replicas This write will fail: the client only manages to contact 2 replicas and must “abort” the operation (we use this terminology even though we aren’t doing transactions)

27. Quorums Must satisfy two basic rules 1. A quorum read should “intersect” any prior quorum write at >= 1 processes 2. A quorum write should also intersect any other quorum write So, in a group of size N: 1. Q r + Q w > N, and 2. Q w + Q w > N

28. Versions of replicated data Replicated data items have “versions”, and these are numbered I.e. can’t just say “X p =3”. Instead say that X p has timestamp [7,q] and value 3 Timestamp must increase monotonically and includes a process id to break ties This is NOT the pid of the update source… we’ll see where it comes from

29. Doing a read is easy Send RPCs until Q r processes reply Then use the value with the largest timestamp Break ties by looking at the pid For example [6,x] < [9,a] (first look at the “time”) [7,p] < [7,q] (but use pid as a tie-breaker) Even if a process owns a replica, it can’t just trust it’s own data. Every “read access” must collect Q r values first…

30. Doing a write is trickier First, we can’t support incremental updates (x=x+1), since no process can “trust” its own replica. Such updates require a read followed by a write. When we initiate the write, we don’t know if we’ll succeed in updating a quorum of processes wE can’t update just some subset; that could confuse a reader Hence need to use a commit protocol Moreover, must implement a mechanism to determine the version number as part of the protocol. We’ll use a form of voting

31. The sequence of events 1.Propose the write: “I would like to set X=3” 2.Members “lock” the variable against reads, put the request into a queue of pending writes (must store this on disk or in some form of crash-tolerant memory), and send back: “OK. I propose time [t,pid]” Here, time is a logical clock. Pid is the member’s own pid 3.Initiator collects replies, hoping to receive Q w (or more)  Q w OKs Compute maximum of proposed [t,pid] pairs. Commit at that time Abort < Q w OKs

32. Which votes got counted? It turns out that we also need to know which votes were “counted” E.g. suppose there are five group members, A…E and they vote: {[17,A] [19,B] [20,C] [200,D] [21,E]} But somehow the vote from D didn’t get through and the maximum is picked as [21,E] We’ll need to also remember that the votes used to make this decision were from {A,B,C,E}

33. What’s with the [t,pid] stuff? Lamport’s suggestion: use logical clocks Each process receives an update message Places it in an ordered queue And responds with a proposed time: [t,pid] using its own process id for the time The update source takes the maximum Commit message says “commit at [t,pid]” Group members who’s votes were considered deliver committed updates in timestamp order Group members who votes were not considered discard the update and don’t do it, at all.

34. Reliable Distributed Systems Models for Distributed Computing. The Fischer, Lynch and Paterson Result

35. Who needs failure “models”? Role of a failure model Lets us reduce fault-tolerance to a mathematical question In model M, can problem P be solved? How costly is it to do so? What are the best solutions? What tradeoffs arise? And clarifies what we are saying Lacking a model, confusion is common

36. Categories of failures Crash faults, message loss These are common in real systems Crash failures: process simply stops, and does nothing wrong that would be externally visible before it stops These faults can’t be directly detected

37. Categories of failures Fail-stop failures These require system support Idea is that the process fails by crashing, and the system notifies anyone who was talking to it With fail-stop failures we can overcome message loss by just resending packets, which must be uniquely numbered Easy to work with… but rarely supported

38. Categories of failures Non-malicious Byzantine failures This is the best way to understand many kinds of corruption and buggy behaviors Program can do pretty much anything, including sending corrupted messages But it doesn’t do so with the intention of screwing up our protocols Unfortunately, a pretty common mode of failure

39. Categories of failure Malicious, true Byzantine, failures Model is of an attacker who has studied the system and wants to break it She can corrupt or replay messages, intercept them at will, compromise programs and substitute hacked versions This is a worst-case scenario mindset In practice, doesn’t actually happen Very costly to defend against; typically used in very limited ways (e.g. key mgt. server)

40. Models of failure Question here concerns how failures appear in formal models used when proving things about protocols Think back to Lamport’s happens-before relationship,  Model already has processes, messages, temporal ordering Assumes messages are reliably delivered

41. Recall: Two kinds of models We tend to work within two models Asynchronous model makes no assumptions about time Lamport’s model is a good fit Processes have no clocks, will wait indefinitely for messages, could run arbitrarily fast/slow Distributed computing at an “eons” timescale Synchronous model assumes a lock-step execution in which processes share a clock

42. Adding failures in Lamport’s model Also called the asynchronous model Normally we just assume that a failed process “crashes:” it stops doing anything Notice that in this model, a failed process is indistinguishable from a delayed process In fact, the decision that something has failed takes on an arbitrary flavor Suppose that at point e in its execution, process p decides to treat q as faulty….”

43. What about the synchronous model? Here, we also have processes and messages But communication is usually assumed to be reliable: any message sent at time t is delivered by time t+  Algorithms are often structured into rounds, each lasting some fixed amount of time , giving time for each process to communicate with every other process In this model, a crash failure is easily detected When people have considered malicious failures, they often used this model

44. Neither model is realistic Value of the asynchronous model is that it is so stripped down and simple If we can do something “well” in this model we can do at least as well in the real world So we’ll want “best” solutions Value of the synchronous model is that it adds a lot of “unrealistic” mechanism If we can’t solve a problem with all this help, we probably can’t solve it in a more realistic setting! So seek impossibility results

45. Tougher failure models We’ve focused on crash failures In the synchronous model these look like a “farewell cruel world” message Some call it the “failstop model”. A faulty process is viewed as first saying goodbye, then crashing What about tougher kinds of failures? Corrupted messages Processes that don’t follow the algorithm Malicious processes out to cause havoc?

46. Here the situation is much harder Generally we need at least 3f+1 processes in a system to tolerate f Byzantine failures For example, to tolerate 1 failure we need 4 or more processes We also need f+1 “rounds” Let’s see why this happens

47. Byzantine scenario Generals (N of them) surround a city They communicate by courier Each has an opinion: “attack” or “wait” In fact, an attack would succeed: the city will fall. Waiting will succeed too: the city will surrender. But if some attack and some wait, disaster ensues Some Generals (f of them) are traitors… it doesn’t matter if they attack or wait, but we must prevent them from disrupting the battle Traitor can’t forge messages from other Generals

48. Byzantine scenario Attack! Wait… Attack! Attack! No, wait! Surrender! Wait…

49. A timeline perspective Suppose that p and q favor attack, r is a traitor and s and t favor waiting… assume that in a tie vote, we attack p q r s t

50. A timeline perspective After first round collected votes are: {attack, attack, wait, wait, traitor’s-vote} p q r s t

51. What can the traitor do? Add a legitimate vote of “attack” Anyone with 3 votes to attack knows the outcome Add a legitimate vote of “wait” Vote now favors “wait” Or send different votes to different folks Or don’t send a vote, at all, to some

52. Outcomes? Traitor simply votes: Either all see {a,a,a,w,w} Or all see {a,a,w,w,w} Traitor double-votes Some see {a,a,a,w,w} and some {a,a,w,w,w} Traitor withholds some vote(s) Some see {a,a,w,w}, perhaps others see {a,a,a,w,w,} and still others see {a,a,w,w,w} Notice that traitor can’t manipulate votes of loyal Generals!

53. What can we do? Clearly we can’t decide yet; some loyal Generals might have contradictory data In fact if anyone has 3 votes to attack, they can already “decide”. Similarly, anyone with just 4 votes can decide But with 3 votes to “wait” a General isn’t sure (one could be a traitor…) So: in round 2, each sends out “witness” messages: here’s what I saw in round 1 General Smith send me: “attack (signed) Smith ”

54. Digital signatures These require a cryptographic system For example, RSA Each player has a secret (private) key K -1 and a public key K. She can publish her public key RSA gives us a single “encrypt” function: Encrypt(Encrypt(M,K),K -1 ) = Encrypt(Encrypt(M,K -1 ),K) = M Encrypt a hash of the message to “sign” it

55. With such a system A can send a message to B that only A could have sent A just encrypts the body with her private key … or one that only B can read A encrypts it with B’s public key Or can sign it as proof she sent it B can recompute the signature and decrypt A’s hashed signature to see if they match These capabilities limit what our traitor can do: he can’t forge or modify a message

56. A timeline perspective In second round if the traitor didn’t behave identically for all Generals, we can weed out his faulty votes p q r s t

57. A timeline perspective We attack! p q r s t Attack!! Damn! They’re on to me

58. Traitor is stymied Our loyal generals can deduce that the decision was to attack Traitor can’t disrupt this… Either forced to vote legitimately, or is caught But costs were steep! (f+1) * n 2,messages! Rounds can also be slow…. “Early stopping” protocols: min(t+2, f+1) rounds; t is true number of faults

59. Recent work with Byzantine model Focus is typically on using it to secure particularly sensitive, ultra-critical services For example the “certification authority” that hands out keys in a domain Or a database maintaining top-secret data Researchers have suggested that for such purposes, a “Byzantine Quorum” approach can work well They are implementing this in real systems by simulating rounds using various tricks

60. Split secrets In fact BA algorithms are just the tip of a broader “coding theory” iceberg One exciting idea is called a “split secret” Idea is to spread a secret among n servers so that any k can reconstruct the secret, but no individual actually has all the bits Protocol lets the client obtain the “shares” without the servers seeing one-another’s messages The servers keep but can’t read the secret! Question: In what ways is this better than just encrypting a secret?

61. Take-aways? Fault-tolerance matters in many systems But we need to agree on what a “fault” is Extreme models lead to high costs! Common to reduce fault-tolerance to some form of data or “state” replication In this case fault-tolerance is often provided by some form of broadcast Mechanism for detecting faults is also important in many systems. Timeout is common… but can behave inconsistently “View change” notification is used in some systems. They typically implement a fault agreement protocol.

62. Reliable Distributed Systems Fault Tolerance (Recoverability  High Availability)

63. Reliability and transactions Transactions are well matched to database model and recoverability goals Transactions don’t work well for non- database applications (general purpose O/S applications) or availability goals (systems that must keep running if applications fail) When building high availability systems, encounter replication issue

64. Types of reliability Recoverability Server can restart without intervention in a sensible state Transactions do give us this High availability System remains operational during failure Challenge is to replicate critical data needed for continued operation

65. Replicating a transactional server Two broad approaches Just use distributed transactions to update multiple copies of each replicated data item We already know how to do this, with 2PC Each server has “equal status” Somehow treat replication as a special situation Leads to a primary server approach with a “warm standby”

66. Replication with 2PC Our goal will be “1-copy serializability” Defined to mean that the multi-copy system behaves indistinguishably from a single-copy system Considerable form and theoretical work has been done on this As a practical matter Replicate each data item Transaction manager Reads any single copy Updates all copies

67. Observation Notice that transaction manager must know where the copies reside In fact there are two models Static replication set: basically, the set is fixed, although some members may be down Dynamic: the set changes while the system runs, but only has operational members listed within it Today stick to the static case

68. Replication and Availability A series of potential issues How can we update an object during periods when one of its replicas may be inaccessible? How can 2PC protocol be made fault- tolerant? A topic we’ll study in more depth But the bottom line is: we can’t!

69. Usual responses? Quorum methods: Each replicated object has an update and a read quorum Designed so Q u +Q r > # replicas and Q u+ Q u > # replicas Idea is that any read or update will overlap with the last update

70. Quorum example X is replicated at {a,b,c,d,e} Possible values? Q u = 1, Q r = 5 (violates Q U +Q u > 5) Q u = 2, Q r = 4 (same issue) Q u = 3, Q r = 3 Q u = 4, Q r = 2 Q u = 5, Q r = 1 (violates availability) Probably prefer Q u =4, Q r =2

71. Things to notice Even reading a data item requires that multiple copies be accessed! This could be much slower than normal local access performance Also, notice that we won’t know if we succeeded in reaching the update quorum until we get responses Implies that any quorum replication scheme needs a 2PC protocol to commit

72. Next issue? Now we know that we can solve the availability problem for reads and updates if we have enough copies What about for 2PC? Need to tolerate crashes before or during runs of the protocol A well-known problem

73. Availability of 2PC It is easy to see that 2PC is not able to guarantee availability Suppose that manager talks to 3 processes And suppose 1 process and manager fail The other 2 are “stuck” and can’t terminate the protocol

74. What can be done? We’ll revisit this issue soon Basically, Can extend to a 3PC protocol that will tolerate failures if we have a reliable way to detect them But network problems can be indistinguishable from failures Hence there is no commit protocol that can tolerate failures Anyhow, cost of 3PC is very high

75. A quandry? We set out to replicate data for increased availability And concluded that Quorum scheme works for updates But commit is required And represents a vulnerability Other options?

76. Other options We mentioned primary-backup schemes These are a second way to solve the problem Based on the log at the data manager

77. Server replication Suppose the primary sends the log to the backup server It replays the log and applies committed transactions to its replicated state If primary crashes, the backup soon catches up and can take over

78. Primary/backup primary backup Clients initially connected to primary, which keeps backup up to date. Backup tracks log log

79. Primary/backup primary backup Primary crashes. Backup sees the channel break, applies committed updates. But it may have missed the last few updates!

80. Primary/backup primary backup Clients detect the failure and reconnect to backup. But some clients may have “gone away”. Backup state could be slightly stale. New transactions might suffer from this

81. Issues? Under what conditions should backup take over Revisits the consistency problem seen earlier with clients and servers Could end up with a “split brain” Also notice that still needs 2PC to ensure that primary and backup stay in same states!

82. Split brain: reminder primary backup Clients initially connected to primary, which keeps backup up to date. Backup follows log log

83. Split brain: reminder Transient problem causes some links to break but not all. Backup thinks it is now primary, primary thinks backup is down primary backup

84. Split brain: reminder Some clients still connected to primary, but one has switched to backup and one is completely disconnected from both primary backup

85. Implication? A strict interpretation of ACID leads to conclusions that There are no ACID replication schemes that provide high availability Most real systems solve by weakening ACID

86. Real systems They use primary-backup with logging But they simply omit the 2PC Server might take over in the wrong state (may lag state of primary) Can use hardware to reduce or eliminate split brain problem

87. How does hardware help? Idea is that primary and backup share a disk Hardware is configured so only one can write the disk If server takes over it grabs the “token” Token loss causes primary to shut down (if it hasn’t actually crashed)

88. Reconciliation This is the problem of fixing the transactions impacted by lack of 2PC Usually just a handful of transactions They committed but backup doesn’t know because never saw commit record Later. server recovers and we discover the problem Need to apply the missing ones Also causes cascaded rollback Worst case may require human intervention

89. Summary Reliability can be understood in terms of Availability: system keeps running during a crash Recoverability: system can recover automatically Transactions are best for latter Some systems need both sorts of mechanisms, but there are “deep” tradeoffs involved

90. Replication and High Availability All is not lost! Suppose we move away from the transactional model Can we replicate data at lower cost and with high availability? Leads to “virtual synchrony” model Treats data as the “state” of a group of participating processes Replicated update: done with multicast

91. Steps to a solution First look more closely at 2PC, 3PC, failure detection 2PC and 3PC both “block” in real settings But we can replace failure detection by consensus on membership Then these protocols become non-blocking (although solving a slightly different problem) Generalized approach leads to ordered atomic multicast in dynamic process groups

92. Non-blocking Commit Goal: a protocol that allows all operational processes to terminate the protocol even if some subset crash Needed if we are to build high availability transactional systems (or systems that use quorum replication)

93. Definition of problem Given a set of processes, one of which wants to initiate an action Participants may vote for or against the action Originator will perform the action only if all vote in favor; if any votes against (or don’t vote), we will “abort” the protocol and not take the action Goal is all-or-nothing outcome

94. Non-triviality Want to avoid solutions that do nothing (trivial case of “all or none”) Would like to say that if all vote for commit, protocol will commit... but in distributed systems we can’t be sure votes will reach the coordinator! any “live” protocol risks making a mistake and counting a live process that voted to commit as a failed process, leading to an abort Hence, non-triviality condition is hard to capture

95. Typical protocol Coordinator asks all processes if they can take the action Processes decide if they can and send back “ok” or “abort” Coordinator collects all the answers (or times out) Coordinator computes outcome and sends it back

96. Commit protocol illustrated ok to commit?

97. Commit protocol illustrated ok to commit? ok with us

98. Commit protocol illustrated ok to commit? ok with us commit Note: garbage collection protocol not shown here

99. Failure issues So far, have implicitly assumed that processes fail by halting (and hence not voting) In real systems a process could fail in arbitrary ways, even maliciously This has lead to work on the “Byzantine generals” problem, which is a variation on commit set in a “synchronous” model with malicious failures

100. Failure model impacts costs! Byzantine model is very costly: 3t+1 processes needed to overcome t failures, protocol runs in t+1 rounds This cost is unacceptable for most real systems, hence protocols are rarely used Main area of application: hardware fault- tolerance, security systems For these reasons, we won’t study such protocols

101. Commit with simpler failure model Assume processes fail by halting Coordinator detects failures (unreliably) using timouts. It can make mistakes! Now the challenge is to terminate the protocol if the coordinator fails instead of, or in addition to, a participant!

102. Commit protocol illustrated ok to commit? ok with us … times out abort! Note: garbage collection protocol not shown here crashed!

103. Example of a hard scenario Coordinator starts the protocol One participant votes to abort, all others to commit Coordinator and one participant now fail... we now lack the information to correctly terminate the protocol!

104. Commit protocol illustrated ok to commit? ok decision unknown! vote unknown! ok

105. Example of a hard scenario Problem is that if coordinator told the failed participant to abort, all must abort If it voted for commit and was told to commit, all must commit Surviving participants can’t deduce the outcome without knowing how failed participant voted Thus protocol “blocks” until recovery occurs

106. Skeen: Three-phase commit Seeks to increase availability Makes an unrealistic assumption that failures are accurately detectable With this, can terminate the protocol even if a failure does occur

107. Skeen: Three-phase commit Coordinator starts protocol by sending request Participants vote to commit or to abort Coordinator collects votes, decides on outcome Coordinator can abort immediately To commit, coordinator first sends a “prepare to commit” message Participants acknowledge, commit occurs during a final round of “commit” messages

108. Three phase commit protocol illustrated ok to commit? ok.... commit prepare to commit prepared... Note: garbage collection protocol not shown here

109. Observations about 3PC If any process is in “prepare to commit” all voted for commit Protocol commits only when all surviving processes have acknowledged prepare to commit After coordinator fails, it is easy to run the protocol forward to commit state (or back to abort state)

110. Assumptions about failures If the coordinator suspects a failure, the failure is “real” and the faulty process, if it later recovers, will know it was faulty Failures are detectable with bounded delay On recovery, process must go through a reconnection protocol to rejoin the system! (Find out status of pending protocols that terminated while it was not operational)

111. Problems with 3PC With realistic failure detectors (that can make mistakes), protocol still blocks! Bad case arises during “network partitioning” when the network splits the participating processes into two or more sets of operational processes Can prove that this problem is not avoidable: there are no non-blocking commit protocols for asynchronous networks

112. Situation in practical systems? Most use protocols based on 2PC: 3PC is more costly and ultimately, still subject to blocking! Need to extend with a form of garbage collection mechanism to avoid accumulation of protocol state information (can solve in the background) Some systems simply accept the risk of blocking when a failure occurs Others reduce the consistency property to make progress at risk of inconsistency with failed proc.

113. Process groups To overcome cost of replication will introduce dynamic process group model (processes that join, leave while system is running) Will also relax our consistency goal: seek only consistency within a set of processes that all remain operational and members of the system In this model, 3PC is non-blocking! Yields an extremely cheap replication scheme!

114. Failure detection Basic question: how to detect a failure Wait until the process recovers. If it was dead, it tells you I died, but I feel much better now Could be a long wait Use some form of probe But might make mistakes Substitute agreement on membership Now, failure is a “soft” concept Rather than “up” or “down” we think about whether a process is behaving acceptably in the eyes of peer processes

115. Architecture Membership Agreement, “join/leave” and “P seems to be unresponsive” 3PC-like protocols use membership changes instead of failure notification Applications use replicated data for high availability

116. Issues? How to “detect” failures Can use timeout Or could use other system monitoring tools and interfaces Sometimes can exploit hardware Tracking membership Basically, need a new replicated service System membership “lists” are the data it manages We’ll say it takes join/leave requests as input and produces “views” as output

117. Architecture GMS A B C D join leave join A seems to have failed {A} {A,B,D} {A,D} {A,D,C} {D,C} XYZ Application processes GMS processes membership views

118. Issues Group membership service (GMS) has just a small number of members This core set will tracks membership for a large number of system processes Internally it runs a group membership protocol (GMP) Full system membership list is just replicated data managed by GMS members, updated using multicast

119. GMP design What protocol should we use to track the membership of GMS Must avoid split-brain problem Desire continuous availability We’ll see that a version of 3PC can be used But can’t “always” guarantee liveness

120. Class Project Analysis

121. Approach 1 primary backup Clients initially connected to primary, which keeps backup up to date. Backup follows log log

122. Approach 1 primary backup Clients detect the failure and reconnect to backup. But some clients may have “gone away”. Backup state could be slightly stale. New transactions might suffer from this log

123. Approach 2 primary backup Clients initially connected to primary, but through an interface. Backup follows log log

124. Approach 2 primary backup The interface module detects the failure and automatically reconnects to backup. But some clients may have “gone away”. Backup state could be slightly stale. New transactions might suffer from this log