1. Raft consensus
Landon Cox
February 15, 2017

2. Paxos ACM TOCS: Submitted: 1990. Accepted: 1998 Introduced:
Transactions on Computer Systems
Submitted: Accepted: 1998
Introduced:

3. Butler W. Lampson
Butler Lampson is a Technical Fellow at Microsoft Corporation and an Adjunct Professor at MIT…..He was one of the designers of the SDS 940 time-sharing system, the Alto personal distributed computing system, the Xerox 9700 laser printer, two-phase commit protocols, the Autonet LAN, the SPKI system for network security, the Microsoft Tablet PC software, the Microsoft Palladium high-assurance stack, and several programming languages. He received the ACM Software Systems Award in 1984 for his work on the Alto, the IEEE Computer Pioneer award in 1996 and von Neumann Medal in 2001, the Turing Award in 1992, and the NAE’s Draper Prize in 2004.

4. Barbara Liskov MIT professor 2008 Turing Award
“View-stamped replication”
PODC ’88
Very similar to Raft
Different election process

5. At any moment, machine exists in a “state”
State machines
At any moment, machine exists in a “state”
What is a state?
Should think of as a set of named variables and their values

6. State machines What is your state? 4 5 3 2 6 1 Client My state is “2”
Clients can ask a machine about its current state.
What is your state? 4 5 3 2 6 1
Client
My state is “2”

7. “actions” change the machine’s state
State machines
“actions” change the machine’s state
What is an action?
Command that updates named variables’ values

8. “actions” change the machine’s state
State machines
“actions” change the machine’s state
Is an action’s effect deterministic?
For our purposes, yes. Given a state and an action,
we can determine next state w/ 100% certainty.

9. “actions” change the machine’s state
State machines
“actions” change the machine’s state
Is the effect of a sequence of actions deterministic?
Yes, given a state and a sequence of actions, can be 100% certain of end state

10. Replicated state machines
Each state machine should compute same state, even if some fail.
Client
What is the state?
What is the state?
Client
What is the state?
Client

11. Replicated state machines
What has to be true of the actions that clients submit?
Applied in same order
Client
Apply action c.
Apply action a.
Client
Apply action b.
Client

12. State machines
How should a machine make sure it applies action in same order across reboots?
Store them in a log!
Action …

13. Replicated state machines
Can reduce problem of consistent, replicated states to consistent, replicated logs … … … …

14. Replicated state machines
How to make sure that logs are consistent?
Two-phase commit? … … … … …

15. Replicated state machines
What is the heart of the matter?
Have to agree on the leader, outside of the logs.
Leader=L
Leader=L
Apply action a. … …
Client
Leader=L
Leader=L … …

16. Key elements of consensus
Leader election
Who is in charge?
Log replication
What are the actions and what is their order?
Safety
What is true for all states, in all executions (including failures)?
e.g., either we haven’t agreed or we all agree on the same value

17. Key elements of consensus
Leader election
Who is in charge?
Log replication
What are the actions and what is their order?
Safety
What is true for all states, in all executions (including failures?
e.g., either we haven’t agreed or we all agree on the same value

18. Server states
Three states: leader, follower, candidate F L C

19. Server state: follower
Passive state: respond to candidates and leaders F L C

20. Server state: leader Server handles all client requests F L
What should happen if a client sends a request to a follower?
Follower forwards request to leader.

21. Server state: candidate
An intermediate state, used during elections F L C

22. Time divided into terms
Election
Normal operation
Election
Normal operation
Leader unknown
Leader known
What happened here?
Election failed

23. Terms as a logical clock
All servers maintain the current term
Terms increase monotonically
Maintained as a logical clock
Terms are exchanged whenever servers communicate
What if A’s term is bigger than B’s?
B updates its term to A’s
What if A’s term is bigger and B thinks of itself as the leader?
B reverts to a follower state
What if A’s term is bigger, and it receives a request from B?
A rejects B’s request
B must be up to date to issue requests

24. Server state: followerC
Current term = 0 S1 F L C
Current term = 0 S3 F L C
Current term = 0
All servers start as followers.
All servers have local timers.
Note: no bootstrapping problem!

25. Server state: followerC
Current term = 0 S1 F L C
Current term = 0 S3 F L C
Current term = 0
Server remains follower as long as it receives periodic valid messages from a leader or candidate.
Called a “heartbeat” message.

26. Server state: followerC
Current term = 0 S1 F L C
Current term = 0 S3 F L C
Current term = 0
What should server assume if no heartbeat?
Assume no viable leader, start election.

27. Server state: followerC
Current term = 0 S1 F L C
Current term = 0 S3 F L C
Current term = 0
Who should the server nominate?
How about itself? At least it knows that it’s running.

28. Server state: candidateF L C
Current term = 0 S1 F L C
Current term = 1 S3 F L C
Current term = 0
To start an election:
Increment current term and set state to candidate

29. Server state: candidateF L C
Current term = 0 S1 F L C
Current term = 1 S3 F L C
Current term = 0
Need to collect votes. For whom should the server vote?
Itself, of course! Major qualification: It’s running.

30. Server state: candidateF L C
Current term = 0 S1
Vote in term 1 F L C
Current term = 1 S3
Votes
S1=S1
S2=?
S3=?
Vote in term 1 F L C
Current term = 0
How should S2, S3 respond to vote request?

31. Server state: candidateF L C
Current term = 1 S1
S1 in term 1 F L C
Current term = 1 S3
Votes
S1=S1
S2=?
S3=?
S1 in term 1 F L C
Current term = 1
How should S2, S3 respond to vote request?
Increment term, vote for S1 … why vote for S1?
Our goal is consensus, and we know the collector voted for itself.

32. Server state: candidateF L C
Current term = 1 S1 F L C
Current term = 1 S3
Votes
S1=S1 S2=S1 S3=S1 F L C
Current term = 1
What should S1 do next?
Count votes (majority wins)
Make itself the leader, start sending heartbeats.

33. Server state: candidateF L C
Current term = 1
Leader = S1 S1
S1 is leader F L C
Current term = 1 S3
Leader = S1
S1 is leader F L C
Current term = 1
Leader = S1

34. Server state: candidateF L C
Current term = 1
Leader = S1 S1 F L C
Current term = 1 S3
Leader = S1 F L C
Current term = 1
Leader = S1
How many faults can we tolerate?
One. Need two/three to vote for same new leader

35. Server state: candidateF L C
Current term = 1 S1
Vote in term 1 F L C
Current term = 1 S3
Votes
S1=S1
S2=?
S3=?
Vote in term 1 F L C
Current term = 1 C
Votes
S1=?
S2=? S3=S3 F
Who votes for whom if S1 and S3 both call elections?

36. Server state: candidateF L C
Current term = 1 S1
Vote in term 1 F L C
Current term = 1 S3
Votes
S1=S1 S2=S3 S3=S3
Vote in term 1 F L C
Current term = 1 C
Votes
S1=S1 S2=S3 S3=S3 F
Who votes for whom if S1 and S3 both call elections?
S2 votes for the server that asked first

37. Server state: candidateF L C
Current term = 1 S1 F L C
Current term = 1 S3
Votes
S1=S1 S2=S3 S3=S3 F L C
Current term = 1 C
Votes
S1=S1 S2=S3 S3=S3 F
What does S1 do if it loses the election?

38. Server state: candidateF L C
Current term = 1 S1 F L C
Current term = 1 S3
Votes
S1=S1 S2=S3 S3=S3 F L C
Current term = 1 C
Votes
S1=S1 S2=S3 S3=S3 F
What does S1 do if it loses the election?
Moves back to a follower state.

39. Server state: candidateF L C
Current term = 1
S3 is leader
Leader = S3 S1 L
Current term = 1 C S3
Leader = S3 F F L C
Current term = 1 C
Leader = S3 F

40. Server state: followerC
Current term = 0 S1 F L C
Current term = 0 S3 F L C
Current term = 0
Can all three servers nominate themselves?

41. Server state: candidateF L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3 S1 F L C
Current term = 1 S3
Votes
S1=S1 S2=S2 S3=S3 F L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3
Can all three servers nominate themselves?
Sure!

42. Server state: candidateF L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3 S1 F L C
Current term = 1 S3
Votes
S1=S1 S2=S2 S3=S3 F L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3
Could this happen indefinitely?

43. Server state: candidateF L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3 S1 F L C
Current term = 1 S3
Votes
S1=S1 S2=S2 S3=S3 F L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3
Could this happen indefinitely?
Yes, there is not way to prevent this from happening.
The worst possible thing is still possible.

44. Server state: candidateF L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3 S1 F L C
Current term = 1 S3
Votes
S1=S1 S2=S2 S3=S3 F L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3
How do we make this less likely to occur?

45. Server state: candidateF L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3 S1 F L C
Current term = 1 S3
Votes
S1=S1S2=S2 S3=S3 F L C
Current term = 1
Votes
S1=S1 S2=S2 S3=S3
How do we make this less likely to occur?
Randomize election timeouts
i.e., wait a random period before new election

46. Server state: candidateF L C
Current term = 1 S1 F L C
Current term = 1 S3 F L C
Current term = 1
Can we ever have two leaders?

47. Server state: candidateF L C
Current term = 1 S1 F L C
Current term = 1 S3 F L C
Current term = 1
Can we ever have two leaders?
No, votes either split or converge w/ three nodes.

48. Key elements of consensus
Leader election
Who is in charge?
Log replication
What are the actions and what is their order?
Safety
What is true for all states, in all executions (including failures)?
e.g., either we haven’t agreed or we all agree on the same value

49. Each node maintains an action log.
Leader=L
Each node maintains an action log.
Each entry contains an action and a term.
The term indicates when the leader received the action.
Leader=L 1
x  3 1
y  1
Leader=L 1
x  3 1
y  1 1
x  3 1
y  1

50. When a request comes in, leader appends action to its log.
Leader=L
When a request comes in, leader appends action to its log.
Client
y  5
Leader=L 1
x  3 1
y  1
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1

51. Next, the leader tells other servers to append action.
Leader=L
Next, the leader tells other servers to append action.
Client
y  5 1
y  5
Leader=L 1
x  3 1
y  1
Leader=L 1
y  5 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1

52. The leader waits for confirmation that the entry was appended.
Leader=L
The leader waits for confirmation that the entry was appended.
Client
y  5
ack
Leader=L 1
x  3 1
y  1 1
y  5
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1 1
y  5

53. Leader=L y  5 Client 3/3 … commit! Leader=L Leader=L
As with leader election, majority rules.
Entry is committed once the leader that received it has replicated the entry on a majority of servers (for now).
Leader=L
Client
y  5
3/3 … commit!
Leader=L 1
x  3 1
y  1 1
y  5
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1 1
y  5

54. If action commits, leader updates state machine. Leader=L
Client
y  5
Leader=L 1
x  3 1
y  1 1
y  5
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1 1
y  5

55. Leader reports success to client and other servers if action commits
Leader=L
Client
y  5 C C
Leader=L 1
x  3 1
y  1 1
y  5
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1 1
y  5

56. What happens if one follower fails?
Action will still commit, since 2/3 are alive.
Leader=L
Leader=L 1
x  3 1
y  1
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1 1
y  5

57. Leader=L Leader=L Leader=L
What happens if both followers fail: (1) after adding entry to logs, and (2) before acking?
Leader=L
The leader will keep trying to append. If one server comes back, the action will eventually commit.
Leader=L 1
x  3 1
y  1 1
y  5
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1 1
y  5

58. Leader=L Leader=L Leader=L
How long might a client have to wait in this case?
Leader=L
A long time, i.e., until two machines append the action
Leader=L 1
x  3 1
y  1 1
y  5
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1 1
y  5

59. Leader=L Leader=L Leader=L
What happens if leader fails: (1) after adding entry to log, and (2) before contacting followers?
Leader=L
Action will not commit. New entries will occur under next term.
Leader=L 1
x  3 1
y  1
Leader=L 1
x  3 1
y  1 1
y  5 1
x  3 1
y  1

60. As with leader election, majority rules for commit.
For now: entry is committed once the leader that received it has replicated the entry on a majority of servers 1 2 3 4 5 6 7 8
Log index 1
x  3
y  1
y  9 2
x  2 3
x  0
y  7
x  5
x  4
Leader
Followers

61. In this example, which committed entry has the highest index? Entry 71 2 3 4 5 6 7 8
Log index 1
x  3
y  1
y  9 2
x  2 3
x  0
y  7
x  5
x  4
Leader
Followers

62. <sigh>This picture is far too orderly and easy to understand.
Committing a log entry also commits all entries in leader’s log with lower index.
Note: the current leader’s index is the one that counts! 1 2 3 4 5 6 7 8
Log index 1
x  3
y  1
y  9 2
x  2 3
x  0
y  7
x  5
x  4
Leader
Followers
<sigh>This picture is far too orderly and easy to understand.
No guarantee the world will look like this.</sigh>

63. This can be the state of the logs when a leader comes to power.
Each server has assigned each entry (1) a term, and (2) an index.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6 6
Followers
(d) 1 1 1 4 4 5 5 6 6 6 7 7
(e) 1 1 1 4 4 4 4
(f) 1 1 1 2 2 2 3 3 3 3 3

64. What is the current term?
We are in term 8.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6 6
Followers
(d) 1 1 1 4 4 5 5 6 6 6 7 7
(e) 1 1 1 4 4 4 4
(f) 1 1 1 2 2 2 3 3 3 3 3

65. Why aren’t there any entries for term 8?
Because the leader hasn’t received any requests yet.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6 6
Followers
(d) 1 1 1 4 4 5 5 6 6 6 7 7
(e) 1 1 1 4 4 4 4
(f) 1 1 1 2 2 2 3 3 3 3 3

66. This can be the state of the logs when the leader comes to power.
Each server has assigned each entry (1) a term, and (2) an index.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
(b) 1 1 1 4
What’s wrong with the logs of (a) and (b)?
They are missing log entries.

67. How might this have happened?
This can be the state of the logs when the leader comes to power.
Each server has assigned each entry (1) a term, and (2) an index.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
(b) 1 1 1 4
How might this have happened?
They could have gone offline and come back; (a) during term 6, (b) during term 4

68. This can be the state of the logs when the leader comes to power.
Each server has assigned each entry (1) a term, and (2) an index.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
What’s wrong with the logs of (c) and (d)?
They have extra log entries.
(c) 1 1 1 4 4 5 5 6 6 6 6
(d) 1 1 1 4 4 5 5 6 6 6 7 7

69. How might this have happened?
This can be the state of the logs when the leader comes to power.
Each server has assigned each entry (1) a term, and (2) an index.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
How might this have happened?
(c) was leader for term 6, added entry and crashed; (d) was leader for 7, added entries and crashed
(c) 1 1 1 4 4 5 5 6 6 6 6
(d) 1 1 1 4 4 5 5 6 6 6 7 7

70. This can be the state of the logs when the leader comes to power.
Each server has assigned each entry (1) a term, and (2) an index.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
What’s wrong with the logs of (e) and (f)?
They have extra log entries and missing log entries.
(e) 1 1 1 4 4 4 4
(f) 1 1 1 2 2 2 3 3 3 3 3

71. This can be the state of the logs when the leader comes to power.
Each server has assigned each entry (1) a term, and (2) an index.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(f) was leader for term 2, added several entries and crashed.
(f) quickly restarted and became leader for term 3, added more entries and crashed before any entries from terms 2 or 3 could commit
How could this have happened to (f)?
(e) 1 1 1 4 4 4 4
(f) 1 1 1 2 2 2 3 3 3 3 3

72. Goal: (eventually) converge on a sane state from logs like this.
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6 6
Followers
(d) 1 1 1 4 4 5 5 6 6 6 7 7
(e) 1 1 1 4 4 4 4
(f) 1 1 1 2 2 2 3 3 3 3 3

73. Servers have to keep track of committed log entry with highest index
Servers have to keep track of committed log entry with highest index. What are those here?
Leader=L
2 for all three servers.
Client
y  5
Leader=L 1
x  3 1
y  1 1 2
Leader=L 1
x  3 1
y  1 1
y  5 1 2 3 1
x  3 1
y  1 1 2

74. Leader=L Client y  5 Leader=L 1 2 Leader=L 1 2 3 1 2
Leader reports its highest index of committed action when forwarding request to followers.
This is how followers update their state machines.
Leader=L
Client
y  5
Term=1
y  5
High=2
Leader=L 1
x  3 1
y  1 1 2
Leader=L
Term=1
y  5
High=2 1
x  3 1
y  1 1
y  5 1 2 3 1
x  3 1
y  1 1 2

75. Leader also reports highest index immediately preceding current append request.
Leader=L
Client
y  5
Term=1
y  5
High=2
Pred=2
Leader=L 1
x  3 1
y  1 1 2
Leader=L
Term=1
y  5
High=2
Pred=2 1
x  3 1
y  1 1
y  5 1 2 3 1
x  3 1
y  1 1 2

76. Could this happen?
Yes, if follower failed before it received action with index 2 from term 1
Leader=L
Client
y  5
Term=1
y  5
High=2
Pred=2
Leader=L 1
x  3 1
Leader=L
Term=1
y  5
High=2
Pred=2 1
x  3 1
y  1 1
y  5 1 2 3 1
x  3 1
y  1 1 2

77. Should the recovered follower append the new action?
If it did, then it would have a different action in index 2 during term 1. Better to reject new action and append missing committed actions first.
Leader=L
Client
y  5
Term=1
y  5
High=2
Pred=2
Leader=L 1
x  3 1
Leader=L
Term=1
y  5
High=2
Pred=2 1
x  3 1
y  1 1
y  5 1 2 3 1
x  3 1
y  1 1 2

78. Yes, since we can still achieve majority w/o it.
Can new action commit while recovered server receives actions it missed while down?
Leader=L
Yes, since we can still achieve majority w/o it.
Client
y  5
Term=1
y  5
High=2
Pred=2
Leader=L 1
x  3 1
Leader=L
Term=1
y  5
High=2
Pred=2 1
x  3 1
y  1 1
y  5 1 2 3 1
x  3 1
y  1 1 2

79. This gives us a very important system property:
Two entries in different logs w/ same index and term always have the same action.
Leader=L
Client
y  5
Term=1
y  5
High=2
Pred=2
Leader=L 1
x  3 1
Leader=L
Term=1
y  5
High=2
Pred=2 1
x  3 1
y  1 1
y  5 1 2 3 1
x  3 1
y  1 1 2

80. Leader=L Client y  5 Leader=L 1 Leader=L 1 2 3 1 2
Two entries in different logs w/ same index and term always have the same action. Why?
Leader=L
In a term, leader creates at most one entry with a given index. Followers catch up first when behind.
Client
y  5
Term=1
y  5
High=2
Pred=2
Leader=L 1
x  3 1
Leader=L
Term=1
y  5
High=2
Pred=2 1
x  3 1
y  1 1
y  5 1 2 3 1
x  3 1
y  1 1 2

81. Log Matching Property
Two entries in different logs with same index and term
 The entries store the same action
The logs are identical in all preceding entries
Does the matching property hold below?
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader 1 1 1 4 4 5 5 6 6 6 7 7 1 1 1 4 4 4 4
Followers 1 1 1 2 2 2 3 3 3 3 3

82. Log Matching Property
Two entries in different logs with same index and term
 The entries store the same action
The logs are identical in all preceding entries
Why is the first part always true?
One leader/term
If leader fails, term changes
Leader inserts a new entry once at a given index
NOTE: this is not enough to ensure the second part

83. Log Matching Property
Two entries in different logs with same index and term
 The entries store the same action
The logs are identical in all preceding entries
Ensuring the second part requires extra check by follower
Leader sends followers append(term, last_index, action)
Follower must check that last entry has same term and last_index
If not, the follower refuses the new entry
Otherwise, follower may append new entry containing action

84. Log Matching Property
Two entries in different logs with same index and term
 The entries store the same action
The logs are identical in all preceding entries
Do logs always agree on indexes, terms?
No, indexes at logs may have different actions and terms
Goal of repairing logs is to increase entries w/ same index, term 1 1 1 4 4 5 5 6 6 6 7 7 1 1 1 4 4 4 4 1 1 1 2 2 2 3 3 3 3 3

85. How do we make the logs look more like one another?
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6 6
Followers
(d) 1 1 1 4 4 5 5 6 6 6 7 7
(e) 1 1 1 4 4 4 4
(f) 1 1 1 2 2 2 3 3 3 3 3

86. How do we make the logs look more like one another?
Raft: How do we make the logs look like the leader’s?
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6 6
Followers
(d) 1 1 1 4 4 5 5 6 6 6 7 7
(e) 1 1 1 4 4 4 4
(f) 1 1 1 2 2 2 3 3 3 3 3

87. Repairing logs Leader Append-only Property
Leader’s log is treated as ground truth
Leader never overwrites or deletes log entries
Leader only appends to its log
Basic idea: make the leader’s life simple
How would a leader know that follower is missing entries?
Follower refuses an append request (based on last_index)
Leader must monitor each follower’s progress (next_index)
Send log entries until followers are caught up

88. Leader initially assumes followers’ logs look like hers
next_index[a]=11
next_index[b]=11
next_index[c]=11
Leader initially assumes followers’ logs look like hers
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6
Leader
(a) 1 1 1 4 4 5 5 6 6
Followers
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6 6

89. next_index[a]=11
next_index[b]=11
next_index[c]=11
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader
(a) 1 1 1 4 4 5 5 6 6 8
Prev={term 6,
index 10}
(b) 1 1 1 4 8
Prev={term 6,
index 10}
(c) 1 1 1 4 4 5 5 6 6 6 6 8
Prev={term 6,
index 10}

90. next_index[a]=10
next_index[b]=10
next_index[c]=10
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader ✗
(a) 1 1 1 4 4 5 5 6 6 8
Prev={term 6,
index 10} ✗
(b) 1 1 1 4 8
Prev={term 6,
index 10} ✗
(c) 1 1 1 4 4 5 5 6 6 6 6 8
Prev={term 6,
index 10}

91. next_index[a]=10
next_index[b]=10
next_index[c]=10
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader
(a) 1 1 1 4 4 5 5 6 6 6
Prev={term 6,
index 9}
(b) 1 1 1 4 6
Prev={term 6,
index 9}
(c) 1 1 1 4 4 5 5 6 6 6 6
Prev={term 6,
index 9} 6

92. What should happen to this log entry?
next_index[a]=10
next_index[b]=10
next_index[c]=10
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader ✓
(a) 1 1 1 4 4 5 5 6 6 6
Prev={term 6,
index 9} ✗
(b) 1 1 1 4 6
Prev={term 6,
index 9} ✓
(c) 1 1 1 4 4 5 5 6 6 6
Prev={term 6,
index 9} 6 6
What should happen to this log entry?
It should be deleted

93. What entry does the leader send to (a) next?
next_index[a]=11
next_index[b]=9
next_index[c]=11
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader
(a) 1 1 1 4 4 5 5 6 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6
What entry does the leader send to (a) next?
Entry at index 11 (term 8); this succeeds.

94. What entry does the leader send to (b) next?
next_index[a]=11
next_index[b]=9
next_index[c]=11
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader
(a) 1 1 1 4 4 5 5 6 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6
What entry does the leader send to (b) next?
Entry at index 9 (term 6); this fails and next_index[b]  8.
Eventually, (b) will accept entry from term 4 at index 5, and it will catch up with everyone else.

95. What entry does the leader send to (c) next?
next_index[a]=11
next_index[b]=9
next_index[c]=11
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader
(a) 1 1 1 4 4 5 5 6 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6
What entry does the leader send to (c) next?
Entry at index 11 (term 8); this succeeds.

96. next_index[a]=11
next_index[b]=9
next_index[c]=11
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader
(a) 1 1 1 4 4 5 5 6 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6
As described, this should make you uncomfortable. Why?
We could choose a bad leader (e.g., one w/ an empty log).
This could delete committed entries!

97. How do we prevent this from happening?
next_index[a]=11
next_index[b]=9
next_index[c]=11
Log index 1 2 3 4 5 6 7 8 9 10 11 12 1 1 1 4 4 5 5 6 6 6 8
Leader
(a) 1 1 1 4 4 5 5 6 6 6
(b) 1 1 1 4
(c) 1 1 1 4 4 5 5 6 6 6
How do we prevent this from happening?
We have to be smarter about electing an leader…

98. Refined leader election
Goal
Leader must have all committed actions
How actions are committed
Leader accepts action from client
Leader forwards action to followers
Majority append must occur before next term
If new term starts before majority append, no commit
Commit cannot occur until later action commits

99. Refined leader election
Initial election criterion
Any candidate can be elected
Normally the candidate who starts the election wins
Problem: leader can force stale log on followers
What must be true of an elected leader?
Leader must have all prior committed actions
If leader is missing a committed action, it may be lost

100. Refined leader election
New election criterion
Vote for candidate with most “up to date” log
If voter’s log is more recent than candidate’s, vote self
Which log is more up to date?
(a) because its last entry is from term 4 (> term 3)
If last log entry is from later term, log is more up to date
(a) 1 1 1 4 4 4 4
(b) 1 1 1 2 2 2 3 3 3 3 3

101. Refined leader election
New election criterion
Vote for candidate with most “up to date” log
If voter’s log is more recent than candidate’s, vote self
Which log is more up to date?
(b) because its last term-4 entry has higher index
When last log entries are from same term, higher index is more up to date
(a) 1 1 1 4 4 4 4
(b) 1 1 1 4 4 4 4 4

102. Refined leader election
New election criterion
Vote for candidate with most “up to date” log
If voter’s log is more recent than candidate’s, vote self
How to decide which log is more up to date
If last log entry is from later term, log is more up to date
When last log entries are from same term, higher index is more up to date

103. Refined leader election
Goal
Leader must have all committed actions
Recall how actions had been committed
Leader accepts action from client
Leader forwards action to followers
If a majority of followers append action, it’s committed
If action commits, leader returns to client
If leader fails, future leaders can commit
Unfortunately, there’s a problem …

104. Committing actions
Log index 1 2
(a) 1 2
(b) 1 2
(c) 1
(d) 1
(e) 1

105. How was (e) elected if (b) has a more up-to-date log?
Committing actions
Log index 1 2
(a) 1 2
How was (e) elected if (b) has a more up-to-date log?
(b) 1 2
(c) 1
Votes from (c) and (d)
(d) 1
(e) 1

106. Committing actions
Log index 1 2
(a) 1 2
(b) 1 2
(c) 1
(d) 1
(e) 1 3

107. Committing actions
Log index 1 2
(a) 1 2
(b) 1 2
(c) 1
(d) 1
(e) 1 3

108. What is the first thing that (a) will do?
Committing actions
Log index 1 2 3
(a) 1 2 4
What is the first thing that (a) will do?
(b) 1 2
(c) 1
Forward entry at index 3 to followers, but this will fail at (c), because (c) does not have entry at index 2 yet
(d) 1
(e) 1 3

109. Which entry will (a) send to (c) first?
Committing actions
Log index 1 2 3
(a) 1 2 4
Which entry will (a) send to (c) first?
(b) 1 2
(c) 1 2
Entry at index 2
(d) 1
(e) 1 3

110. Has the entry at index 2 been committed?
Committing actions
Log index 1 2 3
(a) 1 2 4
Has the entry at index 2 been committed?
(b) 1 2
(c) 1 2
By our previous definition, yes. But note that it reached a majority of nodes in term 4, not term 2 (i.e., after it was created)
(d) 1
(e) 1 3

111. Committing actions 1 2 4 1 2 1 2 1 1 3 Log index 1 2 3 (a) (b) (c) (d)

112. How could (e) have been elected?
Committing actions
Log index 1 2 3
(a) 1 2 4
How could (e) have been elected?
(b) 1 2
(c) 1 2
Votes from (b),(c), and (d)
(d) 1
(e) 1 3

113. What is the first thing that (e) will do?
Committing actions
Log index 1 2 3
(a) 1 2 4
What is the first thing that (e) will do?
(b) 1 2
(c) 1 2
Forward entry at index 2 to followers
(d) 1
(e) 1 3

114. What will happen to other logs’ entries at index 2?
Committing actions
Log index 1 2 3
(a) 1 2 4
What will happen to other logs’ entries at index 2?
(b) 1 2 3
(c) 1 3 2
They will be replaced by the leader’s entry from term 3
(d) 1 3
(e) 1 3

115. What will happen to (a)’s entry at index 3?
Committing actions
Log index 1 2 3
(a) 1 3 2 4
What will happen to (a)’s entry at index 3?
(b) 1 2 3
(c) 1 3 2
It will be deleted
(d) 1 3
(e) 1 3

116. Committing actions
So, are actions stored on a majority really committed?
No, only if they reach a majority in the term they’re created.
Log index 1 2 3
(a) 1 3 2
(b) 1 2 3
(c) 1 2 3
(d) 1 3
(e) 1 3

117. Can (e) win the election?
Committing actions
Log index 1 2
(a) 1 2
Can (e) win the election?
(b) 1 2
(c) 1 2
No, only (d) will vote for it
(d) 1
(e) 1

118. Can (b) or (c) win the election?
Committing actions
Log index 1 2
(a) 1 2
Can (b) or (c) win the election?
(b) 1 2
(c) 1 2
Yes, either can win.
(d) 1
(e) 1

119. What will (b) send to (d) and (e) first?
Committing actions
Log index 1 2 3
(a) 1 2
What will (b) send to (d) and (e) first?
(b) 1 2 3
STOPPED
(c) 1 2 3
Entry at index 2
(d) 1 2 3
(e) 1 2 3

120. Committing actions (say 2 reached majority in term 4) 1 2 4 1 2 1 2 1
Log index 1 2 3
(a) 1 2 4
Can entry 2 ever commit?
(b) 1 2
(c) 1 2
Yes, if a subsequent entry (e.g., index 3, term 4) commits, all prior entries are also committed
(d) 1
(e) 1 3

121. Entry 3 is now committed, so entry 2 is as well
Committing actions
(say 2 reached majority in term 4)
Log index 1 2 3
(a) 1 2 4
Entry 3 is now committed, so entry 2 is as well
(b) 1 2 4
(c) 1 2 4
(d) 1
(e) 1 3

122. Intertwining elections and log repair
Log index 1 2 3 4 5 6 7 8 9 10 11 12
(a) 1 1 1 4 4 5 5 6 6 6
(b) 1 1 1 4 4 4 4
(c) 1 1 1 2 2 2 3 3 3 7
Is it possible for servers’ logs to reach this state?
Why or why not?

123. Intertwining elections and log repair
Log index 1 2 3 4 5 6 7 8 9 10 11 12
(a) 1 1 1 5 5 6 6 6 6
(b) 1 1 1 5 5 5 5
(c) 1 1 1 2 4
Is it possible for servers’ logs to reach this state?
Why or why not?

124. Intertwining elections and log repair
Log index 1 2 3 4 5 6 7 8 9 10 11 12
(a) 1 1 1 2 6
(b) 1 1 1 2 4
(c) 1 1 1 2 5
Is it possible for servers’ logs to reach this state?
Why or why not?

125. Key elements of consensus
Leader election
Who is in charge?
Log replication
What are the actions and what is their order?
Safety
What is true for all states, in all executions?
e.g., either we haven’t agreed or we all agree on the same value

126. Review: Log Matching Property
Two entries in different logs with same index and term
 The entries store the same action
The logs are identical in all preceding entries
Critical for Safety proof that follows

127. Log Matching Property
Two entries in different logs with same index and term
 The entries store the same action
The logs are identical in all preceding entries
Why is the first part always true?
One leader/term
If leader fails, term changes
Leader inserts a new entry once at a given index
 every action is assigned a unique term and index
NOTE: this is not enough to ensure the second part

128. Log Matching Property
Two entries in different logs with same index and term
 The entries store the same action
The logs are identical in all preceding entries
Ensuring the second part requires extra check by follower
Leader sends followers append(term, last_index, action)
Follower must check that last entry has same term and last_index
If not, the follower refuses the new entry
Otherwise, follower may append new entry containing action

129. Log Matching Property
Two entries in different logs with same index and term
 The entries store the same action
The logs are identical in all preceding entries
Do logs always agree on indexes, terms?
No, indexes at logs may have different actions and terms
Goal of repairing logs is to increase entries w/ same index, term 1 1 1 4 4 5 5 6 6 6 7 7 1 1 1 4 4 4 4 1 1 1 2 2 2 3 3 3 3 3

130. Safety Leader Completeness Property Proof setup
If a log entry is committed in a given term
entry will be in leaders’ logs in all future terms
Proof setup
Majorities required to commit and elect
What must be true of any two majorities?
They must overlap
This will be the linchpin of the proof
Question: is a node with committed entries to be elected?

131. Proof by contradiction
Assume that node missing a committed entry e is elected
Entry e was committed in term T
Leader missing committed entry e is elected in term U
U is earliest term after T whose leader is missing e

132. Proof by contradiction
Assume that node missing a committed entry e is elected
Entry e was committed in term T
Leader missing committed entry e is elected in term U
U is earliest term after T whose leader is missing e
Majorities are needed to commit and elect
At least one node accepted e in term T and voted for leader in U
Call this node “the voter”

133. Proof by contradiction
Assume that node missing a committed entry e is elected
Entry e was committed in term T
Leader missing committed entry e is elected in term U
U is earliest term after T whose leader is missing e
Majorities are needed to commit and elect
At least one node accepted e in term T and voted for leader in U
Call this node “the voter”
Possible that the voter obtained entry e after voting in U?
If so, the voter would have rejected e from the leader in term T
And we know that the voter accepted e in term T
So, we know that the voter obtained e before voting in term U

134. Proof by contradiction
Assume that node missing a committed entry e is elected
Entry e was committed in term T
Leader missing committed entry e is elected in term U
U is earliest term after T whose leader is missing e
Majorities are needed to commit and elect
At least one node accepted e in term T and voted for leader in U
Call this node “the voter”
Voter obtained e before voting for leader in U
We know that the voter voted for leader in U
 Leader in U’s log must have been as up to date as the voter’s log

135. Proof by contradiction
Assume that node missing a committed entry e is elected
Entry e was committed in term T
Leader missing committed entry e is elected in term U
U is earliest term after T whose leader is missing e
Majorities are needed to commit and elect
At least one node accepted e in term T and voted for leader in U
Call this node “the voter”
Voter obtained e before voting for leader in U
Leader in U’s log must have been as up to date as the voter’s log
This will give us our contradictions

136. Proof by contradiction
Leader in U’s log must have been as up to date as the voter’s log
If leader’s log was the same as the voter’s, then what must be true?
Leader’s log must have contained committed entry e
This contradicts our assumption, so leader’s log must ≠ voter’s

137. Proof by contradiction
Leader in U’s log must have been as up to date as the voter’s log
If leader’s log was the same as the voter’s, then what must be true?
Leader’s log must have contained committed entry e
This contradicts our assumption, so leader’s log must ≠ voter’s
What if leader’s log has same last term as voter’s?
Then leader’s log must have been longer than the voter’s
If longer with same last term, then there must be an entry from last term in common
Log Matching Property: entries with same index and term  all preceding are same
Thus, the leader’s log must contain entry e
This contradicts our assumption, so leader’s last log term > voter’s last log term

138. Proof by contradiction
Leader in U’s log must have been as up to date as the voter’s log
If leader’s log was the same as the voter’s, then what must be true?
Leader’s log must have contained committed entry e
This contradicts our assumption, so leader’s log must ≠ voter’s
What if leader’s log has same last term as voter’s?
Then leader’s log must have been longer than the voter’s
If longer with same last term, then there must be an entry from last term in common
Log Matching Property: entries with same index and term  all preceding are same
Thus, the leader’s log must contain entry e
This contradicts our assumption, so leader’s last log term > voter’s last log term
What if leader’s last log term is later than voter’s last log term?

139. Log index 1 2 3 4 5 6 7 8 9 10 11 12 L
Leader(U)
Voter T
Leader (L)
Last term in Leader(U)’s log is L, and L > the last term in Voter’s log
What does this tell us about the relationship of L and T?
L > T, since Voter has entry e from T
Entry e: term T at index 6 in Voter’s log

140. Log index 1 2 3 4 5 6 7 8 9 10 11 12 L
Leader(U)
Voter T
Leader (L) T
Last term in Leader(U)’s log is L, and L > the last term in Voter’s log
Do we know whether the leader for L, Leader(L), has e?
Yes, it must, since we assumed that Leader(U) was first w/o e
Entry e: term T at index 6 in Voter’s log

141. Log index 1 2 3 4 5 6 7 8 9 10 11 12 L
Leader(U)
Voter T
Leader (L) T L
Last term in Leader(U)’s log is L, and L > the last term in Voter’s log
Do we know whether Leader(L) has the last entry in Leader(U)’s log?
Yes, it must, since updates only flow from leaders to followers
Entry e: term T at index 6 in Voter’s and Leader(L)’s log

142. Log index 1 2 3 4 5 6 7 8 9 10 11 12 T L
Leader(U)
Voter T
Leader (L) T L
Last term in Leader(U)’s log is L, and L > the last term in Voter’s log
Finally, apply the Log Matching property to infer that Leader(U) has e.
Since Leader(L) and Leader(U) both have the last entry in Leader(U)’s log
at the same index and term, all preceding entries are identical.
And since Leader(L) has e and (T < L), Leader(U) must have e.
Entry e: term T at index 6 in Voter’s and Leader(L)’s log

143. Log index 1 2 3 4 5 6 7 8 9 10 11 12 T L
Leader(U)
Voter T
Leader (L) T L
Last term in Leader(U)’s log is L, and L > the last term in Voter’s log
Finally, apply the Log Matching property to infer that Leader(U) has e.
Since Leader(L) and Leader(U) both have the last entry in Leader(U)’s log
at the same index and term, all preceding entries are identical.
And since Leader(L) has e and (T < L), Leader(U) must have e. ✓
Entry e: term T at index 6 in Voter’s, Leader(L)’s, and Leader(U)’s log

144. Proof by contradiction
Leader in U’s log must have been as up to date as the voter’s log
If leader’s log was the same as the voter’s, then what must be true?
Leader’s log must have contained committed entry e
This contradicts our assumption, so leader’s log must ≠ voter’s
What if leader’s log has same last term as voter’s?
Then leader’s log must have been longer than the voter’s
If longer with same last term, then there must be an entry from last term in common
Log Matching Property: entries with same index and term  all preceding are same
Thus, the leader’s log must contain entry e
This contradicts our assumption, so leader’s last log term > voter’s last log term
What if leader’s last log term is later than voter’s last log term?
Last term in leader’s log is L, and L > T since voter’s log contains entry e
Leader in L must have had e, since leader in U is first to not have it
By Log Matching Property, leader in U’s log must also contain e

145. Proof by contradiction
Assume that node missing a committed entry e is elected
Entry e was committed in term T
Leader missing committed entry e is elected in term U
U is earliest term after T whose leader is missing e
Majorities are needed to commit and elect
At least one node accepted e in term T and voted for leader in U
Call this node “the voter”
Voter obtained e before voting for leader in U
Leader in U’s log must have been as up to date as the voter’s log
Nodes missing committed entries cannot be elected
Committed entries are always preserved across terms/elections

146. One final issue Cluster membership changes
Membership is part of the cluster’s configuration
What is the easy way to handle this?
Take the cluster down
Change configuration on all machines
Restart the cluster
Why isn’t this ideal?
Would rather not take the availability hit
Manually editing configuration can be dangerous

147. Should never have two leaders
Problem: cannot guarantee when change happens
What makes this period dangerous?
Disjoint majorities
(a)
old
new
(b)
old
new
(c)
old
new
(d)
new
(e)
new
Time

148. Should never have two leaders
Problem: cannot guarantee when change happens
How can you have disjoint majorities?
Old (a,b)
New (c,d,e)
(a)
old
new
(b)
old
new
(c)
old
new
(d)
new
(e)
new
Time

149. Raft’s approach Change configuration in two phases
First, “joint consensus”
Second, transition to new configuration
Joint consensus rules
Log entries replicated to all servers in both configs
Any server, from either config, may serve as leader
Agreement requires majorities from both configs

150. Configuration change
Leaders initiate configuration change from old to new
They send out a special log entry, Cold,new, describing both
As soon as a server logs Cold,new, it plays by joint consensus rules
For Cold,new to commit, it must reach a majority of Cold and Cnew
If the leader fails, the new leader could be in Cold or Cold,new
Important thing is that membership of Cnew cannot act unilaterally
In other words, majority of Cold still required for commit, elections
Eventually, Cold,new will commit (we may have to try several times)
If Cold,new commits, what must be true of future leaders?
By Leader Completeness, they must be running Cold,new config

151. Configuration change Once, Cold,new commits, we’re in good shape
Leader can now try to commit Cnew
Leader logs and uses Cnew immediately
i.e., only requires a majority of Cnew to commit entries
Nasty corner case: what if the leader isn’t in Cnew?
(i.e., managing a cluster of which they are not a member)
It’s fine! Majorities don’t have to include the leader
The leader continues to count votes and push updates
When and how should the leader exit?
Wait until Cnew commits, then shut down
Cnew will elect a new leader

152. Midterm exam Logistics Topics Format
Wednesday, February 22nd during lecture
In-class, closed-note
Ali will proctor
Topics
Concurrency and virtual memory
Single-host: THE, Multics, and UNIX
Distributed: RPC/MAUI, Grapevine, VAXClusters, and Raft
Anything from reading/lecture is fair game
Format
Multi-part, short answer