1. Replicated data consistency explained through baseball
Landon Cox
April 4, 2018

2. Distributed systems guru (PARC, MSR, now at Amazon)
Lecture by Doug Terry
Distributed systems guru (PARC, MSR, now at Amazon)
Baseball fan

3. Replicated-data consistency
A set of invariants on each read operation
Which writes are guaranteed to be reflected the data that’s returned?
i.e., what write orders are guaranteed?
Consistency is an application-level concern
When consistency is too weak, applications break
Example: auction site must not tell two people they won
What are the negative consequences of too-strong consistency?
Worse performance (for reads and writes)
Worse availability (for reads and writes)

4. Assumptions for our discussion
Clients perform reads and writes
Data is replicated among a set of servers
Writes are serialized (one logical writer)
Eventually performed in the same order at all servers
Write order consistent with write-request order
Reads reflect one or more past writes

5. writes may never stop, “eventually” could be very far into the future
Consistency models
What this really means:
If writes stop,
reads eventually see effect of all writes
Strong consistency
Reader sees effect of all prior writes
Eventual consistency
Reader sees effect of some subset of prior writes
Consistent prefix
Reader sees effect of initial sequence of writes
Bounded staleness
Reader sees effect of all “old” writes
Monotonic reads
Reader sees effect of increasing subset of writes
Read my writes
Reader sees effect of all writes performed by reader
Major caveats:
writes may never stop, “eventually” could be very far into the future

6. Baseball rules Time is measured in innings During an inning
Games are normally nine innings long
Can be longer if the score is tied after nine innings
Points are called runs
During an inning
Visiting team bats until it is out three times
Home team bats until it is out three times
Goto next inning

7. Pseudo-code baseball game
Write (“visitors”, 0);
Write (“home”, 0);
for inning = 1..9
outs = 0;
while outs < 3
visiting player bats;
for each run scored
score = Read (“visitors”);
Write (“visitors”, score + 1);
home player bats;
score = Read (“home”);
Write (“home”, score + 1);
end game;
Primary game thread. Only thread that issues writes.

8. Baseball applications
Application: entity that accesses the score
Score keeper
Umpire
Radio reporter
Game recapper
Statistician

9. Baseball applications
Applications have different requirements
Some must have up-to-date score
Others are more tolerant of stale scores
Nearly all need some kind of guarantee across score accesses

10. V V V H H H Visitors’ score Home score W R W R R R R S1 S2 S3 S4 S5 S6
Reader
Writer
(also reads)
Reader

11. Example 1: score keeper score = Read (“visitors”);
Write (“visitors”, score + 1); …
score = Read (“home”);
Write (“home”, score + 1);

12. The score keeper makes sure that both scores increase monotonically.
Example 1: score keeper
The score keeper makes sure that both scores increase monotonically.
Write (“home”, 1);
Write (“visitors”, 1);
Write (“home”, 2);
Write (“home”, 3);
Write (“visitors”, 2);
Write (“home”, 4);
Write (“home”, 5);
Visitors = 2
Home = 5

13. Reads must show effect of all prior writes
Example 1: score keeper
What invariant must the store provide so the score keeper can ensure monotonically increasing scores?
Write (“home”, 1);
Write (“visitors”, 1);
Write (“home”, 2);
Write (“home”, 3);
Write (“visitors”, 2);
Write (“home”, 4);
Write (“home”, 5);
Reads must show effect of all prior writes
(strong consistency)
Visitors = 2
Home = 5

14. Example 1: score keeper Write (“home”, 1); Write (“visitors”, 1);
Under strong consistency, what possible scores can the score keeper read after this write completes?
Write (“home”, 1);
Write (“visitors”, 1);
Write (“home”, 2);
Write (“home”, 3);
Write (“visitors”, 2);
Write (“home”, 4);
Write (“home”, 5);
2-5
Visitors = 2
Home = 5

15. Example 1: score keeper Write (“home”, 1); Write (“visitors”, 1);
Under read-my-writes, what possible scores can the score keeper read after this write completes?
Write (“home”, 1);
Write (“visitors”, 1);
Write (“home”, 2);
Write (“home”, 3);
Write (“visitors”, 2);
Write (“home”, 4);
Write (“home”, 5);
2-5
Visitors = 2
Home = 5

16. V V V H H H Visitors’ score Home score W W W S1 S2 S3 S4 S5 S6 Writer
(also reads)
Writer
(also reads)
Reader

17. V V1 V2 H H1 H Visitors’ score Home score W R
Under strong consistency, who must S3 have spoken to (directly or indirectly) to satisfy the read request? W R
S2, S5
Writer
(also reads)
Writer
(also reads)
Reader

18. V V1 V2 H H1 H Visitors’ score Home score W R
When does S3 have to talk to S2 and S5? Before writes return or before read returns? W R
Implementation can be flexible. Guarantee is that exchange occurs before read completes.
Writer
(also reads)
Writer
(also reads)
Reader

19. V V1 V2 H H1 H Visitors’ score Home score W R
Under read-my-writes, who must S3 have spoken to (directly or indirectly) to satisfy read request? W R S5
Writer
(also reads)
Writer
(also reads)
Reader

20. Application only has one writer. Not true in general.
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V V1 V2 H H1 H
For score keeper, why is read-my-writes equivalent to strong consistency, even though it is “weaker”? R
Application only has one writer. Not true in general.
Reader
Writer
(also reads)
Reader

21. Example 1: score keeper Write (“home”, 1); Write (“visitors”, 1);
Common theme:
Consider application invariants
Reason about what store must ensure to support application invariants
Visitors = 2
Home = 5

22. Example 2: umpire if first half of 9th inning complete then
vScore = Read (“visitors”);
hScore = Read (“home”);
if vScore < hScore
end game;
Idea: home team doesn’t need another chance to bat if they are already ahead going into final half inning

23. Example 2: umpire if first half of 9th inning complete then
vScore = Read (“visitors”);
hScore = Read (“home”);
if vScore < hScore
end game;
Umpire invariant: Game should end if home team leads going into final half inning.

24. Reads must show effect of all prior writes
Example 2: umpire
if first half of 9th inning complete then
vScore = Read (“visitors”);
hScore = Read (“home”);
if vScore < hScore
end game;
What subset of writes must be visible to the umpire to ensure game ends appropriately?
Reads must show effect of all prior writes
(strong consistency)

25. Example 2: umpire if first half of 9th inning complete then
vScore = Read (“visitors”);
hScore = Read (“home”);
if vScore < hScore
end game;
Would read-my-writes work (as it did for the score keeper)?
No, since the umpire doesn’t issue any writes

26. Reader’s prior accesses (reads or writes) affect guarantees
Consistency models
Strong consistency
Reader sees effect of all prior writes
Eventual consistency
Reader sees effect of subset of prior writes
Consistent prefix
Reader sees effect of initial sequence of writes
Bounded staleness
Reader sees effect of all “old” writes
Monotonic reads
Reader sees effect of increasing subset of writes
Read my writes
Reader sees effect of all writes performed by reader
Reader’s prior accesses (reads or writes) affect guarantees

27. V V V H H H Visitors’ score Home score W1 W2 W3 S1 S2 S3 S4 S5 S6
Reader
Writer
Reader

28. V V H W2 W1 W3 Visitors’ score Home score R1 R1
Strong consistency: which writes could be reflected in the answer to R1?
(V, W) = (W2, W3) R1 R1
Reader
Writer
Reader

29. V V H W2 W1 W3 Visitors’ score Home score R2 R2
Strong consistency: which writes could be reflected in the answer to R2?
(V, H) = (W2, W3) R2 R2
Reader
Writer
Reader

30. V V H W2 W1 W3 Visitors’ score Home score R1 R1
Eventual consistency: which writes could be reflected in the answer to R1?
(0, 0), (0, W1), (0, W3), (W2, 0), (W2, W1), (W2, W3) R1 R1
Reader
Writer
Reader

31. V V H W2 W1 W3 Visitors’ score Home score R2 R2
Eventual consistency: which writes could be reflected in the answer to R2?
(0, 0), (0, W1), (0, W3), (W2, 0), (W2, W1), (W2, W3) R2 R2
Reader
Writer
Reader

32. which writes could be reflected in the answer to R1?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V W2 V W1 W3 H
Consistent prefix:
which writes could be reflected in the answer to R1?
(0,0), (0, W1),
(W2, W1), (W2, W3) R1 R1
Reader
Writer
Reader

33. which writes could be reflected in the answer to R2?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V W2 V W1 W3 H
Consistent prefix:
which writes could be reflected in the answer to R2?
(0,0), (0, W1),
(W2, W1), (W2, W3) R2 R2
Reader
Writer
Reader

34. Consistent prefix Provides a guarantee across variables
Similar to “snapshot isolation”
Must see version of data store that existed at some point
e.g., must see score that really occurred during the game
Assumes that reads are logically grouped
Via what mechanism are reads grouped? (DB question)
Reads (and writes) are logically grouped via transactions
Reads/writes are treated together, rather than individually
Normally provided with additional guarantees
e.g., all reads/writes in transaction succeed/fail together (atomicity)

35. which writes could be reflected in the answer to R1?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V W2 V W1 W3 H
Monotonic reads:
which writes could be reflected in the answer to R1?
(0, 0), (0, W1), (0, W3), (W2, 0), (W2, W1), (W2, W3) R1 R1
Reader
Writer
Reader

36. V V H W2 W1 W3 Visitors’ score Home score R2 R2 Monotonic reads:
which writes could be reflected in the answer to R2?
If R1=(0,0), then (0,0), (0, W1), (0, W3), (W2, 0), (W2, W1), (W2, W3)
If R1=(0, W1), then (0, W1), (0, W3), (W2, W1), (W2, W3)
If R1=(0, W3), then (0, W3), (W2, W3)
If R1=(W2,0), then (W2,0), (W2, W1), (W2, W3)
If R1=(W2, W1), then (W2, W1), (W2, W3)
If R1=(W2, W3), then (W2, W3) R2 R2
Reader
Writer
Reader

37. which writes could be reflected in the answer to R1?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V W2 V W1 W3 H
Read my writes:
which writes could be reflected in the answer to R1?
(0, 0), (0, W1), (0, W3), (W2, 0), (W2, W1), (W2, W3) R1 R1
Reader
Writer
Reader

38. which writes could be reflected in the answer to R2?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V W2 V W1 W3 H
Read my writes:
which writes could be reflected in the answer to R2?
(0, 0), (0, W1), (0, W3), (W2, 0), (W2, W1), (W2, W3) R2 R2
Reader
Writer
Reader

39. which writes could be reflected in the answer to R1?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V W2 V W1 W3 H
Read my writes:
which writes could be reflected in the answer to R1?
(W2, W3) R1 R1
Reader
Writer
Reader

40. which writes could be reflected in the answer to R1?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V W2 V W1 W3 H
Bounded staleness:
which writes could be reflected in the answer to R1?
Must see all writes that occurred more than bound time earlier than R1, could also see more recent writes R1 R1
Reader
Writer
Reader

41. which writes could be reflected in the answer to R2?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6 V W2 V W1 W3 H
Bounded staleness:
which writes could be reflected in the answer to R2?
Must see all writes that occurred more than bound time earlier than R2, could also see more recent writes R2 R2
Reader
Writer
Reader

42. Example 3: radio reporter
do {
vScore = Read (“visitors”);
hScore = Read (“home”);
report vScore, hScore;
sleep (30 minutes); }
Idea: periodically read score and broadcast it to listeners

43. Example 3: radio reporter
do {
vScore = Read (“visitors”);
hScore = Read (“home”);
report vScore, hScore;
sleep (30 minutes); }
Invariant: reporter should only report scores that actually occurred, and score should monotonically increase.

44. Example 3: radio reporter
do {
vScore = Read (“visitors”);
hScore = Read (“home”);
report vScore, hScore;
sleep (30 minutes); }
Do we need strong consistency?
No, since listeners can accept slightly old scores.

45. Example 3: radio reporter
do {
vScore = Read (“visitors”);
hScore = Read (“home”);
report vScore, hScore;
sleep (30 minutes); }
Can we get away with eventual consistency (some subset of writes is visible)?
No, eventual consistency can return scores that never occurred.

46. Example 3: radio reporter
Under eventual consistency, what possible scores could the radio reporter read after this write completes?
Write (“home”, 1);
Write (“visitors”, 1);
Write (“home”, 2);
Write (“home”, 3);
Write (“visitors”, 2);
Write (“home”, 4);
Write (“home”, 5);
0-0, 0-1, 0-2, 0-4, 0-5, 1-0, … 2-4, 2-5
Visitors = 2
Home = 5

47. V V V H H H Visitors’ score Home score W1 W2 W3 S1 S2 S3 S4 S5 S6
Reader
Score keeper
Radio reporter

48. How could reporter read a score of 1-0 (eventual consistency)?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6
V=0
V=1
V=0
H=1
H=2
H=0 W2 W1 W3
How could reporter read a score of 1-0 (eventual consistency)?
Reader
Score keeper
Radio reporter

49. Visitors’ score Home score V=0 V=1 V=0 H=1 H=2 H=0 W2 W1 W3 R R 1-0 S1
Reader
Score keeper
Radio reporter
1-0

50. Example 3: radio reporter
do {
vScore = Read (“visitors”);
hScore = Read (“home”);
report vScore, hScore;
sleep (30 minutes); }
How about only consistent prefix (some sequence of writes is visible)?
No. Would give us scores that occurred, but not monotonically increasing.

51. V V V H H H Visitors’ score Home score W1 W2 W3 S1 S2 S3 S4 S5 S6
Reader
Score keeper
Radio reporter

52. What prefix of writes is visible?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6
V=0
V=1
V=0
H=1
H=2
H=0 W2 W1 W3
What prefix of writes is visible? R1 R1 W1
Reader
Score keeper
Radio reporter
0-1

53. What prefix of writes is visible?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6
V=0
V=1
V=0
H=1
H=2
H=0 W2 W1 W3 R2
What prefix of writes is visible? R2
(initial state)
0-0
Reader
Score keeper
Radio reporter
0-1

54. Visitors’ score Home score V=0 V=1 V=0 H=1 H=2 H=0 W2 W1 W3 R R 0-0
What additional guarantee do we need? R
Also need monotonic reads (see increasing subset of writes)
0-0
Reader
Score keeper
Radio reporter
0-1

55. Monotonic reads Also called “session consistency”
Reads are grouped under a “session”
What extra state/logic is needed for monotonic reads?
System has to how read requests are temporally related
Related reads have to be assigned a sequence (i.e., a total order)
i.e., if I read Wn at time t, then I must read at least Wn at time t+1
What extra state/logic is needed for read-my-writes?
System has to know which reads and writes are temporally related
Related reads and writes have to be assigned a total order
i.e., if I write Wn at time t, then I must read at least Wn at time t+1
Note: reader could always see its own writes and still go back and forth in time …

56. Example 3: radio reporter
do {
vScore = Read (“visitors”);
hScore = Read (“home”);
report vScore, hScore;
sleep (30 minutes); }
Can we get away with bounded staleness (see all “old” writes)?
If we also have consistent prefix, and as long as bound is
< 30 minutes.

57. Example 3: radio reporter
T0 Read (“home”);
T1 Read (“visitors”);
T2 sleep (30 minutes);
T3 Read (“home”);
T4 Read (“visitors”);
T5 sleep (30 minutes);
T6 Read (“visitors”);
T7 Read (“home”);
T8 sleep (30 minutes); …
Under bounded staleness (bound = 15 minutes, no consistent prefix), what writes must these reads reflect?
Any write that occurred before
T4 – 15min

58. Example 3: radio reporter
T0 Read (“home”);
T1 Read (“visitors”);
T2 sleep (30 minutes);
T3 Read (“home”);
T4 Read (“visitors”);
T5 sleep (30 minutes);
T6 Read (“visitors”);
T7 Read (“home”);
T8 sleep (30 minutes); …
Why isn’t bounded staleness by itself sufficient?
Score must reflect writes that occurred before
T3 – (15min), could also reflect more recent writes

59. Visitors’ score Home score V=0 V=0 V=0 H=0 H=0 H=0 R1 R1
Sleep 30 minutes
Reader
Score keeper
Radio reporter
0-0

60. Visitors’ score Home score V=0 V=0 V=0 H=0 H=0 H=0 W1 W2 W3 0-0 S1 S2
Wake up in 10 minutes
Reader
Score keeper
Radio reporter
0-0

61. Under bounded staleness, what writes could a reporter see?
Visitors’ score
Home score S1 S2 S3 S4 S5 S6
V=0
V=1
V=0
H=1
H=2
H=0 W2 W1 W3
Under bounded staleness, what writes could a reporter see?
Wake up!
W1, W2, or W3
Reader
Score keeper
Radio reporter
0-0

62. Visitors’ score Home score V=0 V=1 V=0 H=1 H=2 H=0 W2 W1 W3 R2 R2 0-2
Reader
Score keeper
Radio reporter
0-1

63. Visitors’ score Home score V=0 V=1 V=0 H=1 H=2 H=0 W2 W1 W3 R2 R2 0-2
What additional guarantee do we need? R2
Also need monotonic reads (see increasing subset of writes)
0-2
Reader
Score keeper
Radio reporter
0-1

64. Example 4: game-recap writer
while not end of game {
drink beer;
check Twitter; }
go out to dinner;
vScore = Read (“visitors”);
hScore = Read (“home”);
write recap;
Idea: write about game several hours after it has ended

65. Example 4: game-recap writer
while not end of game {
drink beer;
check Twitter; }
go out to dinner;
vScore = Read (“visitors”);
hScore = Read (“home”);
write recap;
The recap must reflect all writes

66. Example 4: game-recap writer
while not end of game {
drink beer;
check Twitter; }
go out to dinner;
vScore = Read (“visitors”);
hScore = Read (“home”);
write recap;
What consistency guarantees could she use?
Strong consistency or bounded staleness w/ bound < time to eat dinner

67. Example 4: game-recap writer
while not end of game {
drink beer;
check Twitter; }
go out to dinner;
vScore = Read (“visitors”);
hScore = Read (“home”);
write recap;
What about eventual consistency?
Probably OK most of the time. Bounded to ensure you always get right output.

68. Example 5: team statistician
wait for end of game;
hScore = Read (“home”);
stat = Read (“season-runs”);
Write (“season-runs”, stat + hScore);
Season-runs increases monotonically by amount home team scored at the end of the game

69. Example 5: team statistician
wait for end of game;
hScore = Read (“home”);
stat = Read (“season-runs”);
Write (“season-runs”, stat + hScore);
What consistency is appropriate for this read?
Could use strong consistency, bounded staleness (with appropriate bound), maybe eventual consistency

70. Example 5: team statistician
wait for end of game;
hScore = Read (“home”);
stat = Read (“season-runs”);
Write (“season-runs”, stat + hScore);
What consistency is appropriate for this read?
Could use strong consistency, bounded staleness, or read-my-writes if statistician is only writer

71. Spectrum of consistency guarantees
Better performance and availability
Easier to reason about and use
(appropriate for all applications)
Eventual Consistency
Strong Consistency
How do we build something that provides the expected consistency guarantees?