1. Replication Improves reliability Improves availability
(What good is a reliable system if it is not available?)
Replication must be transparent and create the illusion of a single copy.

2. Updating replicated data
shared
Separate replicas F F’
F’’
Alice
Bob
Alice
Bob
Update and consistency are primary issues.

3. Passive replication 4 L=3 1 3 L=3 primary 2 clients backup
Each client communicates with one
replica called the primary server
Each client maintains a variable L
(leader) that specifies the replica to
which it will send requests. Requests
are queued at the primary server.
Backup servers ignore client requests. 4
L=3 1 3
L=3
primary 2
clients
backup

4. Primary-backup protocol
Receive. Receive the request from the client and update the state if appropriate.
Broadcast. Broadcast an update of the state to all other replicas.
Reply. Send a response to the client.
client
req
reply
primary
update
backup

5. Primary-backup protocol
If the client fails to get a response due
to the crash of the primary, then the
request is retransmitted until a
backup is promoted as the primary.
{The switch should ideally be
Instantaneous, but practically
it is not so}
Failover time is the duration when
there is no primary server.
New primary
elected
client
req
reply
primary
update
heartbeat ?
backup
election

6. Active replication
Each server receives client requests, and broadcasts them to the other servers. They collectively implement a fault-tolerant state machine. In presence of crash, all the correct processes reach the same next state.
input
State
Next state

7. Fault-tolerant state machine
This formalism is based on a survey by Fred Schneider.
The clients must receive correct response even if up to
m replica servers fail (either fail-stop or byzantine).
For fail-stop, ≥ (m+1) replicas are needed. If a client queries
the replicas, the first one that responds gives a correct value.
For byzantine failure ≥ (2m+1) replicas are needed. m bad
responses can be voted out by the (m+1) good responses.
But the states of the good processes must be correctly
updated (byzantine consensus is needed). This is possible
when n> 3m.
Fault intolerant
Fault tolerant

8. Replica coordination
Agreement. Every correct replica receives all the requests.
Order. Every correct replica receives the requests in the same order.
Agreement part is solved by atomic multicast.
Order part is solved by total order multicast.
The order part solves the consensus problem
where servers will agree about the next update.
It requires a synchronous model. Why?
server
client

9. Agreement client With fail-stop processors, the agreement part
is solved by reliable atomic multicast.
To deal with byzantine failures, an interactive
consistency protocol needs to be implemented.
Thus, with an oral message protocol, n ≥ 3m+1
processors will be required.
server

10. Order Let timestamps determine the message order. client
A request is stable at a server, when the
it does not expect to receive any other
client request with a lower timestamp.
Assume three clients are trying to send an update, the channels are FIFO, and their timestamps are 20, 30, 42. Only if global FIFO behavior can be guaranteed, each server will first update its copy with the value that has the timestamp 20. 30 20
server 42

11. Order But some clients may not have any update.
How long should the server wait?
Require clients to send null messages (as heartbeat signals) with some timestamp ts. A message (null, 35) means that the client will not send any update till ts=35. These can be part of periodic heartbeat messages.
An alternative is to use virtual time, where processes are able to undo actions.
client 30
null 35
server 42

12. What is replica consistency?
clients
Consistency models define a contract between the data manager and
the clients regarding the responses to read and write operations.

13. Replica Consistency Data Centric
Client communicates with the same replica
Client centric
Client communicates with different replica at different times. This may be the case with mobile clients.

14. Data-centric Consistency Models
1. Strict consistency
2. Linearizability
3. Sequential consistency
Causal consistency
Eventual consistency (as in DNS)
Weak consistency
There are many other models

15. Strict consistency
Strict consistency corresponds to true replication transparency. If one of the processes executes x:= 5 at real time t and this is the latest write operation, then at a real time t’ > t, every process trying to read x will receive the value 5. Too strict! Why?
W(x:=5) p1
R(x=5) p2 t t’
{Assume the read or write operations are non-blocking}

16. Sequential consistency
Some interleaving of the local temporal order of events at the different replicas is a consistent trace.
W(x:=100)
W(x:=99]
R(x=100)
R(x=99)

17. Sequential consistency
Is sequential consistency satisfied here? Initially x = y = 0
W(x:=10)
W(x:=8]
R(x:=10)
W(x=20)
R(x=20)
R(x=10)

18. Causal consistency
All writes that are causally related must be seen by every process in the same order.
W(x:=10)
W(x:=20)
R(x=10)
R(x=20)
R(x=10)
R(x=20)

19. Linearizability
Linearizability is a correctness criterion for concurrent object (Herlihy & Wing ACM TOPLAS 1990). It provides the illusion that each operation on the object takes effect in zero time, and the results are “equivalent to” some legal sequential computation.

20. Linearizability W (x:=0) R (x=1) W (x:=0) R(x=1) W (x:=1)
A trace is in a read-write system is consistent, when every read returns the latest value written into the shared variable preceding that read operation. A trace is linearizable, when (1) it is consistent, and (2) the temporal ordering among the reads and writes is respected (may be based on real time or logical time).
W (x:=0)
R (x=1)
W (x:=0)
ts=10
ts=21
ts=27
R(x=1)
W (x:=1)
(Initially x=y=0)
ts=19
ts=38
Linearizability is stronger than sequential consistency,
i.e. every linearizable object is also sequentially consistent.
Is it a linearizable trace?

21. Exercise
What consistency model is satisfied by the above?

22. Implementing consistency models
Why are there so many consistency models?
Each model has a use in some type of application.
The cost of implementation (as measured by message complexity) decreases as the models become “weaker”.

23. Implementing linearizability
W (x:=20)
Read x
W(x:=10)
Read x
Needs total order multicast of all reads and writes

24. Implementing linearizability
The total order multicast forces every process to accept and handle all reads and writes in the same temporal order.
The peers update their copies in response to a write, but only send acknowledgments for reads. After all updates and acknowledgments are received, the local copy is returned to the client.

25. Implementing sequential consistency
Use total order broadcast all writes only,
but for reads, immediately return local copies.

26. Eventual consistency
Only guarantees that all replicas eventually receive all updates, regardless of the order.
The system does not provide replication transparency but large scale systems like Bayou allows this. Conflicting updates are resolved using occasional anti-entropy sessions that incrementally steer the system towards a consistent configuration.

27. Implementing eventual consistency
Updates are propagated via epidemic protocols. Server S1 randomly picks a neighboring server S2, and passes on the update.
Case 1. S2 did not receive the update before. In this case, S2 accepts the update, and both S1 and S2 continue the process.
Case 2. S2 already received the update from someone else. In that case, S1 loses interest in sending updates to S2 (reduces the probability of transmission to S2 to 1/p (p is a tunable parameter)
There is always a finite probability that some servers do not receive all updates. The number can be controlled by changing p.

28. Anti-entropy sessions
These sessions minimize the “degree of chaos” in the states of the replicas.
During such a session, server S1 will “pull” the update from S2, and server S3 can “push” the update to S4 30
Timestamp of update 30 S4 26 32 30 S3 S2 S1 24

29. Exercise Let x, y be two shared variables Process P Process Q
{initially x=0} {initially y=0}
x :=1; y:=1;
if y=0  x:=2 fi; if x=0  y:=2 fi;
Print x Print y
If sequential consistency is preserved, then what are the possible values of the printouts? List all of them.

30. Client centric consistency model
Relevant in the cloud storage environment

31. Client-centric consistency model
Read-after-read
If read from A is followed by read from B then the second read should return a data that is as least as old the previous read. A B
Iowa City
San Francisco
{All the s read at location A must be marked as read in location B}

32. Client-centric consistency model
Read-after-write (a.k.a read your writes} Consider a large distributed store containing a massive collection of music. Clients set up password-protected accounts for purchasing and downloading music. Alice changed her password in Iowa City, traveled to a Minneapolis, and tried to access the collection by logging into the account using her new password, then she must be able to do so.

33. Client-centric consistency model
Write-after-read (a.k.a. write-follows-read)
Each write operation following a read should take effect on the previously read copy, or a more recent version of it.
Use your bank card to pay $500
in a store in Denver
Alice then went to
San Francisco
Balance = $1500
Balance:=balance-$500
Write should
take effect on
Balance = $1500
Balance in Iowa city bank after
your paycheck was credited
But the payment did not go through!
Write-after-read consistency was violated

34. Client-centric consistency model
Write-after-write (a.k.a. monotonic write)
When write at S is followed by write at a different server S’, the updates at S must be visible before the data is updates at S’. S’ S
San Francisco
Dallas
Alice then went to
San Francisco
Only ½ of the updates
at S are visible here
Alice gave a raise to each
of her 100 employees
Alice then decided to give a 10% bonus on the new salary
to every employee
½ of the employees
will receive a lower bonus
Write-after-read consistency was violated

35. Implementing client-centric consistency
Read set RS, write set WS
Before an operation at a different server is initiated, the appropriate
RS or WS is fetched from another server.

36. Quorum-based protocols
A quorum system engages only a designated minimum number of the replicas for every read or write operation – this number is called the read or write quorum. When the quorum is not met, the operation (read or write) is not performed.
Improves reliability, availability, and reduces the load on individual servers

37. Quorum-based protocols
Use 2-phase locking to update all the copies
(value, version #)
Write quorum
Thomas rule
To write, update > N/2 of them, and tag it with new version number.
To read, access > N/2 replicas, and access the value from the copy with the largest version number. Otherwise abandon the read
Read quorum

38. Rationale N = no of replicas. Ver 3 Ver 2
If different replicas store different version numbers for an item,
the state associated with a larger version number is more recent
than the state associated with a smaller version number.
We require that R+W > N, i.e., read quorums always intersect with
write quorums. This will ensure that read results always reflect the
result of the most recent write (because the read quorum will include
at least one replica from the most recent write). Ch

39. How it works N = no of replicas.
1. Send a write request containing the state and new version number to all the replicas and waits to receive acknowledgements from a write quorum. At that point the write operation is complete. The replicas are locked when the write is in progress.
2. Send a read request for the version number to all the replicas, and wait for replies from a read quorum.

40. Quorum-based protocols
After a partition, only the larger segment runs the protocol. The smaller segment contains stale data, until the network is repaired.
Ver.1
Ver.0

41. Quorum-based protocols: Generalized version
Asymmetric quorum:
W + R > N
W > N/2
No two writes overlap
No read overlaps with a write.
R = read quorum W = write quorum
This generalization is due to Gifford.

42. Brewer’s CAP Theorem
In an invited talk in the PODC 2000 conference, Eric Brewer presented a conjecture: that it is impossible for a web service to provide all three of the following guarantees: consistency (C), Availability (A), and partition-tolerance (P). Individually each of these guarantees is highly desirable, however, a web-service can meet at most two of the three guarantees.

43. A High-level View of CAP Theorem
For consistency and availability, propagate the update from the left to the right partition. But how can you do it? So sacrifice partition tolerance
If you prefer partition tolerance and availability, the sacrifice consistency.
Or if you prefer both partition-tolerance and consistency, then sacrifice
availability – users in the right partition will wait indefinitely until the partition is restored and the update is propagated to the right.

44. Amazon Dynamo
Amazon’s Dynamo is a highly scalable and highly available key-value
storage designed to support the implementation of its various e-commerce
services.
Dynamo serves tens of millions of customers at peak times using
thousands of servers located across numerous data centers around the world
Dynamo uses distributed hash tables (DHT) to map its servers in a circular
key space using consistent hashing commonly used in many P2P networks. .

45. Amazon Dynamo
(a) The key K is stored in the server SG and is also replicated in
servers like SH and SA (b) The evolution of multi-version data as
reflected by the values of the vector clocks.

46. Amazon Dynamo
Multiple versions of data are however rare. In a 24-hour profile of the
shopping cart service, 99.94% of requests saw exactly one version, and
% of requests saw 2 versions.
Write: the coordinator generates the vector clock for the new version, and sends it to the top T reachable nodes. If at least W nodes respond, then the write is considered successful.
Read: the coordinator sends a request for all existing version to the T top reachable servers. If it receives R responses then the read is considered successful
Uses sloppy quorum -- T, R, and W are limited to the first set of reachable non-faulty servers in the consistent hashing ring -- this speeds up the read and the write operations by avoiding the slow servers. Typically, (T,R,W) = (3,2,2)

47. Amazon Dynamo Maintains the spirit of “always write”
When a designated server S is inaccessible or down, the write is directed to a different server S’ with a hint that this update is meant for S . S’ later delivers the update to S when it recovers (Hinted handoff).
Service level agreement
Quite stringent -- a typical SLA requires that 99.9% of the read and write requests execute within 300ms, otherwise customers lose interest and business suffers.