Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN Chapter 7/B Consistency And Replication Modified by Dr. Gheith Abandah

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Overview Reasons for Replication Data-centric Consistency Models Continuous Consistency Consistent Ordering of Operations Client-centric Consistency Models Eventual Consistency Monotonic Reads Monotonic Writes Read Your Writes Writes Follow Reads Replica Management Replica-Server Placement Content Replication and Placement Content Distribution Consistency Protocols Continuous Consistency Primary-Based Protocols Replicated-Write Protocols Cache-Coherence Protocols Implementing Client-Centric Consistency

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Client-centric Consistency Models We can avoid system-wide consistency by concentrating on what specific clients want, instead of what should be maintained by servers. In many cases the system Has much more reads than updates Only one process is responsible of updates

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Eventual Consistency (1) In some system, it is sufficient that after some time (eventually) all updates propagate to all replicas. Example Consider a distributed database (shown in the following slide) to which you have access through your notebook. At location A you access the database doing reads and updates. At location B you continue your work, but unless you access the same server as the one at location A, you may detect inconsistencies:

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Eventual Consistency (2) your updates at A may not have yet been propagated to B you may be reading newer entries than the ones available at A your updates at B may eventually conflict with those at A

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Eventual Consistency (3) The only thing you really want is that the entries you updated and/or read at A, are in B the way you left them in A. In that case, the database will appear to be consistent to you. Notation WS(x i [t]) is the set of write operations (at L i ) that lead to version x i of x (at time t) WS(x i [t1];x j [t2]) indicates that it is known that WS(x i [t1]) is part of WS(x j [t2]). Note: Parameter t is omitted from figures.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Monotonic Reads (1) A data store is said to provide monotonic-read consistency if the following condition holds: If a process reads the value of a data item x … any successive read operation on x by that process will always return that same value or a more recent value. Example: Checking the inbox.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Monotonic Reads (2) The read operations performed by a single process P at two different local copies of the same data store. (a) A monotonic-read consistent data store.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Monotonic Reads (3) (b) A data store that does not provide monotonic reads. The problem is that only WS(x 2 ) are performed in L2, WS(x 1 ) are not performed in L2.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Monotonic Writes (1) In a monotonic-write consistent store, the following condition holds: A write operation by a process on a data item x … is completed before any successive write operation on x by the same process. Example: Maintaining versions of replicated files

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Monotonic Writes (2) The write operations performed by a single process P at two different local copies of the same data store. (a) A monotonic-write consistent data store.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Monotonic Writes (3) (b) A data store that does not provide monotonic-write consistency. The problem is that WS(x 1 ) is not performed in L2.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Read Your Writes (1) A data store is said to provide read-your-writes consistency, if the following condition holds: The effect of a write operation by a process on data item x … will always be seen by a successive read operation on x by the same process. Example: Updating you own web pages.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Read Your Writes (2) (a) A data store that provides read-your-writes consistency.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Read Your Writes (3) (b) A data store that does not.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Writes Follow Reads (1) A data store is said to provide writes-follow-reads consistency, if the following holds: A write operation by a process … on a data item x following a previous read operation on x by the same process … is guaranteed to take place on the same or a more recent value of x that was read. Example: Store reactions to posted articles only if you have the original posting

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Writes Follow Reads (2) (a) A writes-follow-reads consistent data store.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Writes Follow Reads (3) (b) A data store that does not provide writes-follow-reads consistency.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Overview Reasons for Replication Data-centric Consistency Models Continuous Consistency Consistent Ordering of Operations Client-centric Consistency Models Eventual Consistency Monotonic Reads Monotonic Writes Read Your Writes Writes Follow Reads Replica Management Replica-Server Placement Content Replication and Placement Content Distribution Consistency Protocols Continuous Consistency Primary-Based Protocols Replicated-Write Protocols Cache-Coherence Protocols Implementing Client-Centric Consistency

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Replica-Server Placement (1) Figure out what the best K places are out of N possible locations. Select best location out of N – K for which the average distance to clients is minimal. Then choose the next best server. (Note: The first chosen location minimizes the average distance to all clients.) Computationally expensive. Select the K-th largest autonomous system and place a server at the best-connected host. Computationally expensive. Position nodes in a d-dimensional geometric space, where distance reflects latency. Identify the K regions with highest density and place a server in every one. Computationally cheap.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Replica-Server Placement (2) Choosing a proper cell size for server placement.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Content Replication and Placement (1) Distinguish different processes A process is capable of hosting a replica of an object or data: Permanent replicas: Process/machine always having a replica Server-initiated replica: Process that can dynamically host a replica on request of another server in the data store Client-initiated replica: Process that can dynamically host a replica on request of a client (client cache)

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Content Replication and Placement (2) The logical organization of different kinds of copies of a data store into three concentric rings.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Server-Initiated Replicas Keep track of access counts per file, aggregated by considering server closest to requesting clients Number of accesses drops below threshold D → drop file Number of accesses exceeds threshold R → replicate file Number of access between D and R → migrate file

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Content Distribution (1) Possibilities for what is to be propagated: 1.Propagate only a notification of an update. 2.Transfer data from one copy to another. 3.Propagate the update operation to other copies. Note: No single approach is the best, but depends highly on available bandwidth and read- to-write ratio at replicas.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Content Distribution (2) Pushing updates: server-initiated approach, in which update is propagated regardless whether target asked for it. Pulling updates: client-initiated approach, in which client requests to be updated.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Overview Reasons for Replication Data-centric Consistency Models Continuous Consistency Consistent Ordering of Operations Client-centric Consistency Models Eventual Consistency Monotonic Reads Monotonic Writes Read Your Writes Writes Follow Reads Replica Management Replica-Server Placement Content Replication and Placement Content Distribution Consistency Protocols Continuous Consistency Primary-Based Protocols Replicated-Write Protocols Cache-Coherence Protocols Implementing Client-Centric Consistency

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Remote-Write Protocols The principle of a primary-backup protocol.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Local-Write Protocols Primary-backup protocol in which the primary migrates to the process wanting to perform an update.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Quorum-Based Protocols Three examples of the voting algorithm. (a) A correct choice of read and write set. (b) A choice that may lead to write-write conflicts. (c) A correct choice, known as ROWA (read one, write all). required: N R +N W > N and N W > N/2 Ensure that each operation is carried out in such a way that a majority vote is established: distinguish read quorum and write quorum: