1. Consistency and Replication
Chapter 6
OS2 –2nd Semester, Rasool Jalili

2. OS2 –2nd Semester, 94-95 Rasool Jalili
Replication
Important issue in DSs to enhance reliability and improve performance.
The major problem is how to make replicas consistent.
Much attention on consistency have been paid on consistency models of DSM systems by parallel computer designers.
Issue:
Client-Centric consistency
Implementation of replication and distribution of updates.
Models of consistency: Strong? How much?
OS2 –2nd Semester, Rasool Jalili

3. OS2 –2nd Semester, 94-95 Rasool Jalili
Object Replication (1)
Organization of a distributed remote object shared by two different clients.
OS2 –2nd Semester, Rasool Jalili

4. OS2 –2nd Semester, 94-95 Rasool Jalili
The first problem is how to handle concurrent accesses to the remote object by two clients.
1- The object itself handles concurrent invocations
2- The server is responsible for concurrency control and the object is unprotected.
The two scenarios in the next slide.
OS2 –2nd Semester, Rasool Jalili

5. OS2 –2nd Semester, 94-95 Rasool Jalili
Object Replication (2)
A remote object capable of handling concurrent invocations on its own.
A remote object for which an object adapter is required to handle concurrent invocations
OS2 –2nd Semester, Rasool Jalili

6. OS2 –2nd Semester, 94-95 Rasool Jalili
Object Replication (3)
A distributed system for replication-aware distributed objects.
A distributed system responsible for replica management
OS2 –2nd Semester, Rasool Jalili

7. OS2 –2nd Semester, 94-95 Rasool Jalili
Replication
Replication and Scaling Technology?
Consistency is against scalability!
Consistency Model is a contract between processes and the data store.
OS2 –2nd Semester, Rasool Jalili

8. Data-Centric Consistency Models
The general organization of a logical data store, physically distributed and replicated across multiple processes.
OS2 –2nd Semester, Rasool Jalili

9. OS2 –2nd Semester, 94-95 Rasool Jalili
Strict Consistency
Any read on a data item X returns a value corresponding to the result of the most recent write on X
Behavior of two processes, operating on the same data item.
(a) A strictly consistent store.
(b) A store that is not strictly consistent.
OS2 –2nd Semester, Rasool Jalili

10. Linearizability and Sequential Consistency (1)
A sequentially consistent data store.
A data store that is not sequentially consistent.
OS2 –2nd Semester, Rasool Jalili

11. Linearizability and Sequential Consistency (2)
Process P1
Process P2
Process P3
x = 1;
print ( y, z);
y = 1;
print (x, z);
z = 1;
print (x, y);
Three concurrently executing processes.
OS2 –2nd Semester, Rasool Jalili

12. Linearizability and Sequential Consistency (3)
Four valid execution sequences for the processes of the previous slide. The vertical axis is time.
x = 1;
print ((y, z);
y = 1;
print (x, z);
z = 1;
print (x, y);
Prints:
Signature:
(a)
print (x,z);
print(y, z);
Prints:
Signature:
(b)
print (y, z);
Prints:
Signature:
110101
(c)
Prints:
111111
(d)
or are impossible.
If a DSM do not produce some valid patterns!!
OS2 –2nd Semester, Rasool Jalili

13. OS2 –2nd Semester, 94-95 Rasool Jalili
Casual Consistency (1)
Necessary condition: Writes that are potentially casually related must be seen by all processes in the same order. Concurrent writes may be seen in a different order on different machines.
OS2 –2nd Semester, Rasool Jalili

14. OS2 –2nd Semester, 94-95 Rasool Jalili
Casual Consistency (2)
This sequence is allowed with a casually-consistent store, but not with sequentially or strictly consistent store.
OS2 –2nd Semester, Rasool Jalili

15. OS2 –2nd Semester, 94-95 Rasool Jalili
Casual Consistency (3)
A violation of a casually-consistent store.
A correct sequence of events in a casually-consistent store.
OS2 –2nd Semester, Rasool Jalili

16. OS2 –2nd Semester, 94-95 Rasool Jalili
FIFO Consistency (1)
Necessary Condition: Writes done by a single process are seen by all other processes in the order in which they were issued, but writes from different processes may be seen in a different order by different processes.
OS2 –2nd Semester, Rasool Jalili

17. OS2 –2nd Semester, 94-95 Rasool Jalili
FIFO Consistency (2)
A valid sequence of events of FIFO consistency
OS2 –2nd Semester, Rasool Jalili

18. OS2 –2nd Semester, 94-95 Rasool Jalili
FIFO Consistency (4)
Process P1
Process P2
x = 1;
if (y == 0) kill (P2);
y = 1;
if (x == 0) kill (P1);
Two concurrent processes.
Compare Sequential and FIFO consistency.
OS2 –2nd Semester, Rasool Jalili

19. OS2 –2nd Semester, 94-95 Rasool Jalili
Weak Consistency (1)
Properties:
Accesses to synchronization variables associated with a data store are sequentially consistent
No operation on a synchronization variable is allowed to be performed until all previous writes have been completed everywhere
No read or write operation on data items are allowed to be performed until all previous operations to synchronization variables have been performed.
OS2 –2nd Semester, Rasool Jalili

20. OS2 –2nd Semester, 94-95 Rasool Jalili
Weak Consistency (2)
int a, b, c, d, e, x, y; /* variables */ int *p, *q; /* pointers */ int f( int *p, int *q); /* function prototype */
a = x * x; /* a stored in register */ b = y * y; /* b as well */ c = a*a*a + b*b + a * b; /* used later */ d = a * a * c; /* used later */ p = &a; /* p gets address of a */ q = &b /* q gets address of b */ e = f(p, q) /* function call */
A program fragment in which some variables may be kept in registers; a and b in registers, but when calling f, they should be synchronized.
OS2 –2nd Semester, Rasool Jalili

21. OS2 –2nd Semester, 94-95 Rasool Jalili
Weak Consistency (3)
A valid sequence of events for weak consistency.
An invalid sequence for weak consistency.
OS2 –2nd Semester, Rasool Jalili

22. OS2 –2nd Semester, 94-95 Rasool Jalili
Release Consistency
Problem with Weak Consistency: Access to a synch variable might be because of finishing or starting of write operation on shared data items.  in all cases, it should propagates all recent writes around the system.
A more efficient way is to separate entering and finishing of a critical region.
Acquire and Release.
The programmer is responsible to put them in its program
Similar to lock and unlock.
OS2 –2nd Semester, Rasool Jalili

23. Release Consistency (1)
A valid event sequence for release consistency.
OS2 –2nd Semester, Rasool Jalili

24. Release Consistency (2)
Rules:
Before a read or write operation on shared data is performed, all previous acquires done by the process must have completed successfully.
Before a release is allowed to be performed, all previous reads and writes by the process must have completed
Accesses to synchronization variables are FIFO consistent (sequential consistency is not required).
OS2 –2nd Semester, Rasool Jalili

25. OS2 –2nd Semester, 94-95 Rasool Jalili
Lazy or Eager
Eager release consistency pushes the update on Release
Lazy release consistency does not do anything on release. The process doing Acquire after that, updates its copies.
OS2 –2nd Semester, Rasool Jalili

26. OS2 –2nd Semester, 94-95 Rasool Jalili
Entry Consistency (1)
A valid event sequence for entry consistency.
OS2 –2nd Semester, Rasool Jalili

27. OS2 –2nd Semester, 94-95 Rasool Jalili
Entry Consistency (2)
Conditions:
An acquire access of a synchronization variable is not allowed to perform with respect to a process until all updates to the guarded shared data have been performed with respect to that process.
Before an exclusive mode access to a synchronization variable by a process is allowed to perform with respect to that process, no other process may hold the synchronization variable, not even in nonexclusive mode.
After an exclusive mode access to a synchronization variable has been performed, any other process's next nonexclusive mode access to that synchronization variable may not be performed until it has performed with respect to that variable's owner.
OS2 –2nd Semester, Rasool Jalili

28. Summary of Consistency Models
Consistency models not using synchronization operations.
Models with synchronization operations.
Consistency
Description
Strict
Absolute time ordering of all shared accesses matters.
Linearizability
All processes must see all shared accesses in the same order. Accesses are furthermore ordered according to a (nonunique) global timestamp
Sequential
All processes see all shared accesses in the same order. Accesses are not ordered in time
Causal
All processes see causally-related shared accesses in the same order.
FIFO
All processes see writes from each other in the order they were used. Writes from different processes may not always be seen in that order
(a)
Weak
Shared data can be counted on to be consistent only after a synchronization is done
Release
Shared data are made consistent when a critical region is exited
Entry
Shared data pertaining to a critical region are made consistent when a critical region is entered.
(b)
OS2 –2nd Semester, Rasool Jalili

29. Client-Centric Consistency
Previous discussion was concentrated on data items.
Assuming read operations are much more than concurrent writes:: so we need even weaker consistency
The level of concurrency and the level of requirement of being so consistent varies. The question is that: How fast updates should be made available to only-reading processes?
Example: DNS, Web pages? These are able to tolerate a high degree of inconsistency.
The base idea: If no updates take place for a long time, all replicas will gradually become consistent  Eventual consistency.
Generally a small set of replicas do write and so write-write conflicts can be handled.
OS2 –2nd Semester, Rasool Jalili

30. OS2 –2nd Semester, 94-95 Rasool Jalili
Eventual Consistency
The problem of having mobile clients moving around and connect
to different replicas!  Client-Centric Consistency.
Guarantees consistency for a single client
OS2 –2nd Semester, Rasool Jalili

31. Client-Centric Consistency
Assume each client access its nearest replica and does its read and write
Updates are eventually propagated to the other copies.
We can simplify the issue by considering an owner for each data item who has the authority on the data item.
xi [t] means the version of x at Li at time t.
WS(xi [t]) denotes the set of writes on x.
4 models of Client-Centric consistency as follows …
OS2 –2nd Semester, Rasool Jalili

32. OS2 –2nd Semester, 94-95 Rasool Jalili
Monotonic Reads (1)
A data store is said to provide Monotonic Read consistency if:
When a process reads a data item x, successive read operations of the same process on x always returns the same value or a more recent value.
Example: Our mailbox, even when it is stored in a replicated and fully distributed around the world.
OS2 –2nd Semester, Rasool Jalili

33. OS2 –2nd Semester, 94-95 Rasool Jalili
Monotonic Reads
The read operations performed by a single process P at two different local copies of the same data store.
A monotonic-read consistent data store
A data store that does not provide monotonic reads.
OS2 –2nd Semester, Rasool Jalili

34. OS2 –2nd Semester, 94-95 Rasool Jalili
Monotonic Writes(1)
A write operation by a process on a data item x is completed before any successive write on x by the same process, no matter where the operation was initiated.
Similar to the concept of FIFO consistency on data-centric approaches, but here we are restricted to a single process, not all concurrent processes.
Ex: software library update
OS2 –2nd Semester, Rasool Jalili

35. OS2 –2nd Semester, 94-95 Rasool Jalili
Monotonic Writes
The write operations performed by a single process P at two different local copies of the same data store
A monotonic-write consistent data store.
A data store that does not provide monotonic-write consistency.
OS2 –2nd Semester, Rasool Jalili

36. OS2 –2nd Semester, 94-95 Rasool Jalili
Read Your Writes
The effect of a write operation by a process on data item x will always be seen by a successive read op on x by the same process.
Ex: Updating web page and seeing that !
OS2 –2nd Semester, Rasool Jalili

37. OS2 –2nd Semester, 94-95 Rasool Jalili
Read Your Writes
A data store that provides read-your-writes consistency.
A data store that does not.
OS2 –2nd Semester, Rasool Jalili

38. OS2 –2nd Semester, 94-95 Rasool Jalili
Writes Follow Reads
A write operation by a process on a data item x following a previous read on x by the same process, is guaranteed to take place on the same or a more recent value of x that was read.
Ex: Posting of a reaction on an article in newsgroups after seeing the original article.
OS2 –2nd Semester, Rasool Jalili

39. OS2 –2nd Semester, 94-95 Rasool Jalili
Writes Follow Reads
A writes-follow-reads consistent data store
A data store that does not provide writes-follow-reads consistency
OS2 –2nd Semester, Rasool Jalili

40. OS2 –2nd Semester, 94-95 Rasool Jalili
Replicas
Permanent replicas; number of them is limited. Mirroring is a form of permanent replicas
Server initiated replicas
Client-initiated replicas.
OS2 –2nd Semester, Rasool Jalili

41. OS2 –2nd Semester, 94-95 Rasool Jalili
Replica Placement
The logical organization of different kinds of copies of a data store into three concentric rings.
OS2 –2nd Semester, Rasool Jalili

42. Server initiated replicas
Copies of data store exist to enhance performance; created at the initiative of the owner of the data store due to a burst traffic :: Push cache.
Where and when replicas should be created and deleted?
An approach to dynamically replicate files , designed for web-hosting :: updates are rare compare to read ops.
OS2 –2nd Semester, Rasool Jalili

43. OS2 –2nd Semester, 94-95 Rasool Jalili
Two issues:
Replication for reduction of load on a server
Migration or replication of specific files to the server closer to the clients who issue many requests to those files.
Each server keeps track of access counts per file and the location of the access. Each server can determine which server is closest to a given client C.
When the number of req for a file at server S drops below a deletion threshold, it can be removed from S.
Two thresholds: del (S, F), and rep (S, F)
OS2 –2nd Semester, Rasool Jalili

44. Server-Initiated Replicas
Counting access requests from different clients.
OS2 –2nd Semester, Rasool Jalili

45. Client-initiated replicas
Created at the initiative of clients
Known as (client) caches.
Cache hit
Period of keeping data on caches
OS2 –2nd Semester, Rasool Jalili

46. OS2 –2nd Semester, 94-95 Rasool Jalili
Update Propagation
Update normally initiated at the client side and subsequently forwarded to one of the copies.  update should be propagated toward other copies whilst maintaining consistency.
Issues:
What should be propagated: State or operations?
Only a notification of an update :: invalidation
Transfer data from one copy to another.
Propagate the update op to other copies.
Push updates or pull them?
OS2 –2nd Semester, Rasool Jalili

47. Pull versus Push Protocols
Issue
Push-based
Pull-based
State of server
List of client replicas and caches
None
Messages sent
Update (and possibly fetch update later)
Poll and update
Response time at client
Immediate (or fetch-update time)
Fetch-update time
A comparison between push-based and pull-based protocols in the case of multiple client (each having their own cache), single server systems.
OS2 –2nd Semester, Rasool Jalili

48. Consistency Protocols
So far, we concentrated on consistency models.
A consistency protocol describes an implementation of a specific consistency model.
Primary-Based protocols
Client-server: all RW ops from a server (distributed or centralized)
Primary-Backup protocols (Write operations toward the primary)
OS2 –2nd Semester, Rasool Jalili

49. Remote-Write Protocols (1)
Primary-based remote-write protocol with a fixed server to which all read and write operations are forwarded.
OS2 –2nd Semester, Rasool Jalili

50. Remote-Write Protocols (2)
The principle of primary-backup protocol.
OS2 –2nd Semester, Rasool Jalili

51. Local-Write Protocols
Two kinds
Only a single copy of each item exists: No replicas
For each operation, a copy the data item is transferred to the process and the operation is done.
Overhead of maintaining who is where!
A variant is a primary-backup protocol in which the primary migrates between processes.
OS2 –2nd Semester, Rasool Jalili

52. Local-Write Protocols (1)
Primary-based local-write protocol in which a single copy is migrated between processes.
OS2 –2nd Semester, Rasool Jalili

53. Local-Write Protocols (2)
Primary-backup protocol in which the primary migrates to the process wanting to perform an update.
OS2 –2nd Semester, Rasool Jalili

54. Replicated Write Protocols
Write operations can be carried out at multiple replicas.
Active replication: an operation is forwarded to all replicas
Majority voting
OS2 –2nd Semester, Rasool Jalili

55. OS2 –2nd Semester, 94-95 Rasool Jalili
Active Replication
Each replica has an associated process that carries out update operations
Operations need to be done in the same order everywhere  totally-ordered multicast is needed.
A sequencer is needed somehow to send requests first to it and then to all replicas.
The other problem is replicated invocation.
OS2 –2nd Semester, Rasool Jalili

56. OS2 –2nd Semester, 94-95 Rasool Jalili
Active Replication (1)
The problem of replicated invocations.
OS2 –2nd Semester, Rasool Jalili

57. OS2 –2nd Semester, 94-95 Rasool Jalili
Active Replication (2)
Forwarding an invocation request from a replicated object.
Returning a reply to a replicated object.
OS2 –2nd Semester, Rasool Jalili

58. Quorum-Based Protocols
Three examples of the voting algorithm:
A correct choice of read and write set
A choice that may lead to write-write conflicts
A correct choice, known as ROWA (read one, write all)
OS2 –2nd Semester, Rasool Jalili

59. OS2 –2nd Semester, 94-95 Rasool Jalili
Orca
OBJECT IMPLEMENTATION stack; top: integer; # variable indicating the top stack: ARRAY[integer 0..N-1] OF integer # storage for the stack
OPERATION push (item: integer) # function returning nothing BEGIN GUARD top < N DO stack [top] := item; # push item onto the stack top := top + 1; # increment the stack pointer OD; END;
OPERATION pop():integer; # function returning an integer BEGIN GUARD top > 0 DO # suspend if the stack is empty top := top – 1; # decrement the stack pointer RETURN stack [top]; # return the top item OD; END; BEGIN top := 0; # initialization END;
A simplified stack object in Orca, with internal data and two operations.
OS2 –2nd Semester, Rasool Jalili

60. OS2 –2nd Semester, 94-95 Rasool Jalili
???
End of Chapter 6
OS2 –2nd Semester, Rasool Jalili

61. Management of Shared Objects in Orca
Four cases of a process P performing an operation on an object O in Orca.
OS2 –2nd Semester, Rasool Jalili

62. Casually-Consistent Lazy Replication
The general organization of a distributed data store. Clients are assumed to also handle consistency-related communication.
OS2 –2nd Semester, Rasool Jalili

63. Processing Read Operations
Performing a read operation at a local copy.
OS2 –2nd Semester, Rasool Jalili

64. Processing Write Operations
Performing a write operation at a local copy.
OS2 –2nd Semester, Rasool Jalili

65. OS2 –2nd Semester, 94-95 Rasool Jalili
FIFO Consistency (3)
x = 1;
print (y, z);
y = 1;
print(x, z);
z = 1;
print (x, y);
Prints: 00
(a)
print ( y, z);
Prints: 10
(b)
print (x, z);
Prints: 01
(c)
Statement execution as seen by the three processes from the previous slide. The statements in bold are the ones that generate the output shown.
OS2 –2nd Semester, Rasool Jalili