1. 4. Synchronization Clock synchronization
Logical clocks and global state
Election Algorithms
Mutual exclusion
Distributed Transactions 1

2. Learning Objectives
To study how processes cooperate and synchronize with one another.
To examine the issues of synchronization based on real time, and to understand the key features of Cristian's algorithm, the Berkeley algorithm and the Network Time Protocol.
To understand the utility of logical clocks and global state, the rules for updating them and their limitations.
To be familiar with various election algorithms in DS to appoint a coordinator;
To study and understand the mutual exclusion and transactions methods in DS. 2

3. Clock Synchronization: Introduction
We need to measure time accurately:
to know the time an event occurred at a computer
to do this we need to synchronize its clock with an authoritative external clock
Algorithms for clock synchronization useful for
concurrency control based on timestamp ordering
authenticity of requests e.g. in Kerberos (in late lecture)
There is no global clock in a distributed system
Logical time is an alternative
It gives ordering of events - also useful for consistency of replicated data 3

4. Clock Synchronization: Sources of Accurate Timing Signals
International Atomic Time is based on very accurate physical clocks (drift rate 10-13).
Coordinated Universal Time (UTC) is an international standard for time keeping.
It is broadcast from radio stations on land and satellite (e.g. GPS).
Computers with receivers can synchronize their clocks with these timing signals.
Signals from land-based stations are accurate to about millisecond.
Signals from GPS are accurate to about 1 microsecond. 4

5. Clock Synchronization
Difficulties in clock synchronization in DS
Not all sites have direct access to accurate time source such as a GPS receivers due to the cost.
Sites has to synchronize their local clocks with those have more accurate time.
Synchronization needs to be done periodically due to clock drift: rate is change in the offset (difference in reading) between the clock and a normal perfect reference clock per unit of time measured (about 10-6 or 1 sec. in 11.6 days).
If a site’s clock is ahead of the reference (time server) to which it synchronizes to, it cannot be simply set back (why?).
This may have effect that a event that has happened but its time-stamp is in the future (Example: compiler and UNIX make). 5

6. Clock Synchronization
For process pi
Ci(t) – software clock
Hi(t) – Hardware clock, I.e., the time given by hardware clock
Thus: Ci(t) = H(t)+ 
Various notions of correctness for clocks have been suggested.
It is common to define a hardware clock H to be correct if its drifts fall within a known bound  > 0 (e.g. for real-time t’ and t, t’>t).
(1-  )(t’-t) ≤ H(t’) – H(t) ≤ (1+ )(t’-t) 6

7. External and Internal Clock Synchronization
External synchronization
A computer’s clock Ci is synchronized with an external authoritative time source S, so that:
|S(t) - Ci(t)| < D for i = 1, 2, … N over an interval, I of real time
The clocks Ci are accurate to within the bound D.
Internal synchronization
The clocks of a pair of computers are synchronized with one another so that:
| Ci(t) - Cj(t)| < D for i = 1, 2, … N over an interval, I of real time
The clocks Ci and Cj agree within the bound D.
Internally synchronized clocks are not necessarily externally synchronized, as they may drift collectively.
if the set of processes P is synchronized externally within a bound D, it is also internally synchronized within bound 2D. 7

8. Clock Synchronization: Cristian Algorithm
Client C requests time in Request message sent to time server S, then it receives time value t in message CUTC from S. t is the current time in S before sending back CUTC to C.
Let Ttrans be time taken for CUTC message from S to C, then C should set its time to t + Ttrans .
Ttrans can be variant. You may say: Ttrans = min+x, x ≥ 0 and min = time of message transmission if no interference of other process and no other messages, but x is unknown!
Solution (See the next slide): (1) Record total round trip time as Tround (between sending Request and receiving CUTC,, i.e., T1-T0) (2) If the time received in Request message is t, then C can estimate its time as: t + Tround/2. 8

9. Cristian's Algorithm
Getting the current time from a time server. 9

10. Clock Synchronization: Cristian Algorithm
Keep in mind: C’s time is t + Tround/2.
Let the minimum transmission time be min, and min is known.
Therefore, t’ (the time in S when C receives CUTC ) lies in the range [t+min, t+Tround - min]
t+ Tround -min - (t+min) = Tround -2min
The accuracy for this time adjustment is
(Tround/2-min)
Problem: single-server failure.
Solution: providing group synchronization time servers. 10

11. Clock Synchronization: Berkeley Algorithm
In Berkeley UNIX, one of a group sites (computers) is chosen as coordinator, (master or time server).
(1) The master (actually a time daemon in the master) periodically polls the other sites (slaves) to ask what time it is there, by telling its own time (the time in master) to these slaves.
(2) The slaves response with how far ahead or behind the time daemon they are, that is, the difference between its time and the time in master.
(3) Based on the replies from the slaves, the master averages the time obtained (including its own).
(4) The master sends time rate adjust value (+ or -) to slaves, requesting them to adjust their time rate.
Question: How the Cristian’s algorithm is applied to the above Berkeley algorithm? 11

12. The Berkeley Algorithm: An Example
The time daemon asks all the other machines for their clock values
The machines answer
The time daemon tells everyone how to adjust their clock 12

13. Clock Synchronization: Network Time Protocol (NTP)
NTP is used in the Internet. Time servers are arranged in levels called strata. NTP intends to:
provide a service enabling clients in Internet to be synchronized accurately to UTC.
provide reliable service to losses of connectivity.
enable clients to resynchronize sufficiently and frequently.
provide protection against interference with the time service. 13

14. Clock Synchronization: Network Time Protocol
Synchronization clock: Network Time Protocol (NTP)
Stratum 1: Primary server (PS) sync directly with UTC sources.
Stratum 2: Secondary servers (SS) sync directly to PS.
Stratum 3: Lowest servers (LS) execute in user sites sync with SS.
Accuracy: the number of levers (strata). 1 2 3 14

15. Clock Synchronization: Network Time Protocol
Multiple servers provide for redundancies and hence availability of time sources and fault-tolerance.
NTP synchronizes with one another in one of three modes:
Multicast: Used on LAN. One or more servers periodically broadcast the time to servers running in workstations which set their clocks assuming a small delay. Accuracy is low.
Procedure-call: Similar to Cristian’s method. Workstations ask the server the time. Accuracy is higher than multicast.
Symmetric mode: Used by high level to achieve highest accuracy. A pair of servers exchange messages bearing timing information. 15

16. Logical Time and Logical Clock
For many purposes, it is sufficient that all machines agree on the same time. It is not essential that this time also agrees with the real time as announced on the radio every hour.
What usually matters is not that all processes agree on exactly what time it is, but rather that they agree on the order in which events occur. 16

17. Logical Time and Logical Clock
Lamport causal ordering x-P->y means x occurs before y in process P.
happened-before “->”
HB1: If for process P: x-P->y, then x-> y.
HB2: For the same message m, send(m)->receive(m)
HB3: If x, y, and z are events, x->y, y->z then x->z. p1 a b m1 p2 c d m2 e f p3 17

18. Logical Time and Logical Clock
Logical clock can capture happened-before ordering by logical clock.
A logical clock need not bear particularly relationship with physical clock.
Each process just keeps its own clock Cp. It is used to timestamp events.
Use Cp(a) and Cp(b) to timestamp events a and b happen in process p and C(b) for event b at whatever process it occurred. 18

19. Logical Time and Logical Clock
Logical clock update and transmission rules:
LC1:
Cp is incremented before each event is issued at p: Cp:= Cp+1;
LC2:
a) When p sends a message m, it piggybacks on m the value t=Cp.
b) On receiving (m, t), a process q computes Cq:=max(Cq, t) and then applies LC1 before time-stamping the event receive(m). 19

20. Logical Time and Logical Clock
Totally ordered logical clocks
for pair of distinct events and we take process id into consideration.
If a is an event occurring at pa with local timestamp Ta, and b is an event occurring at pb with local timestamp Tb.
Define the global logical timestamp for those events as (Ta, a) and (Tb, b), respectively.
(Ta, a) < (Tb, b) if and only if either
Ta < Tb or (Ta =Tb and pa < pb) 20

21. Example: Totally-Ordered Multicasting
Multicast: A message is sent to multiple receivers.
Totally-ordered multicast: all multicast messages are delivered in the same order to each receiver.
For example, to improve query performance, a bank may place copies of an account database in two different cities, say A and B.
* A customer in B wants to add $100 to his account that currently contains $1,000 (update 1);
* At the same time, a bank employee in A initiates an update by which the customer’s account is to increased with 1% interest (update 2). 21

22. Example: Totally-Ordered Multicasting
Both updates should be carried out at both copies (in A and B) of the database.
* If update 2 is performed before update 1 in A, the A’ database records $1,110;
* If update 1 is performed before update 2 in B, the B’ database records $1,111;
An inconsistence occurs if the two updates are not performed in the same order!
Solution: Using Lamport timestamp (with logical time) to implement totally-ordered multicast (for update messages), so that update operations are performed in the same order at each copy. 22

23. Example: Totally-Ordered Multicasting
Updating a replicated database and leaving it in an inconsistent state. 23

24. Totally-Ordered Multicasting Using Logical Time
For a group of processes multicasting messages to each other, we assume:
* Each message is timestamped with current logical time of its sender;
* The sender is also a receiver of its own sending message;
* The messages from the same sender are received in the order they were sent, and no messages are lost.
When process receives a message, it is put into a local queue, ordered according to its timestamp. The receiver multicasts an acknowledgement to the other processes. 24

25. Totally-Ordered Multicasting Using Logical Time
A process can deliver a queued message to the application it is running only when the message is at the head of the queue and has been acknowledged by each other process.
All processes will eventually have the same copy of the local queue by using Lamport’s timestamp..
Lamport’s clock ensures that no two messages have the same timestamp, and timestamps reflect a consistent global order of events.
Therefore, all message are delivered in the same order everywhere. That is, we have established totally-ordered multicasting. 25

26. Global State
It is often useful to know the global state in which a distributed system is currently residing, e.g., to detect the termination of a distributed computation, or deadlock etc.
Global state of a DS: The local state of each process, and the messages that are currently in transit.
Distributed snapshot: reflects a (consistent global) state in which the DS might have been. It is not desirable that a snapshot contains the recording of messages that have been received but never sent.
Cut: A graphical representation of global state, as shown in the next slide. 26

27. Global State (1)
A consistent cut
An inconsistent cut 27

28. Global State: Algorithm for Taking Snapshot I
Assume that the DS can be represented as a collection of processes connected to each other through unidirectional point-to-point communication channels (e.g., TCP connection).
Any initiating process, say P, may start by recording its own local state; then it sends a marker along each of its outgoing channels;
When a process Q receives a marker from an incoming channel C,
* If Q has not saved its local state, it first saves the state, then sends a marker along each of its own outgoing channels. 28

29. Global State: Algorithm for Taking Snapshot II
* If Q has recorded the local state, it records the state of channel C: the sequence of messages that have been received by Q since the last time Q recorded its local state, and before it received the marker.
When a process has received a marker along each of its incoming channels, and processed each one, its recorded local state and state of each channel are collected and sent to process P.
Because any process can initiate the algorithm, several snapshots can be constructed at the same time. To identify different processes of snapshot construction, a marker can be tagged with identifier (even version) of process that initiates the snapshot. 29

30. Global State (2)
Organization of a process and channels for a distributed snapshot 30

31. Global State (3)
Process Q receives a marker for the first time and records its local state
Q records all incoming message
Q receives a marker for its incoming channel and finishes recording the state of the incoming channel 31

32. Global State: Algorithm for Termination Detection I
The termination of a distributed computation can be detected using the algorithm of taking a snapshot.
A snapshot may show a global state in which message are still in transit, which means the distributed computation has not terminated.
The termination detection algorithm is a simple modification of the snapshot algorithm. Assume that a process Q receives the marker from process P, requesting a snapshot for the first time, then P is said to be the predecessor of Q.
When a process Q finishes its part of the snapshot, it either returns DONE or CONTINUE message to its predecessor: 32

33. Global State: Algorithm for Termination Detection II
* DONE: all of Q’s successors have returned a DONE message; and Q has not received any message between the point it recorded its state and the point it had received the marker along each of its incoming channel;
* CONTINUE: In all other cases.
When the original initiator of the snapshot, say P, receives only the DONE messages, it concludes that the distributed computation has terminated; Otherwise, P initiates another snapshot, and continues to do so until only DONE message are eventually returned. 33

34. Election Algorithms
Many distributed applications require one processor act coordinator, such as Berkeley algorithm for clock synchronization, or otherwise perform some special role.
If all processes are exactly the same, with no distinguishing characteristics, there is no way to select one of them to be special.
Assume that each processor has a unique number. The process with the highest number often could be elected.
The goal of an election algorithm is to ensure that when election starts, it concludes with all processes agreeing on who the new coordinator is to be. 34

35. Election Algorithms Need to find one process that is the coordinator
Assume
Each process has a unique identifier
network address for example
One process per machine
Every process knows the process number of every other process
Processes don’t know which processes are down and which ones are still running
End result of the algorithm: all processes agree on who is the new coordinator/leader
Bully algorithm & Ring Algorithm 35

36. Bully Algorithm (Garcia-Molina)
A process notices that coordinator is not responding
it starts an election (any process can start one)
Election algorithm
P sends an ELECTION message to processes with higher numbers
If no one responds, P wins the election
If some process with higher process number responds
P’s job is done, that process takes over
the receiver sends an OK message to P
receiver starts an election process
Eventually all processes give up, except one
This process sends out a message saying that it is the new “COORDINATOR”
A process that was down, when it comes back up starts a new election of its own 36

37. The Bully Algorithm (1)
The bully election algorithm; pcs 7 as coordinator=> crashed
Process 4 is the first on noticed the crashed >> send ELECTION process to 5, 6 and 7 (higher pcs)
Process 5 and 6 respond, telling 4 to stop
Now 5 and 6 each hold an election 37

38. The Bully Algorithm (2) Process 6 tells 5 to stop
Process 6 wins and tells everyone 38

39. The Bully Algorithm (Summarize)
It is assumed that every process knows the processor numbers of every other process, but does not know which one is currently up or down.
When any process P notices that the current coordinator is no longer responding the request, it initiates an election:
1. P sends an ELECTION message to all processes with higher numbers;
2. If no one responds, P wins the lection and becomes coordinator;
3. If one of the highest-ups answers, it takes over. P’s job is done. 39

40. The Bully Algorithm (Summarize)
When a process gets a message form one of its lower-numbered processes, it sends an OK message back to the sender to indicate that it is alive and will take over, and it then holds an election (unless it is already doing so).
Eventually, all processes but one give up, and that one is the new coordinator.
The new coordinator announces its victory by sending all processes a message telling them that starting immediately it is the new coordinator.
If a process that was previously down comes back up, it holds an election, and may win the election if it happens to be highest number process among all the running processes. 40

41. Ring Algorithm (1) Does NOT use a token Assume
processes are ordered
each process knows its successor
and the successor’s successor, and so on (needed in case of failures)
Process P detects that the coordinator is dead
sends an ELECTION message to its successor
includes its process number in the message
each process that receives it
adds its own process number and then forwards it to its successor
eventually it gets back that message
now what does it do? 41

42. Ring Algorithm (2)
The process that initiated it, then sends out a message saying “COORDINATOR”
the process with highest number in list is the leader
when this comes back, then process P deletes it 42

43. Election algorithm using a ring.
A Ring Algorithm(1)
Election algorithm using a ring.
We start with 6 processes,     connected in a logical ring.
Process 6 is the leader,     as it has the highest number. 43

44. A Ring Algorithm(2)
Process 6 fails. 44

45. A Ring Algorithm(3) Process 3 notices that Process 6 does not respond
So it starts an election, sending a message containing its id to the next node in the ring. 45

46. A Ring Algorithm(4)
Process 5 passes the message on,     adding its own id to the message 46

47. A Ring Algorithm(5)
Process 0 passes the message on,     adding its own id to the message. 47

48. A Ring Algorithm(6)
Process 1 passes the message on,     adding its own id to the message. 48

49. A Ring Algorithm(7)
Process 4 passes the message on,     adding its own id to the message 49

50. A Ring Algorithm(8)
When Process 3 receives the message back,     it knows the message has gone around the ring,         as its own id is in the list.
Picking the highest id in the list,     it starts the coordinator message         "5 is the leader" around the ring 50

51. A Ring Algorithm(9)
Process 5 passes on the coordinator message 51

52. A Ring Algorithm(10)
Process 0 passes on the coordinator message. 52

53. A Ring Algorithm(11)
Process 1 passes on the coordinator message. 53

54. A Ring Algorithm(12) Process 4 passes on the coordinator message
Process 3 receives the coordinator message,     and stops it. 54

55. Ring Algorithm (Summarize)
Assume that the processes are physically or logically ordered, so that each process knows who its successor is.
When any process notices that the coordinator is not functioning, it builds an ELECTION message containing its process number and sends the message to its successor. If the successor is down, the sender skips over it and goes to the next number along the ring, or the next after that, until a running process is located.
At each step, the sender adds its own process number (as a candidate for coordinator) to the list in the message.
Eventually, the message gets back to the process that started it all. The process can recognize this event. 55

56. Ring Algorithm (Summarize)
The process changes the message type to COORDINATOR and circulates it once again, to inform everyone else who the new coordinator is (the list number with the highest number) and who the numbers of the new ring are.
When the COORDINATOR message has circulated once, it is removed and everyone goes back to work. 56

57. Summary I
There is no notion of a globally shared clock. Accurate timekeeping is important for distributed systems.
There are various ways to synchronize clocks in a DS, essentially all are based on exchanging clock values. The accuracy of clock synchronization depends on the variation in communication delays and how they are dealt with.
For ordering of an arbitrary pair of events at different computers, clock synchronization is not always practical.
The happened-before relation is a partial order on events that reflects a flow of information between them. 57

58. Summary II
Lamport clocks are counters that are updated according to the happened-before relationship between events.
There is no globally shared memory in DS. It is often hard to determine the system’s current state, which can be done by synchronizing all processes so that each collects its local state and the messages currently in transit (using snapshot).
Synchronization between processes often requires that one process acts as a coordinator. A decision on who is going to be the new coordinator is taken by means of election algorithms. These algorithms are primarily used in cases where the coordinator can crash. 58

59. Tutorial
Q1. Discuss the features and applications of Cristian algorithm, Berkeley algorithm, and network time protocol in clock synchronization.
Q2. In the example of the Berkeley algorithm in the lecture (see the next slide), the effect of transmission delay of the message is not considered. How the Cristian’s algorithm is applied to Berkeley method to improve its accuracy? 59

60. Tutorial: An Example of the Berkeley Algorithm
The time daemon asks all the other machines for their clock values
The machines answer
The time daemon tells everyone how to adjust their clock 60

61. Tutorial
Q3. (Cristian’s algorithm) A client attempts to synchronize with a time server. It records the following round-trip times and timestamps returned by the server:
Round trip (ms) Time (hr:min:sec)
:54:23:674
:54:25:450
:54:28:342
Which of these times should it use to set its clock? To what time should it set? Estimate the accuracy of the setting with respect to the server’s clock. If it is known that the time between sending and receiving a message in the system concerned is at least 8 ms, do your answer change? 61

62. Tutorial
Q4. Given the event sequences in the next slide. Suppose each logical clock starts at 0. Show how Lamport’s clock mechanism adjusts the local clocks. Also, show how the totally ordered logical clock can be used so that each event has a unique timestamp using logical clock . 62

63. Tutorial: Event Sequences in Three Processes
a b c d e1 p1
a2 b c2 P2 P3
a b c d3 e f3 63

64. Tutorial
Q5. Suppose that there are three processes P1, P2, and P3. Each process multicasts two messages to the other processes, including itself as follows:
* P1 multicasts message m1 with timestamp (9, 1) and message m2 with timestamp (11, 1);
* P2 multicasts message m3 with timestamp (9, 2) and message m4 with timestamp (10, 2);
* P3 multicasts message m5 with timestamp (10, 3) and message m6 with timestamp (12, 3).
How the totally-ordered multicasting is achieved (under the same assumptions in Lecture Note 5), and what is the order that all the 6 messages are received in each process? 64