1. Distributed Computing
Synchronization
Dr. Yingwu Zhu

2. Topics to Discuss Physical vs. Logical Clocks Lamport Vector Clocks
Lamport Clocks
Lamport Vector Clocks
Mutual Exclusion Algorithms
Election Algorithms

3. Synchronization? What’s for?
Temporal ordering of events produced by concurrent processes
Synchronization between senders and receivers of message
Msg m1 from process P to Q is sent before or after msg m2 from process Q?
Coordination of joint activity
Serialization of concurrent access for shared objects (e.g., access to a shared printer)

4. An Ideal World
All machines’ clocks are perfectly synchronized, synchronization is really easy!

5. Clock Synchronization Example
In centralized systems, no problem for the above Make program
In distributed systems, when each machine has its own clock, an event that occurred after another event may nevertheless be assigned an earlier time.

6. Logical vs. Physical Clocks
Logical clock keeps track of event ordering
Among related (causal) events
Do not care the real time where events occurred
Physical clock keeps time of day
Consistent across systems

7. Physical Clock (timer) in Computers
Real-time Clock: CMOS clock (counter) circuit driven by a quartz oscillator
battery backup to continue measuring time when power is off
OS generally programs a timer circuit to generate an interrupt periodically
e.g., 60, 100, 250, 1000 interrupts per second(Linux 2.6+ adjustable up to 1000 Hz)
Programmable Interval Timer (PIT) –Intel 8253, 8254
Interrupt service procedure adds 1 to a counter in memory

8. Physical Clock Problems
Getting two systems to agree on time
Two clocks hardly ever agree
Quartz oscillators oscillate at slightly different frequencies
Clocks tick at different rates
Create ever-widening gap in perceived time
Clock Drift
Difference between two clocks at one point in time
Clock Skew

9. Clock Drift
Frequencies of perfect, slow and fast clocks

10. Dealing with Drift Assume we set computer to true time
Not good idea to set clock back
Illusion of time moving backwards can confuse message ordering and software development environments

11. Dealing with Drift Go for gradual clock correction
If fast:
Make clock run slower until it synchronizes
If slow:
Make clock run faster until it synchronizes
Clock synchronization, e.g., Linear compensation function

12. Compensating for a fast clock

13. Getting Accurate Time Not practical solution for every machine
Attach GPS receiver to each computer
±1 msec of UTC (Universal Coordinated Time)
Attach WWV radio receiver
Obtain time broadcasts from Boulder or DC
±3 msec of UTC (depending on distance)
Attach GOES receiver
±0.1 msec of UTC
Not practical solution for every machine
Cost, size, convenience, environment

14. Practical Clock Synchronization
NTP (Network Time Protocol)
Berkeley algorithm

15. Clock Synchronization: Network Time Protocol (NTP)
Synchronize from another machine
One with a more accurate clock
Machine/service that provides time information:
Time server (w/ WWV receiver)

16. Clock Synchronization: NTP
Assumption: latency AB and BA is same, and good estimate!
Offset of A to B: theta = T3 - [(T2-T1) + (T4-T3)] / 2
Delay estimate: delta = [(T2-T1) + (T4-T3)] / 2, keeps the minimum one!
Adjust gradually: e.g., to slow down, add a smaller time for each interrupt

17. Clock Synchronization: The Berkeley Algorithm
The time server is active, polling every machine periodically for their time
Based on responses, it computes an average time and tell every machine to adjust their clocks
Used scenarios: No machine has a WWV receiver
All machines agree on the same time, but not necessarily the real time

18. The Berkeley Algorithm (1)
The time daemon asks all the other machines for their clock values.

19. The Berkeley Algorithm (2)
The machines answer.

20. The Berkeley Algorithm (3)
The time daemon tells everyone how to adjust their clock.

21. Logical Clocks In a classic paper (1978), Lamport showed
Although clock synchronization is possible, it need not be absolute
If two processes do not interact, it is not necessary that their clocks be synchronized!
More importantly, the processes should agree on the order in which events occur! This matters!

22. Logical Clocks Assign sequence numbers to messages
All cooperating processes can agree on order of events
vs. physical clocks: time of day
Assume NO central time source
Each system maintains its own local clock
No total ordering of events
No concept of happened-when

23. Lamport’s Logical Clocks (1)
The "happens-before" relation → can be observed directly in two situations:
If a and b are events in the same process, and a occurs before b, then a → b is true.
If a is the event of a message being sent by one process, and b is the event of the message being received by another process, then a → b
Happens-before is transitive: if a  b and b c then a c

24. Logical clocks & concurrency
Assign “clock” value to each event
if ab then clock(a) < clock(b)
since time cannot run backwards
If a and b occur on different processes that do not exchange messages, then neither a  b nor ba are true
These events are concurrent

25. Lamport’s Logical Clocks (1)
(a) Three processes, each with its own clock. The clocks run at different rates.
Lamport clocks: Counters or Sequence numbers

26. Lamport’s Logical Clocks (2)
(b) Lamport’s algorithm corrects the clocks (by adding 1).

27. Lamport’s Logical Clocks (3)
Figure The positioning of Lamport’s logical clocks in distributed systems.

28. Lamport’s Logical Clocks (4)
Updating the local counter Ci for process Pi
Each process maintains a local counter
Before executing an event Pi executes Ci ← Ci + 1.
When process Pi sends a message m to Pj, it sets m’s timestamp ts (m) equal to Ci after having executed the previous step.
Upon the receipt of a message m, process Pj adjusts its own local counter as Cj ← max{Cj , ts (m)}, after which it then executes the first step and delivers the message to the application.

29. Lamport’s algorithm
Each message carries a timestamp of the sender’s clock
When a message arrives:
if receiver’s clock < message timestamp, set system clock to (message timestamp + 1)
else do nothing
Clock must be advanced between any two events in the same process

30. Lamport’s algorithm
Algorithm allows us to maintain time ordering among related events
Partial ordering

31. Summary Algorithm needs monotonically increasing software counter
Incremented at least when events that need to be timestamped occur
Each event has a Lamport timestamp attached to it
For any two events, where ab: C(a) < C(b)

32. Example: Totally Ordered Multicasting
Updating a replicated database and leaving it in an inconsistent state.
Totally Ordered Multicasting:
all msgs are delivered in the same order to each receiver!
Can be implemented by Lamport’s logical clocks (multicast messages and acks, msg queue ordered by timestamp, msg delivered to app if it is the first and acks from all nodes are received)

33. Problems Identical timestamps: two events could be concurrent
Detect causal relations
If C(e) < C(e’), cannot conclude that ee’
Looking at Lamport timestamps, cannot conclude which events are causally related
Solution: use a vector clock

34. Vector Clocks (1)
Concurrent message transmission using logical clocks.
Trcv(m1) < Tsnd(m2), but m1 and m2 are concurrent
Lamport clocks do not capture causality!

35. Vector Clocks (2)
Vector clocks are constructed by letting each process Pi maintain a vector VCi with the following two properties:
VCi [ i ] is the number of events that have occurred so far at Pi. In other words, VCi [ i ] is the local logical clock at process Pi .
If VCi [ j ] = k then Pi knows that k events have occurred at Pj. It is thus Pi’s knowledge of the local time at Pj .
If VC[a] < VC[b] then event a  event b

36. Vector Clocks (3)
Steps carried out to accomplish property 2 of previous slide:
Before executing an event Pi executes VCi [ i ] ← VCi [i ] + 1.
When process Pi sends a message m to Pj, it sets m’s (vector) timestamp ts (m) equal to VCi after having executed the previous step.
Upon the receipt of a message m, process Pj adjusts its own vector by setting VCj [k ] ← max{VCj [k ], ts (m)[k ]} for each k, after which it executes the first step and delivers the message to the application.

37. Enforcing Causal Communication
Figure Enforcing causal communication.
Causally ordered multicasting: weaker than totally ordered multicasting; if 2 msgs are not related to each other, we do not care in which order they are delivered to apps.
Assume clocks are adjusted only when sending/receiving msgs. Sending by incrementing the item in the VC by 1; receiving only by adjusting to max for all components in VC. A msg from process i is delivered to apps only the following 2 conditions are met:
1) ts(m)[i] = VCj [i ] ) ts(m)[k] <= VCj[k] for all k != i

38. Mutual Exclusion A Centralized Algorithm
(a) Process 1 asks the coordinator for permission to access a shared resource. Permission is granted.
(b)Process 2 then asks permission to access the same resource. The coordinator does not reply.
(c) When process 1 releases the resource, it tells the coordinator, which then replies to 2.

39. Mutual Exclusion A Centralized Algorithm
Simple: 3 messages: request, grant, release
Downsides
Simple point of failures
Performance bottleneck

40. Mutual Exclusion : A Distributed Algorithm
The requestor broadcasts a message containing the requested resource, process id, and logical time
Three different cases:
If the receiver is not accessing the resource and does not want to access it, it sends back an OK message to the sender.
If the receiver already has access to the resource, it simply does not reply. Instead, it queues the request.
If the receiver wants to access the resource as well but has not yet done so, it compares the timestamp of the incoming message with the one contained in the message that it has sent everyone. The lowest one wins. (Lamport’s clock vector to implement tm)

41. Mutual Exclusion A Distributed Algorithm
Two processes want to access a shared resource at the same moment.
Process 0 has the lowest timestamp, so it wins.
When process 0 is done, it sends an OK also, so 2 can now go ahead.

42. Mutual Exclusion A Distributed Algorithm
Message complexity: 2(n-1) per entry
Magnify the single point of failure problem in centralized algorithms (n points)
Group membership is known
Bottleneck: each machine handles same load, but machines may be heterogeneous

43. Mutual Exclusion A Token Ring Algorithm
(a) An unordered group of processes on a network. (b) A logical ring constructed in software.

44. Mutual Exclusion A Token Ring Algorithm
Problems
Lost tokens, how to detect them?
Process failures, how to detect them?

45. Election Algorithms
Many distributed systems require one process to act as coordinator/initiator, or perform some special role
Elect one to fit into that role
In general, election algorithms attempt to locate the process with the highest process number as the coordinator
Traditional alg. assumes: message passing is reliable; network topology does not change

46. Election Algorithms The Bully Algorithm
P sends an ELECTION message to all processes with higher numbers.
If no one responds, P wins the election and becomes coordinator.
If one of the higher-ups answers, it takes over. P’s job is done.

47. The Bully Algorithm (1) (a) Process 4 holds an election.
(b) Processes 5 and 6 respond, telling 4 to stop.
(c) Now 5 and 6 each hold an election.

48. The Bully Algorithm (2) (d) Process 6 tells 5 to stop.
(e) Process 6 wins and tells everyone.

49. Election: A Ring Algorithm
Figure Election algorithm using a ring.
After discovering crash of the old coordinator, some process initiate the ELECTION message circulating the ring (containing process numbers whose processes saw the message)
Then, the COORDINATOR message is circulating again, containing all the members, the process with the highest number is the new coordinator