1. Communication & Synchronization Why do processes communicate in DS? –To exchange messages –To synchronize processes Why do processes synchronize in DS? –To coordinate access of shared resources –To order events

2. Time, Clocks and Clock Synchronization Time –Why is time important in DS? –E.g. UNIX make utility Clocks (Timer) –Physical clocks –Logical clocks (introduced by Leslie Lamport) –Vector clocks (introduced by Collin Fidge) Clock Synchronization –How do we synchronize clocks with real-world time? –How do we synchronize clocks with each other? 05 – 1 Distributed Algorithms/5.1 Clock Synchronization

3. Physical Clocks (1/3) Problem: Clock Skew – clocks gradually get out of synch and give different values Solution: Universal Coordinated Time (UTC): Based on the number of transitions per second of the cesium 133 atom (very accurate). At present, the real time is taken as the average of some 50 cesium-clocks around the world – International Atomic Time Introduces a leap second from time to time to compensate that days are getting longer. UTC is broadcasted through short wave radio (with the accuracy of +/- 1 msec) and satellite (Geostationary Environment Operational Satellite, GEOS, with the accuracy of +/- 0.5 msec). Question: Does this solve all our problems? Don’t we now have some global timing mechanism? 05 – 2 Distributed Algorithms/5.1 Clock Synchronization

4. Physical Clocks (2/3) Problem: Suppose we have a distributed system with a UTC- receiver somewhere in it, we still have to distribute its time to each machine. Basic principle: Every machine has a timer that generates an interrupt H (typically 60) times per second. There is a clock in machine p that ticks on each timer interrupt. Denote the value of that clock by Cp ( t), where t is UTC time. Ideally, we have that for each machine p, Cp ( t) = t, or, in other words, dC/ dt = 1 Theoretically, a timer with H=60 should generate 216,000 ticks per hour In practice, the relative error of modern timer chips is 10**-5 (or between 215,998 and 216,002 ticks per hour) 05 – 3 Distributed Algorithms/5.1 Clock Synchronization

5. Physical Clocks (3/3) Goal: Never let two clocks in any system differ by more than  time units => synchronize at least every  2  seconds. 05 – 4 Distributed Algorithms/5.1 Clock Synchronization Where   is the max. drift rate Principle I: Cristian Algorithm Every machine asks a time server for the accurate time. But you need an accurate measure of round trip delay, including interrupt handling and processing incoming messages.

6. Principle II: The Berkeley Algorithm Let the time server scan all machines periodically, calculate an average, and inform each machine how it should adjust its time relative to its present time.  The time server polls periodically every machine for its time  The received times are averaged and each machine is notified of the amount of the time it should adjust Note: you don’t even need to propagate UTC time 05 – 5 Distributed Algorithms/5.1 Clock Synchronization

7. The Happened-Before Relationship Problem: We first need to introduce a notion of ordering before we can order anything. The happened-before relation on the set of events in a distributed system is the smallest relation satisfying: If a and b are two events in the same process, and a comes before b, then a  b. (a happened before b) If a is the sending of a message, and b is the receipt of that message, then a  b. If a  b and b  c, then a  c. (transitive relation) Note: if two events, x and y, happen in different processes that do not exchange messages, then they are said to be concurrent. Note: this introduces a partial ordering of events in a system with concurrently operating processes. 05 – 6 Distributed Algorithms/5.2 Logical Clocks

8. Logical Clocks (1/2) Problem: How do we maintain a global view on the system’s behavior that is consistent with the happened-before relation? Solution: attach a timestamp C ( e) to each event e, satisfying the following properties: P1: If a and b are two events in the same process, and a  b, then we demand that C ( a) < C ( b) P2: If a corresponds to sending a message m, and b to the receipt of that message, then also C ( a) < C ( b) Problem: How do we attach a timestamp to an event when there’s no global clock?  maintain a consistent set of logical clocks, one per process. 05 – 7 Distributed Algorithms/5.2 Logical Clocks

9. Event counting example Three systems: P 0, P 1, P 2 Events a, b, c, … Local event counter on each system Systems occasionally communicate

10. Event counting example ab hi k P1P1 P2P2 P3P3 12 13 21 df g 3 c 2 4 6 e 5 j

11. Logical Clocks (2/2) Each process Pi maintains a local counter Ci and adjusts this counter according to the following rules: (1) For any two successive events that take place within Pi, Ci is incremented by 1. (2) Each time a message m is sent by process Pi, the message receives a timestamp Tm = Ci. (3) Whenever a message m is received by a process Pj, Pj adjusts its local counter Cj : Property P1 is satisfied by (1); Property P2 by (2) and (3). This is called the Lamport’s Algorithm 05 – 8 Distributed Algorithms/5.2 Logical Clocks

12. Logical Clocks – Example 05 – 9 Distributed Algorithms/5.2 Logical Clocks Fig 5-7. (a) Three processes, each with its own clock. The clocks run at different rates. (b) Lamport’s algorithm corrects the clocks

13. Assign the Lamport’s logical clock values for all the events in the above timing diagram. Assume that each process’s local clock is set to 0 initially. a b c d e f g h i j k l P1P2 P3

14. From the above timing diagram, what can you say about the following events? between a and b:a  b between b and f : b  f between e and k:concurrent between c and h:concurrent between k and h:k  h a b c d e f g h i j k l P1P2 P3 1 2 3 4 1 3 4 5 6 1 2 3

15. Total Ordering with Logical Clocks Problem: it can still occur that two events happen at the same time. Avoid this by attaching a process number to an event: Pi timestamps event e with C i ( e).i Then: C i ( a).i happened before C j ( b).j if and only if: 1: C i ( a) < C j ( a) ; or 2: C i ( a) = C j ( b) and i < j 05 – 10 Distributed Algorithms/5.2 Logical Clocks

16. Example: Totally-Ordered Multicast (1/2) Problem: We sometimes need to guarantee that concurrent updates on a replicated database are seen in the same order everywhere: Process P1 adds $100 to an account (initial value: $1000) Process P2 increments account by 1% There are two replicas Outcome: in absence of proper synchronization, replica #1 will end up with $1111, while replica #2 ends up with $1110. 05 – 11 Distributed Algorithms/5.2 Logical Clocks

17. Example: Totally-Ordered Multicast (2/2) Process Pi sends timestamped message msg i to all others. The message itself is put in a local queue queue i. Any incoming message at Pj is queued in queue j, according to its timestamp. Pj passes a message msg i to its application if: (1) msg i is at the head of queue j (2) for each process Pk, there is a message msg k in queue j with a larger timestamp. Note: We are assuming that communication is reliable and FIFO ordered. 05 – 12 Distributed Algorithms/5.2 Logical Clocks

18. Fidge’s Logical Clocks with Lamport’s clocks, one cannot directly compare the timestamps of two events to determine their precedence relationship - if C(a) < C(b) then a  b - if C(a) < C(b), it could be a  b or a  b - e.g., events e and b in the previous example Figure * C(e) = 1 and C(b) = 2 * thus C(e) < C(b) but e  b the main problem is that a simple integer clock can not order both events within a process and events in different processes Collin Fidge developed an algorithm that overcomes this problem Fidge’s clock is represented as a vector [c 1, c 2, …, c n ] with an integer clock value for each process (c i contains the clock value of process i) // / /

19. Fidge’s Algorithm The Fidge’s logical clock is maintained as follows: 1: Initially all clock values are set to the smallest value. 2: The local clock value is incremented at least once before each primitive event in a process. 3: The current value of the entire logical clock vector is delivered to the receiver for every outgoing message. 4: Values in the timestamp vectors are never decremented. 5: Each time a process prepares to send a message, it increments its own logical clock in the vector by one and then sends its entire vector along with the message being sent. 6: Each time a process receives a message, it increments its own logical clock in the vector by one and updates each element in its vector by taking the maximum of the value in its own vector clock and the value in the vector in the received message (for every element).

20. Vector Clock

21. Get r_vector from the received msg sent by process q; if l_vector [q]  r_vector[q] then l_vector[q] : = r_vector[q] + 1; for i : = 1 to n do l_vector[i] := max(l_vector[i], r_vector[i]); Timestamps attached to the events are compared as follows: e p  f q iff T e p [p] < T f q [p] (where e p represents an event e occurring in process p, T e p represents the timestamp vector of the event e p, and the i th element of T e p is denoted by T e p [i].) This means event e p happened before event f q if and only if process q received a direct or indirect message from p and that message was sent after e p had occurred. If e p and f q are in the same process (i,e., p = q), the local elements of their timestamps represent their occurrences in the process.

22. Assign the Lamport’s and Fidge’s logical clock values for all the events in the above timing diagram. Assume that each process’s logical clock is set to 0 initially. a b c d e f g h i j k l P1P2 P3

23. P1P2 P3 a b c d e f g h i j k l 1 2 3 4 1 3 4 5 6 1 2 3 [1,0,0] [2,0,0] [3,0,0] [4,0,0] [0,1,0] [3,2,0] [3,3,3] [3,4,3] [5,5,3] [0,0,3] [0,0,2] [0,0,1]

24. The above diagram shows both Lamport timestamps (an integer value ) and Fidge timestamps (a vector of integer values ) for each event. –Lamport clocks: b  h implies 2 < 5 3 < 4 but c  g. –Fidge clocks: f  h since 2 < 4 is true, b  h since 2 < 3 is true, h  a since 4 < 0 is false, c  h since (3 < 3) is false and (4 < 0) is false.