1. UBI529 Distributed Algorithms 2. Time Synchronization in Distributed Systems

2. 2 Assumptions Process can perform three kinds of atomic actions/events n Local computation n Send a single message to a single process n Receive a single message from a single process Communication model n Point-to-point n Error-free, infinite buffer n Potentially out of order

3. 3 Motivation Many protocols need to impose an ordering among events n If two players open the same treasure chest “almost” at the same time, who should get the treasure? Physical clocks: n Seems to completely solve the problem n But what about theory of relativity? n Even without theory of relativity – efficiency problems How accurate is sufficient? n Without out-of-band communication: Minimum message propagation delay n With out-of-band communication: distance/speed of light n In other words, some time it has to be “quite” accurate

4. 4 Physical Clock Synchronization The relation between clock time and UTC when clocks tick at different rates.

5. 5 Classification Types of Synchronization External Synchronization : Keep as close to UTC as possible Internal Synchronization : Mutual synchronization is important Phase Synchronization Types of clocks Unbounded 0, 1, 2, 3,... Bounded 0,1, 2,... M-1, 0, 1,... Unbounded clocks are not realistic, but are easier to deal with in algorithms. Real clocks are always bounded.

6. 6 Terminology Drift rate  : Max rate two clocks can differ, < 10^-6 for Xtal clocks Clock skew  : Max. Allowed drift Resynchronization interval R : Depends on clock skew Challenges (Drift is unavoidable) Accounting for propagation delay Accounting for processing delay Faulty clocks

7. 7 Internal synchronization A simple averaging algorithm Assume n clocks, at most t are faulty Step 1. Read every clock in the system. Step 2. Discard outliers and substitute them by the value of the local clock. Step 3. Update the clock using the average of these values. Synchronization is maintained if n > 3t Why? Some clocks may be faulty. Exhibits 2-faced behavior

8. 8 Internal synchronization A simple averaging algorithm The maximum difference between the averages computed by two non-faulty nodes is (3td / n) To keep the clocks synchronized, 3td /n < d So, 3t < n

9. 9 Network Time Protocol (NTP) Tiered architecture Broadcast mode - least accurate Procedure call - medium accuracy Peer-to-peer mode - upper level servers use this for max accuracy Time server The tree can reconfigure itself if some node fails. Level 1 Level 0 Level 2

10. 10 P2P mode of NTP Let Q’s time be ahead of P’s time by . Then T2 = T1 + T PQ +  T4 = T3 + T QP -  y = T PQ + T QP = T2 +T4 -T1 -T3 (RTT)  = (T2 -T4 -T1 +T3) / 2 - (T PQ - T QP ) / 2 So, x- y/2 ≤  ≤ x+ y/2 T2 T1T4 T3 Q P Ping several times, and obtain the smallest value of y. Use it to calculate  x Between y/2 and -y/2

11. 11 Cristian’s Algorithm 1. Client gets data from a time server. 2. It computes the round trip time (RTT), and compensates for this delay while adjusting its own clock

12. 12 Berkeley Time Protocol (FromTanenbaum 2007) a) The time daemon asks all the other machines for their clockvalues. b) The machines answer The time daemon tells everyone how to adjust their clock.

13. 13 Software “Clocks” Software “clocks” can incur much lower overhead than maintaining (sufficiently accurate) physical clocks Allows a protocol to infer ordering among events Goal of software “clocks”: Capture event ordering that are visible to users n But what orderings are visible to users without physical clocks?

14. 14 Visible Ordering to Users A  B (process order) B  C (send-receive order) A  C (transitivity) user1 (process1) user2 (process2) user3 (process3) A B C D A ? D B ? D C ? D

15. 15 “Happened-Before” Relation “Happened-before” relation captures the ordering that is visible to users when there is no physical clock n A partial order among events n Process-order, send-receive order, transitivity First introduced by Lamport – Considered to be the first fundamental result in distributed computing Goal of software “clock” is to capture the above “happened-before” relation. Formally, a logical clock C is a map from the set of events E to N (the set of natural numbers) with the following constraint:

16. 16 1: Logical Clocks Each event has a single integer as its logical clock value n Each process has a local counter C n Increment C at each “local computation” and “send” event n When sending a message, logical clock value V is attached to the message. At each “receive” event, C = max(C, V) + 1 user1 (process1) user2 (process2) user3 (process3) 1234 1 3 = max(1,2)+1 45 12 5 3 4 = max(3,3)+1

17. 17 Logical Clock Properties Theorem: n Event s happens before t  the logical clock value of s is smaller than the logical clock value of t. The reverse may not be true user1 (process1) user2 (process2) user3 (process3) 1234 145 12 5 3

18. 18 Lamport’s Logical Clock Algorithm

19. 19 Lamport's logical clock algorithm in Java public class LamportClock { int c; public LamportClock() { c = 1; } public int getValue() { return c; } public void tick() { // on internal actions c = c + 1; } public void sendAction() { // include c in message c = c + 1; } public void receiveAction(int src, int sentValue) { c = Util.max(c, sentValue) + 1; }

20. 20 2: Vector Clocks Logical clock: n Event s happens before event t  the logical clock value of s is smaller than the logical clock value of t. Vector clock: n Event s happens before event t  the vector clock value of s is “smaller” than the vector clock value of t. Each event has a vector of n integers as its vector clock value n v1 = v2 if all n fields same n v1  v2 if every field in v1 is less than or equal to the corresponding field in v2 n v1 < v2 if v1  v2 and v1 ≠ v2 Relation “<“ here is not a total order

21. 21 Vector Clock Protocol Each process i has a local vector C Increment C[i] at each “local computation” and “send” event When sending a message, vector clock value V is attached to the message. At each “receive” event, C = pairwise-max(C, V); C[i]++; user1 (process1) user2 (process2) user3 (process3) (1,0,0)(2,0,0)(3,0,0) (0,0,2) (0,2,2) (0,1,0) (0,0,1) (2,3,2) (2,3,3) C = (0,1,0), V = (0,0,2) pairwise-max(C, V) = (0,1,2)

22. 22 Vector Clock Properties Event s happens before t  vector clock value of s < vector clock value of t: There must be a chain from s to t Event s happens before t  vector clock value of s < vector clock value of t n If s and t on same process, done n If s is on p and t is on q, let VS be s’s vector clock and VT be t’s n VS < VT  VS[p] ≤ VT[p]  Must be a sequence of message from p to q after s and before t q s t p

23. 23 Example Application of Vector Clock Each user has an order placed by a customer n Want all users to know all orders Each order has a vector clock value user1 (process1) user2 (process2) user3 (process3) D1, (1,0,0)(2,0,0)(3,0,0) (0,0,2) (0,2,2) D2, (0,1,0) D3, (0,0,1) (2,3,2) (2,3,3) D1 D3 D1, D2, D3 I have seen all orders whose vector clock is smaller than (2,3,2): I can avoid linear search for existence testing

24. 24 Vector Clock Algorithm

25. 25 A sample execution of the vector clock algorithm

26. 26 Vector clock algorithm in Java public class VectorClock { public int[] v; int myId; int N; public VectorClock(int numProc, int id) { myId = id; N = numProc; v = new int[numProc]; for (int i = 0; i < N; i++) v[i] = 0; v[myId] = 1; } public void tick() { v[myId]++; } public void sendAction() { //include the vector in the message v[myId]++; } public void receiveAction(int[] sentValue) { for (int i = 0; i < N; i++) v[i] = Util.max(v[i], sentValue[i]); v[myId]++; }

27. 27 3: Matrix Clocks Motivation n My vector clock describe what I “see” n In some applications, I also want to know what other people see Matrix clock: n Each event has n vector clocks, one for each process n The i th vector on process i is called process i ’s principle vector n Principle vector is the same as vector clock before n Non-principle vectors are just piggybacked on messages to update “knowledge”

28. 28 Matrix Clock Protocol For principle vector C on process i n Increment C[i] at each “local computation” and “send” event n When sending a message, all n vectors are attached to the message n At each “receive” event, let V be the principle vector of the sender. C = pairwise-max(C, V); C[i]++; For non-principle vector C on process i, suppose it corresponds to process j n At each “receive” event, let V be the vector corresponding to process j as in the received message. C = pairwise-max(C, V)

29. 29 Matrix Clock Example user1 (process1) user2 (process2) user3 (process3) (1,0,0) (0,0,0) (0,1,0) (0,0,0) (0,0,1) (2,0,0) (0,0,0) (3,0,0) (0,0,0) (0,0,2) (0,0,0) (0,2,2) (0,0,2) (2,0,0) (2,3,2) (0,0,2) (2,0,0) (2,3,2) (2,3,3)

30. 30 Matrix Clock Example (1,0,0) (0,0,0) (0,1,0) (0,0,0) (0,0,1) (2,0,0) (0,0,0) (3,0,0) (0,0,0) (0,0,2) (0,0,0) (0,2,2) (0,0,2) (2,0,0) (2,3,2) (0,0,2) (2,0,0) (2,3,2) (2,3,3) Theorem: On any given process, any non-principle vector is smaller than or equal to the principle vector.

31. 31 Application of Matrix Clock user1 (process1) gossip D1 = (1,0,0) user2 (process2) gossip D2 = (0,1,0) user3 (process3) gossip D3 = (0,0,1) (1,0,0) (0,0,0) (0,1,0) (0,0,0) (0,0,1) (2,0,0) (0,0,0) (3,0,0) (0,0,0) (0,0,2) (0,0,0) (0,2,2) (0,0,2) (2,0,0) (2,3,2) (0,0,2) (2,0,0) (2,3,2) (2,3,3) D1 D3 D1, D2, D3 user3 now knows that all 3 users have seen D1

32. 32 Matrix Clock Algorithm

33. 33 The matrix clock algorithm in Java public class MatrixClock { int[][] M; int myId; int N; public MatrixClock(int numProc, int id) { myId = id; N = numProc; M = new int[N][N]; for (int i = 0; i < N; i++) for (int j = 0; j < N; j++) M[i][j] = 0; M[myId][myId] = 1; } public void tick() { M[myId][myId]++; } public void sendAction() { //include the matrix in the message M[myId][myId]++; } public void receiveAction(int[][] W, int srcId) { // component-wise maximum of matrices for (int i = 0; i < N; i++) if (i != myId) { for (int j = 0; j < N; j++) M[i][j] = Util.max(M[i][j], W[i][j]); }

34. 34 Discussions Vector clock tells me what I know n one-dimensional data structure Matrix clock tells me what I know about what other people know n Two-dimensional data structure ?? tells me what I know about what other people know about what other people know n ??-dimensional data structure

35. 35 Summary Goal : Define “time” in a distributed system Logical clock n “happened before”  smaller clock value Vector clock n “happened before”  smaller clock value Matrix clock n Gives a process knowledge about what other processes know

36. 36 Acknowledgements This part is heavily dependent on the course : CS4231 Parallel and Distributed Algorithms, NUS by Dr. Haifeng Yu and Dr. Vijay Gargh Elements of Distributed Computing Book and also Dr. Sukumar Ghosh Iowa University Distributed Systems course 22C:166