1. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Chapter 6 Synchronization (1) DISTRIBUTED SYSTEMS (dDist) 2014 With material from Ken Birman

2. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Plan Clock synchronization in distributed systems –Physical clocks –Logical clocks Ordered multicasting Mutual exclusion Election

3. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time We have been somewhat careful to avoid talking about time until now But, in distributed system we need practical ways to deal with time –E.g., we may need to agree that update A occurred ‘before’ update B –Or offer a “lease” on a resource that expires ‘at’ time 13:42.50 –Or guarantee that a time critical event will reach all interested parties ‘within’ 100ms

4. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 But what does Time mean? Time on a machine’s local clock –But was it set accurately? –And could it drift, e.g. run fast or slow? Time on a global clock –E.g., with GPS receiver –Still not accurate enough to determine which events happens before other events Or could try to agree on a time…

5. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Basic Approaches “Physical” time –Synchronize with external source External synchronization – We look at NTP –Synchronize local clocks within a group Internal synchronization – We look at the Berkeley Algorithm Logical time –Avoid absolute statements about time, just causally relate events –This is actually close to how mother nature handles time Nature has no global clock, only causality, and causality is restricted to the speed of light

6. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Physical Time (1/3) No global clock by general relativity! But atomic clocks are good substitutes on the surface of the earth International Atomic Time (IAT) –One second := 9,192,631,770 oscillations of a Cesium 133 atom Coordinated Universal Time (UTC) –IAT corrected with leap seconds because the earth is slowing down in its rotation (and we want the sun to be up at noon)

7. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Physical Time (2/3) Absolute time on a computer? –Atomic clocks are expensive! Can listen to WWV instead (external sync.) –Call sign for radio transmitter operated by NIST, which broadcasts UTC –Accuracy ~ 10 msec (10 -2 sec) There are other transmitters too

8. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Physical Time (3/3) Can also use the Global Positioning System (GPS) –The satellites continuously broadcasts their position and time From enough such samples and a little knowledge of math and physics one can compute ones position and UTC time! –Accuracy ~ 60 nsec (6 x 10 -8 sec) Still 180 instructions on a 3 GHz machine –External synchronization Most computes do not have a GPS receiver, or a good view to the sky

9. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Computer Clocks Typical computer has timer circuit for keeping tracks of time –Quartz crystal that oscillate at a well- defined frequency –Counter decremented at each oscillation –Holding register incremented when counter reaches zero –System call to read holding register Crystals oscillate at slightly different frequency  clock skew

10. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Clock Synchronization Algorithms System model: A clock server helps keep clients synchronized –Server might have physical time Timers work within a max drift rate –To guarantee that clocks do not drift more than seconds apart, synchronize at least every seconds: 1 skew offset clock time

11. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Cristian’s Algorithm (1/3) External synchronization algorithm Works under two assumptions: 1.Sender a messages A to B takes almost exactly the same time as sending a message B to A 2.The clock skew is so small that the offset changes very little during the time it takes to send two messages Server Client

12. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Cristian’s Algorithm (2/3) A’s offset relative to B:  = t B – t A  = T 2 - (T 1 + T reg ) = T 2 - T 1 - T reg  = T 3 - (T 4 - T res ) = T 3 - T 4 + T res (by assump. 2) 2  = T 2 - T 1 + T 3 - T 4 + (T res - T reg )  T 2 - T 1 + T 3 - T 4 (by assumption 1) Server Client

13. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Cristian’s Algorithm (3/3) t B – t A  ½(T 2 - T 1 + T 3 - T 4 ) If ½(T 2 -T 1 +T 3 -T 4 )>0 then speed up the clock of A If ½(T 2 -T 1 +T 3 -T 4 )<0 then slow down the clock of A Server Client

14. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Network Time Protocol The delay estimation,, is given by –Calculate and multiple (8) times –Choose for which is minimal Divide time servers into strata –External clock is stratum 0 –Stratum A server adjust their time according to stratum B server if B < A, then becomes stratum B+1 server Server Client

15. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 The Berkeley Algorithm (1/2) Time daemon/server is actively requesting current time difference from machines –Calculates average and requests machines to adjust clock Works if it is enough that the group agrees on a time –They might drift from UCT

16. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 The Berkeley Algorithm (2/2) Why do we send adjustment values and not just absolute times? Under which assumptions does this algorithm work?

17. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Logical Time Many distributed algorithms do not need physical time, just some well-defined ordering of events Say two persons are writing to a replicated web board at the same time Enough that all replicas of the web-board post messages in same order –And all answers after their questions! We just want to know if event A happened ”before” or ”after” event B

18. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Lamport’s Approach Leslie Lamport suggested that we should reduce time to its basics –Cannot order events according to a global clock None available… –Can use logical clock Time basically becomes a way of labeling events so that we may ask if event A happened before event B Answer should be consistent with what could have happened with respect to a global clock –Often this is what matters

19. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line Pictures (1/7) p m snd p (m) q rcv q (m) deliv q (m)

20. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line Pictures (2/7) A, B, C and D are “events” –Could be anything meaningful to the application like a file access So are snd p (m) and rcv q (m) and deliv q (m) What ordering claims are meaningful? p m A C B snd p (m) q rcv q (m) deliv q (m) D

21. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line Pictures (3/7) A, C, snd p (m), rcv q (m), B, deliv q (m), D ? C, A, snd p (m), rcv q (m), deliv q (m), B, D ? C, snd p (m), A, rcv q (m), deliv q (m), D, B ? A, C, rcv q (m), snd p (m), deliv q (m), B, D ? p m A C B snd p (m) q rcv q (m) deliv q (m) D

22. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line Pictures (4/7) A happens before B, and C before D –“Local ordering” at a single process –Write and p q m A C B rcv q (m) deliv q (m) snd p (m) D

23. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line Pictures (5/7) snd p (m) also happens before rcv q (m) –“Distributed ordering” introduced by a message –Write p q m A C B rcv q (m) deliv q (m) snd p (m) D

24. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line Pictures (6/7) A happens before D –Transitivity: A happens before snd p (m), which happens before rcv q (m), which happens before D p q m D A C B rcv q (m) deliv q (m) snd p (m)

25. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line Pictures (7/7) B and D are concurrent –Looks like B happens first, but D has no way to know. No information flowed… p q m D A C B rcv q (m) deliv q (m) snd p (m)

26. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 The Happens-Before Relation We’ll say that “A happens-before B”, written A  B, if –1) A  p B according to the local ordering, or –2) A is a snd and B is a rcv and A  M B, or –A and B are related under the transitive closure of rules 1 and 2 Thus, A  D, but neither B  D nor D  B So far, this is just a mathematical notation, not a “systems tool”

27. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Logical Clocks A simple tool that can capture parts of the happens-before relation First version: uses just a single integer –Designed for big (64-bit or more) counters –Each process p maintains a local counter C p, –A message m will carry a counter C m

28. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Rules for managing logical clocks When an event happens at a process p the process increments C p –Any event that matters to p –Normally, also snd and rcv events (since we want receive to occur “after” the matching send) When p sends m, it set –C m = C p When q receives m, set –C q = max(C q, C m )+1

29. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line with LT annotations C(A) = 1, C(snd p (m)) = 2, C(m) = 2, C(B)=3 C(C) = 1, C(rcv q (m)) = max(1,2)+1 = 3, C(deliver q (m)) = 4, … p q m D A C B rcv q (m) deliv q (m) snd p (m) CqCq 0001111333455 CpCp 0112222223333

30. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Logical Clocks vs. Happens-Before If A  B then C(A)<C(B) –If A  p B then C(B) = C(A) + 1 –If A  M B then C(B) = max(C(A), n)+1 ≥ C(A)+1 –If not A  p B or A  M B, then there exist C such that A  C  B; induction says C(A) < C(C) and C(C) < C(B), so C(A) < C(B)

31. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Logical Clocks vs. Happens-Before But converse might not be true: –If C(A)<C(B) can’t be sure that A  B –This is because processes that don’t communicate still assign timestamps and hence events will “seem” to have an order –For example –So C(C) < C(B) but we don’t have C  B (in fact it looks like C happens after B) C(A) = 1 C(B) = 2 C(C) = 1 C(D) = 2

32. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Totally Ordered Multicast (1/3) All multicast messages are delivered in the same order at all receivers Used, e.g., to do updates in the same order at all replicas Lamport clocks can be used to implement totally ordered multicast

33. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Totally Ordered Multicast (2/3) Update 1 –Add 100$ to account Update 2 –Add 1% interest to account Update 1; Update 2 –(1000$ + 100$)*1,01 = 1111$ Update 2; Update 1 –1000$*1,01 + 100$ = 1110$

34. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Totally Ordered Multicast (3/3) Implementation –Use a sequencer Get a number and use that as sequence number for multicast Receiving processes deliver in that order –Distributed Each process attaches a Lamport clock value to message When receiving a message, send acknowledge to everybody Order all messages and acknowledgements in a priority queue according to clock value (and process number) Deliver message at front when it has been acknowledged by all processes Queues eventually become identical and the waiting for ack’s ensure tha the front is identical when delivered

35. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Implementing happens-before Lamport clocks give a total order of all events What if we only are interested in delivering according to causality Concurrent messages might be delivered in different order at different sites If Q1 and Q2 are independent questions and A1 and A2 are the respective answers, then this is not a big problem: –Site1: Q1, Q2, A1, A2 –Site2: Q1, A1, Q2, A2 Might allow faster delivery

36. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Vector Clocks Clock is a vector: e.g. VC(A)=[1, 0] –We’ll just assign p index 0 and q index 1 –Again, each process has its own vector clock and messages are “time-stamped” Rules for managing vector clocks –When event happens at p, increment VC p [index p ] Normally, also increment for snd and rcv events –When sending a message, set VC(m)=VC p –When receiving, set VC q =max(VC q, VC(m)) Where “max” is max on components of vector

37. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line with VT annotations p q m D A C B rcv q (m) deliv q (m) snd p (m) VC q000 0101 0101 0101 0101222 2323 2323 2424 VC p0 1010 1010 2020 2020 2020 2020 2020 2020 3030 3030 3030 3030 VT(m)=[2,0] Could also be [1,0] if we decide not to increment the clock on a snd event. Decision depends on how the timestamps will be used.

38. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Rules for comparison of VCs We’ll say that VC A ≤ VC B if –  i, VC A [i] ≤ VC B [i] And we’ll say that VC A < VC B if –VC A ≤ VC B and VC A ≠ VC B –That is, for some i, VC A [i] < VC B [i] Examples: –[2,4] ≤ [2,4] –[1,3] < [7,3] –[1,3] is “incomparable” to [3,1]

39. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Time-line with VC annotations VC(A)=[1,0]. VC(D)=[2,4]. So VC(A)<VC(D) VC(B)=[3,0]. So VC(B) and VC(D) are concurrent p q m D A C B rcv q (m) deliv q (m) snd p (m) VC q000 0101 0101 0101 0101222 2323 2323 2424 VC p0 1010 1010 2020 2020 2020 2020 2020 2020 3030 3030 3030 3030 VC(m)=[2,0]

40. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Vector time vs. happens-before If A  B, then VC(A)<VC(B) –Proof as for Lamport clocks But we also have that if VC(A)<VC(B) then A  B We show that If ¬(A  B) then ¬(VC(A) < VC(B))

41. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Vector time vs. happens-before If ¬(A  B) then ¬(VC(A) < VC(B)) Proof: –Assume that A occurs at p and B at q We may have that p = q but not necessarily –Let’s say VC(A)[p] = k –There cannot be a sequence of events A=E 0  E 1  E 2  …  E n =B (otherwise A  B) –Thus we must have VC(B)[p] < k B only updates VC(B)[p] to k due to an incoming VC –Consequently ¬(VC(A) < VC(B))

42. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Enforcing Causal Communication Ensure: If snd(m)  snd(m*) then del(m)  del(m*) –Use vector timestamps –Increment when sending –Adjust when delivering Deliver a message m from i at j when: –m is the next message expected from i, i.e, ts(m)[i] = VC j [i] + 1 –All messages that i had seen from other processes have been seen by us, i.e., ts(m)[k] <= VC j [k], k  j

43. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Summary We cannot achieve global, synchronized time in a distributed system Physical time –Can synchronize (with high probability) within a small constant Logical time –Advance time only as consequence of logical middleware events –Can be used to implement, e.g., ordered multicast