1. CS 347Notes 121 CS 347: Parallel and Distributed Data Management Notes12: Time and Clocks Hector Garcia-Molina

2. CS 347Notes 122 Reference: Leslie Lamport, Time, clocks, and the ordering of events in a distributed system, Communications of the ACM, Volume 21, Issue 7 (July 1978), pages: 558 - 565. Available at: http://portal.acm.org/citation.cfm?id=359563

3. CS 347Notes 123 Time and clocks Time and clocks are fundamental concepts in distributed systems e.g.:timeouts identifying calls, transactions,… setting priorities versions of data...

4. CS 347Notes 124 What is time? How can clocks be implemented? Questions

5. CS 347Notes 125 Ordering of events More basic than time: event ordering event ahappened event b 9 am before 10 am Do not need physical time to order events event ahappened before event b can affect

6. CS 347Notes 126 Ordering of events “  ” is a partial ordering [ total ordering: for any two events a,b (a  b) either a  b or b  a partial ordering: a,b can be concurrent ]

7. CS 347Notes 127 The model ProcessProcessProcess P Q R P4P4 P3P3 P2P2 P1P1 R4R4 R3R3 R2R2 R1R1 Q7Q7 Q6Q6 Q3Q3 Q2Q2 Q5Q5 Q1Q1 Q4Q4 TIMETIME

8. CS 347Notes 128 Events within a process are totally ordered Sending or receiving a message is an event No assumptions about message transmission times E.g.: P 1  P 2 P 1  Q 2 P 3, Q 3 concurrent

9. CS 347Notes 129 Definition The relation  on the set of events of a system is the smallest relation satisfying the following 3 conditions: 1) if a,b events in same process, and a comes before b, then a  b 2) if a is sending a message and b is receipt of same message by a different process, then a  b 3) if a  b and b  c, than a  c assume a  a if a  b and b  a, than a,b are concurrent

10. CS 347Notes 1210 Logical Clocks ProcessProcess Process P QR P4P4 P3P3 P2P2 P1P1 R4R4 R3R3 R2R2 R1R1 Q7Q7 Q6Q6 Q3Q3 Q2Q2 Q5Q5 Q1Q1 Q4Q4 TIMETIME C R =9 C R =8 C R =7 C R =6 C R =5

11. Overview Logical Clocks Physical Clocks –basic properties –synchronization scheme –synchronization with clock server –probabilistic synchronization CS 347Notes 1211

12. CS 347Notes 1212 Logical Clock C i =logical clock (counter) at process i C[b] = reading of C j when event b occurs at process j Clock condition for any events a,b IF a  b THEN C[a] < C[b] No relationship to physical time

13. CS 347Notes 1213 Clock Condition If a and b are events in process i and a comes before b, then C i [a] < C i [b] If a is the sending of a message by process i and b is the receipt of that message by process j, then C i [a] < C j [b] C1C1 C2C2 If a  b THEN C[a] < C[b] can be satisfied if the following conditions hold:

14. CS 347Notes 1214 ProcessProcess Process P QR P4P4 P3P3 P2P2 P1P1 R4R4 R3R3 R2R2 R1R1 Q7Q7 Q6Q6 Q3Q3 Q2Q2 Q5Q5 Q1Q1 Q4Q4 TIMETIME C R =7 C R =5 C Q =10 7>5 by C1 10>7 by C2

15. CS 347Notes 1215 To implement C1, C2: IR1 Each process i increments C i between any two successive events IR2 - Let a be event “process i sends message to j” - Message contains timestamp T m = C i [a] - When message arrives at j (1) IF T m > C j THEN C j  T m (2) event “arrival of message” takes place

16. CS 347Notes 1216 Note: C i [a] < C j [b]  a  b Note: a,b concurrent  C i [a] = C j [b] Note: C i [a] = C j [b]  a,b concurrent

17. CS 347Notes 1217 Ordering Events Totally (breaking ties) Example: A server wants to execute requests in order a is event that originated one request (at client) b is event that originates second request (at other client) If C[a] < C[b], service a first (it could be that a  b) If C[a] = C[b], pick one (a,b concurrent) e.g., pick one with lower [ node#, process id ]

18. CS 347Notes 1218 Total Ordering Let “<” be a total ordering of the processes Define total ordering of events “  ” a  b (a event in process i; b in j) if and only if (1) C i [a] < C j [b] or (2) C i [a] = C j [b] and i < j “  ” not unique

19. CS 347Notes 1219 Anomalous behavior C=100C=50 Reserve a TelephoneReserve a seat call seat server Client AClient B 1 2 3

20. CS 347Notes 1220 Anomalous behavior C=100C=50 Reserve a TelephoneReserve a seat call seat server Client AClient B 1 2 3 C= 9am

21. CS 347Notes 1221 Solutions (1) Include “telephone call” in “system” (2) Use perfect physical clocks (3) Use real physical clocks - may be “slightly off” - does not eliminate anomaly, but reduces its likelihood - useful for fault detection

22. CS 347Notes 1222 Physical Clocks C i (t) = clock reading at process i at physical time t time=t C i (t)

23. CS 347Notes 1223 time=t C i (t) C i + (x) C i - (x) C i - (x)=lim C i (x - |  |) C i + (x)=lim C i (x + |  |) 00 x Assume Ci(t) is a continuous, differentiable function except when clock is reset (message arrived) 00

24. CS 347Notes 1224 How do we enforce clock condition? If a  b then C(a) < C(b)

25. CS 347Notes 1225 Each process i increments C i between any two successive events For each process i, if i does not receive a message at time t, then C i is differentiable at t and dCi(t) dt IR1 IR1’ Logical clocks > 0 process i a b TIMETIME

26. CS 347Notes 1226 Let a be event “process i sends message m to j” m contains timestamp T m = C i [a] when m arrives at j (1) IF T m > C j THEN C j  T m (2) event “arrival of m” takes place IR2 Logical clocks

27. CS 347Notes 1227 Let a be event “process i sends message m to j” at physical time t m contains timestamp T m = C i (t) Let  m be minimum transmission delay for m m arrives at process j at physical time t’ t’ > t +  m IF Tm +  m > C j - (t’) THEN C j +  T m +  m After clock adjust, event “arrival of m” takes place IR2’ optional

28. CS 347Notes 1228 i j t t’ m mm IF Tm +  m > C j - (t’) THEN C j +  T m +  m TIMETIME

29. CS 347Notes 1229 Additional properties Clock drift There exists a contstant k <<1 such that for all processes i dCi(t) dt - 1 < k PC1

30. CS 347Notes 1230 For typical crystal controlled clocks, k < 10 -6 time CiCi xx+1 y y+1 k k y+1+k y+1-k dCi(t) dt - 1 < k

31. CS 347Notes 1231 Clock synchronization for all i,j:| Ci(t) - Cj(t) | <  PC2

32. CS 347Notes 1232 can be satisfied if a message is sent on every link at least every  seconds PC2 diameter = d d=4 (not 3)

33. CS 347Notes 1233  let  m be the transmission time of message m  m = minimum delay + unpredictable delay  m =  m + Z m Assume constant  such that for all m: Z m   Assume that for all m:  m =  t 1 t2t2 maximum transit time:  +  shortest transit time: 

34. CS 347Notes 1234 Theorem PC2 holds for all t > t o +  d with   d(2k  +  ) (assuming  +  <<  ).

35. CS 347Notes 1235 Argument for d=1 slow fast sf Worst case scenario: t 1 : m leaves f for s with T m = C f (t 1 ) t 2 : m arrives at s: C s (t 2 ) = C f (t 1 ) +  C f (t 2 ) - C s (t 2 ) is as large as possible,  t 2 +  : no communication for  sec; f is as fast as possible, s as slow as possible. Just before communication at t 2 +  clocks are as far apart as possible, 

36. CS 347Notes 1236 t 2 +  t2t2 f s kk kk    time clock reading So,  = 2  +  Now just need largest possible 

37. CS 347Notes 1237 C f (t 1 ) t 1 C s (t 2 ) t 2 f s ++ C f (t 2 ) = C f (t 1 ) + (1+k)(  +  ) C s (t 2 ) = C f (t 1 ) +   = (1+k)(  +  ) -  max  = k(  +  ) + 

38. CS 347Notes 1238  =2k  +  max  = k(  +  ) +  Thus:  = 2  +  + k(  +  ) Assuming that k<<1 and (  +  ) <<  we get:   2  + 

39. CS 347Notes 1239 Summary Physical clocks: clock condition a  b  C(a) < C(b) drift < k | Ci(t) - Cj(t) | <  can be implemented as discrete

40. CS 347Notes 1240 Uses for Physical Clocks: (1) To order events (2) Timeouts Example: “Reply to my request by time t 1 ” At time t 2 = (  +  +  )( 1+  ) + t 1 we can timeout my clock may drift

41. CS 347Notes 1241 my node other t 1 on my clock  max trans  +  t 1 at remote clock t 2 = (  +  +  )( 1+  ) + t 1

42. CS 347Notes 1242 Do physical clocks rule out anomalous behavior? No anomalous behavior if C s (t 2 )>C f (t 1 )  min f fast clock s slow clock message out of system C f (t 1 ) C s (t 2 ) C s (t 1 )=C f (t 1 ) - 

43. CS 347Notes 1243 Worst possible scenario: Smallest possible transmission time  min C s, C f as far apart as possible C s (t 1 ) = C f (t 1 ) -  Smallest possible C s (t 2 ) = C f (t 1 ) -  + S.P.increment = C f (t 1 ) -  +  min (1-  ) No anomalous behavior if C s (t 2 )>C f (t 1 )

44. CS 347Notes 1244 No A.B. if C s (t 2 ) > C f (t 1 ) I.e., if C f (t 1 ) -  +  min (1-  ) > C f (t 1 )  1 - k  min >

45. CS 347Notes 1245 Using a clock service With IR2’, a fast clock speeds up all clocks Use instead a single, reliable clock service (eg. WWV), assume it is perfect “real time” (k=0) IR2’ is now: C j + (t ’ )  T m +  min whenever timing message arrives If max. transmission time from central clock is <  +  then  k    (1 + k  (Why? )

46. CS 347Notes 1246 Using a clock service - analysis 0  error at synchronization time, node j   node j server Cj=+Cj=+ Cj=+Cj=+ ++++ ++  time slowest clock message fastest clock message

47. CS 347Notes 1247 Using a clock service - analysis - cont.   elapsed time  server time 0  Node that got fastest message Node that got slowest message time  possible difference slow clock fast clock

48. CS 347Notes 1248 Why can period be   ? ++  t t +  server j   fast slow

49. CS 347Notes 1249   elapsed time  server time 0  Node that got fastest message Node that got slowest time    k(   kk worst case for clock reset

50. CS 347Notes 1250  server time 0  Node that got fastest message Node that got slowest time  k(   kk worst case for clock reset |C s (t) - C j (t)|    |C k (t) - C j (t)|  2  

51. CS 347Notes 1251 Setting clocks back C j (t) time timing msg arrives cannot use other solutions?

52. CS 347Notes 1252 How can we reset a clock after a failure?

53. CS 347Notes 1253 Probabilistic clock synchronization |C j (t) - C k (t)|  2k   k  CsCs k = drift  =update period  =max. transmission variability

54. CS 347Notes 1254 Probabilistic clock synchronization |C j (t) - C k (t)|  2k   k  CsCs k = drift  =update period  =max. transmission variability   % of messages transmission time

55. CS 347Notes 1255 Idea #1 When site j wants to synchronize, it requests time from server T 2  v C j  T+  +v/2 server time j assume k =0

56. CS 347Notes 1256 Idea #1 When site j wants to synchronize, it requests time from server T 2  v C j  T+  +v/2 server time j  T+  +v T assume k =0

57. CS 347Notes 1257 Idea #1 When site j wants to synchronize, it requests time from server T 2  v C j  T+  +v/2 server time j  T+  +v T T+  T assume k =0

58. CS 347Notes 1258 Idea #1 When site j wants to synchronize, it requests time from server T 2  v C j  T+  +v/2 server time j  T+  +v T T+  T |C s (t) - C j (t)|  v/2 assume k =0

59. CS 347Notes 1259 If we are lucky, v will be small and we get tight synchronization If we are unlucky, v is large and we lose!! |C s (t) - C j (t)|  v/2

60. CS 347Notes 1260 Idea #2 (a) pick desired desired bound, D (b) repeat request until v/2  D (c) if more than q requests, we die!

61. CS 347Notes 1261 area=p 22 2  D p.d.f of round trip delay Prob. 1 request fails = Pr[roundtrip time > 2   D]= p Prob. q requests fail = p q Prob. We can synchronize = 1 - p q  Choose q so that it is “very likely” that algorithm works...

62. CS 347Notes 1262 If k = 0, we are done –step 1: all nodes synchronize –step 2: we stay synchronized forever If k > 0, synchronize every  seconds homework…..

63. Excecise Consider the clock synchronization scheme described in Slides 54-58 (Notes 10), where a server site has an accurate clock. Assume that communications are asymetric, so that messages TO the server are "slower": –Largest minimum delay is  –Largest unpredictable delay is  2 messages FROM the server are "faster": –Largest minimum delay is  –Largest unpredictable delay is  1 and  1 <  2. Continue to assume that   is zero. At time t1 a node j sends a synchronization request to the server. At time t2 = t1 + v + , node j receives its reply CS 347Notes 1263

64. (1) When j gets the reply from the server at time t2 (containing server time T), to what value should it set its local clock? Answer: Cj(t2) <- T + v/2 +  (2) Right after the local clock is updated at j, how far apart can the local and server clocks be? That is, what is the largest possible value of | Cs(t2) - Cj(t2) | ? Answer: v/2 CS 347Notes 1264