1. CPSC 668Set 13: Clocks1 CPSC 668 Distributed Algorithms and Systems Fall 2009 Prof. Jennifer Welch

2. CPSC 668Set 13: Clocks2 Hardware Clocks Suppose processors have access to some approximation of real time. Mechanism is through hardware clocks, one at each processor. p i 's hardware clock HC i is modeled as a function from real times to clock times. Consider timed executions: associate a real time with each event (increasing). During p i 's computation event at real time t, the value of HC i (t) can be used as input to p i 's transition function.

3. CPSC 668Set 13: Clocks3 Possible H/W Clock Properties HC i is increasing –a minimal property HC i (t) = number of steps taken by p i through real time t –easy to implement in software HC i (t) = t –perfect HC i (t) = t + c i –h/w clock runs at same rate as real time but offset HC i (t) = a i t + b i –h/w clock drifts away from real time

4. CPSC 668Set 13: Clocks4 Adjusted Clocks Clocks are particularly useful if they are synchronized. But typically hardware clocks cannot be changed. Instead, consider adjusted clock, obtained by adding some value to the hardware clock value: AC i (t) = HC i (t) + adj i (t) adj i is adjustment variable of p i

5. CPSC 668Set 13: Clocks5 Measuring Clock Differences How to evaluate how close together clocks are? Skew: how far apart clock times are at a given real time, or Precision: how far apart in real time clocks reach same clock time These are the same when there is no drift…

6. CPSC 668Set 13: Clocks6 Skew and Precision real time clock time skew AC i AC j precision T t

7. CPSC 668Set 13: Clocks7 Synchronizing Clocks If hardware clocks don't drift, then once clocks are adjusted, they stay the same distance apart. Achieving  -synchronized clocks: Termination: no processor assigns to its adj variable after some real time t f  -bounded skew: for all i and j, and all real times t ≥ t f, |AC i (t) - AC j (t)| ≤ .

8. CPSC 668Set 13: Clocks8 Bounded Message Delays We'll study the clock synchronization problem in message passing with bounded delays. Define a timed execution to be admissible if: –every processor takes an infinite number of steps (no failures) –every message has delay in the range [d–u,d]; call u the uncertainty

9. CPSC 668Set 13: Clocks9 Two Processor Algorithm Consider this simple algorithm: p 0 uses its hardware clock as its adjusted clock p 1 adopts (its best estimate of) p 0 's adjusted clock as its adjusted clock How does p 1 do this? p 0 sends its clock time to p 1 in a message How to handle uncertain delay? Assume delay is in the middle of the range: d – u/2

10. CPSC 668Set 13: Clocks10 Code for Two Processor Algorithm p 0 : adj 0 := 0 send HC 0 to p 1 p 1 : when receive T from p 0 : adj 1 := (T + d – u/2) – HC 1

11. CPSC 668Set 13: Clocks11 Analysis of Two Proc. Algorithm What is the skew attained by the algorithm? If message really did take d – u/2 time to arrive, skew is 0 (best case). If message took d or d – u time, skew is u/2 (worst case). Can we do better, perhaps with a more complicated algorithm?

12. CPSC 668Set 13: Clocks12 Proving Lower Bounds on Skew A useful technique for proving lower bounds on skew for clock synchronization is that of shifting executions. To define it, we first need to look at some modeling issues.

13. CPSC 668Set 13: Clocks13 Modeling Executions: Two Ways We've been modeling an execution as a sequence of events. step by p 2 step by p 0 step by p 1

14. CPSC 668Set 13: Clocks14 Modeling Executions: Two Ways An alternative approach is to model with a set of sequences, one sequence per processor. p2p2 p0p0 p1p1

15. CPSC 668Set 13: Clocks15 Modeling Executions: Two Ways Having one sequence per processor is technically convenient for lower bound proofs Can convert back and forth between the two modeling styles

16. CPSC 668Set 13: Clocks16 Processor Views A view of processor p i is: –an initial state of p i –a sequence of events (computation and delivery) occurring at p i –a hardware clock value for each event A timed view of p i is a view with a real time associated with each event (increasing)

17. CPSC 668Set 13: Clocks17 Views vs. Timed Views Two different timed views with the same (untimed) view: 11:15 11:20 11:45 11:52 3:00 3:05 3:10 4:00 h/w clock times real times 8:08 9:00 9:10 10:10 3:00 3:05 3:10 4:00 h/w clock times real times

18. CPSC 668Set 13: Clocks18 Extracting Views from Executions Given a timed execution, straightforward to extract timed views for all the processors: –get initial state of a processor from the initial configuration –get sequence of events occurring at that processor and their times from the events in the execution

19. CPSC 668Set 13: Clocks19 Merging Views into an Execution Given a set of timed views, one per proc: 1.initial config is combination of initial states 2.obtain sequence of events by interleaving events from views in real- time order (break ties with ids) 3.apply events in order to initial config to obtain the other configs.

20. CPSC 668Set 13: Clocks20 But is Result Admissible? The result might not be admissible. Biggest issue is the message delays: must be in range d – u to d.

21. CPSC 668Set 13: Clocks21 Why Care About Views? To prove lower bounds on skew: 1.Start with a (carefully chosen) timed execution 2.Modify processors' views (in a carefully chosen way) 3.Merge resulting views to get a new execution: check that it is admissible show that it violates some bound

22. CPSC 668Set 13: Clocks22 Shifting Timed Executions Given timed execution  and real numbers x 0, x 1, …, x n-1, shift( ,(x 0, x 1, …, x n-1 )) is created by: 1.extracting timed views v 0, …, v n-1 from  2.adding x i to the real time of each event in each v i 3.merging the resulting timed views

23. CPSC 668Set 13: Clocks23 Shifting Examples h/w clock times real times HC i (t) = T t h/w clock times real times HC i (t+x) = T t + x h/w clock times real times HC i (t+x) = T t + x shift by positive amount shift by negative amount

24. CPSC 668Set 13: Clocks24 Facts About Shifted Executions Result of shifting and merging might not be admissible: could shift receipt of a message earlier than its sending, for example. But these facts hold: 1New hardware clock HC' i satisfies: HC' i (t) = HC i (t – x i ) = HC i (t) – x i 2Delay of a msg from p i to p j goes from  to  – x i + x j since msg is sent x i later and received x j later

25. CPSC 668Set 13: Clocks25 Lower Bound for 2 Processors Let A be any 2-proc. alg that achieves  - clock synchronization. Let  be the timed admissible execution of A in which –every msg from p 0 to p 1 has delay d – u –every msg from p 1 to p 0 has delay d After A terminates in , (1) AC 0 ≥ AC 1 – 

26. CPSC 668Set 13: Clocks26 Lower Bound for 2 Processors p0p0 p1p1 d-u d p0p0 p1p1 d shift p 0 backwards by u

27. CPSC 668Set 13: Clocks27 Lower Bound for 2 Processors Let  ' = shift( ,(–u,0)). Shift p 0 earlier by u, leave p 1 alone. In  ', –every msg from p 0 to p 1 has delay d –every msg from p 1 to p 0 has delay d – u After A terminates in  ', AC' 1 ≥ AC' 0 – 

28. CPSC 668Set 13: Clocks28 Lower Bound for 2 Processors AC' 1 ≥ AC' 0 –  implies AC 1 ≥ (AC 0 + u) –  since AC' 1 = AC 1 and AC' 0 = AC 0 + u Remember inequality (1): AC 0 ≥ AC 1 –  ≥ (AC 0 + u –  ) –  (from just above) Implies  ≥ u/2

29. CPSC 668Set 13: Clocks29 Star Algorithm for n Processors Assume the network topology is a clique and message delay range for every edge is d – u to d. Pick one proc (say p 0 ) and let every other proc try to adopt p 0 's clock using the 2-processor algorithm. Worst-case skew can be as large as u (one proc is u/2 behind p 0 's clock and another is u/2 ahead)

30. CPSC 668Set 13: Clocks30 Improved Algorithm for n Processors All processors exchange h/w clock values. Each processor estimates the difference between its own h/w clock and that of each other processor. Each processor computes the average of the differences and sets its adj variable to the result

31. CPSC 668Set 13: Clocks31 Code for Processor p i initially diff[i] = 0 send HC i to all procs when receive T from p j : diff i [j] := (T + d – u/2) – HC i when heard from all procs: adj i := (1/n) ∑ diff i [k] k = 0 n-1

32. CPSC 668Set 13: Clocks32 Analysis of n-Processor Algorithm To bound the skew, start with |AC i – AC j | Then substitute the formula for each AC from the code: HC i + (1/n) ∑ diff i [k] Then do some algebra (rearranging terms and using properties of absolute value) to get…

33. CPSC 668Set 13: Clocks33 Analysis of n-Processor Algorithm |AC i – AC j | ≤ (X + Y + Z)/n where X = |diff j [i] – (HC i – HC j )| error in p j 's estimate of the difference between its own clock and p i 's clock, at most u/2 Y = |diff i [j] – (HC j – HC i )| error in p i 's estimate of the difference between its own clock and p j 's clock, at most u/2 Z = sum over all k other than i and j of |diff i [k] – (HC k – HC i )| + |diff j [k] – (HC k – HC j )| error in p i 's estimate of p k 's clock plus error in p j 's estimate of p k 's clock, at most u/2 + u/2 = u.

34. CPSC 668Set 13: Clocks34 Analysis of n-Processor Algorithm To finish up, |AC i – AC j | ≤ (u/2 + u/2 + (n–2)u)/n = u(1 – 1/n).

35. CPSC 668Set 13: Clocks35 Lower Bound for n-Processor CS Theorem (6.17): No algorithm can achieve  - synchronized clocks for  < u(1–1/n). Proof: Choose any algorithm A that achieves  -synchronized clocks. Let  be a timed admissible exec. s.t. –every msg from p i to p j has delay d – u, i < j. –every msg from p j to p i has delay d, i < j.

36. CPSC 668Set 13: Clocks36 Example of Reference Execution For n = 4, the message delays in  can be represented schematically like this: p0p0 p1p1 p2p2 p3p3 d-u d d d d d d

37. CPSC 668Set 13: Clocks37 Additive Lemma Lemma (6.18): AC k-1 ≤ AC k – u + , for all k. Proof: Take  and shift p 0 through p k-1 earlier by u:  ' = shift( ,(–u,…, –u,0,…,0)) Verify that  ' is admissible by checking that message delays are in range: –if sender and recipient were both shifted, then delays are same as in  –if one is shifted and other is not, then delays that used to be d–u become d and delays that used to be d become d–u.

38. CPSC 668Set 13: Clocks38 Example of Shifted Execution p0p0 p1p1 p2p2 p3p3 d-u d d d d d d p0p0 p1p1 p2p2 p3p3 d d d d d d shift p 0 and p 1 earlier by u

39. CPSC 668Set 13: Clocks39 Additive Lemma Completed Since  ' is admissible and algorithm achieves  -synchronized clocks, after termination AC k-1 ' ≤ AC k ' +  By shifting facts, AC k-1 ' = AC k-1 + u and AC k ' = AC k Thus AC k-1 ≤ AC k – u + .

40. CPSC 668Set 13: Clocks40 Back to Main Lower Bound Proof After termination in  : AC n-1 ≤ AC 0 +  by correctness of algorithm ≤ AC 1 – u + 2  by Additive Lemma ≤ AC 2 – 2u + 3  by Additive Lemma … ≤ AC n-1 – (n–1)u + n  by Additive Lemma Thus  ≥ u(1 – 1/n).

41. CPSC 668Set 13: Clocks41 Message Delays in the Real World In reality, message delays are not uniformly distributed between a minimum and a maximum. Typically the distribution has a spike close to the minimum and a long tail going to infinity. One approach to deal with the lack of a maximum is to fix a "timeout" value d and consider any msg taking longer to be lost. But if d is chosen to be fairly large (to reduce the number of slow msgs incorrectly classified as lost), most msgs will take significantly less than d, and even significantly less than d – u/2.

42. CPSC 668Set 13: Clocks42 Estimating Clock Differences Take advantage of small delays that occur most of the time. p i sends a query to p j, which p j answers immediately with its current clock value. When p i gets the response, it assumes p j 's response took half the round trip time. If the round trip time is small, error is reduced compared to original approach. p i can query repeatedly until getting a round trip time that is "sufficiently" small.

43. CPSC 668Set 13: Clocks43 Clock Drift Hardware clocks typically suffer from drift (gain or lose time). Usually the drift is bounded, though. Bounded Drift: There exists  > 0 such that for all i, and all real times t 1 and t 2, (1 +  ) –1 (t 2 – t 1 ) ≤ HC i (t 2 ) – HC i (t 1 ) ≤ (1 +  )(t 2 – t 1 ) That is, hardware clocks measure elapsed real time approximately correctly.

44. CPSC 668Set 13: Clocks44 Hardware Clock Drift For quartz crystal clocks,  is about 10 – 6 hardware clock HC i real time t HC i (t) max slope < 1+  1+  min slope < (1+  ) -1 (1+  ) -1

45. CPSC 668Set 13: Clocks45 Clock Synchronization with Drift When clocks can drift, processors must continually resynchronize. Two problems: 1.Establish: Get clocks close together. 2.Maintain: Keep clocks close together. We will focus on the maintenance problem, assuming clocks are initially within some B of each other.

46. CPSC 668Set 13: Clocks46 Maintaining Clock Synchronization with Drift Clock Agreement: There exists  s.t. for all i and j, and all real times t: |AC i (t) – AC j (t)| ≤  Clock Validity: There exists  > 0 s.t. for all i and all real times t: (1 +  ) –1 (HC i (t) – HC i (0)) ≤ AC i (t) – AC i (0) ≤ (1 +  )(HC i (t) – HC i (0)) When taking the "long view", adjusted clocks measure elapsed time approximately as well as the hardware clocks.

47. CPSC 668Set 13: Clocks47 Byzantine Failures and Clock Synchronization Suppose up to f processors can exhibit Byzantine failures. Modify definition of maintaining clock synchronization with drift so that clock agreement and clock validity only need to hold for nonfaulty processors. To solve the problem, total number of processors n must satisfy n > 3f.

48. CPSC 668Set 13: Clocks48 Lower Bound on Number of Processors The n > 3f condition is also true of consensus. The consensus problem and the clock maintenance problem are similar. Can we use the n > 3f bound for consensus via a reduction? No one knows how. Instead, we'll do a direct proof, but using familiar ideas –scaling (similar to shifting) –specify faulty behavior with a big ring

49. CPSC 668Set 13: Clocks49 Scaling Clocks Given a timed execution  and a real number s > 0, scale( ,s) is the result of multiplying every real time in  by s. If s > 1, scaling causes clocks to slow down and delays to increase. If s < 1, scaling causes clocks to speed up and delays to decrease.

50. CPSC 668Set 13: Clocks50 Scaling Example real time HC 0 (t) = 3t HC 1 (t) = 4t HC' 0 (t) = (3/2)t HC' 0 (t) = 2t scale by s = 2 2:00 3:00 4:00 6:00 6:00 12:00 6:00 12:00 delay = 1:00 delay = 2:00 p0p0 p1p1 p0p0 p1p1

51. CPSC 668Set 13: Clocks51 Scaling Clocks Lemma (13.1): In  ' = scale( ,s), HC i '(t) = HC i (t/s) AC i '(t) = AC i (t/s) if a msg has delay  in , then it has delay s  in  '. Lemma (13.2): If  satisfies  -clock agreement and  -clock validity for a set of procs, then so does scale( ,s).

52. CPSC 668Set 13: Clocks52 Processor Lower Bound for CS Assume f = 1 –extend to larger f with reduction u ≥ d(1 – (1 +  ) –4 ) –needed for calculations to work out –since  is tiny, this is not a significant restriction (uncertainty must be at least slightly larger than 0)

53. CPSC 668Set 13: Clocks53 Processor Lower Bound for CS Assume in contradiction there is an algorithm (A,B,C) for n = 3 and f = 1 that achieves  -clock agreement and  -clock validity. Consider a ring of k processors, where –k is a multiple of 3 –(1 +  ) 2(k-1) > (1 +  ) 2 needed for the calculations to work out

54. CPSC 668Set 13: Clocks54 Big Ring p0p0 p1p1 p2p2 p3p3 p i-1 pipi p i+1 p k-1 A B C A A B C C

55. CPSC 668Set 13: Clocks55 Execution  on Big Ring p0p0 p1p1 p2p2 p3p3 p i-1 pipi p i+1 p k-1 A B C A A B C C t(1+  ) t(1+  ) 1-2(i-1) t(1+  ) 1-2i t(1+  ) 1-2(i+1) t(1+  ) 1-2(k-1) t(1+  ) -1 t(1+  ) -3 t(1+  ) -5 d(1+  ) 0 = d hardware clocks message delays d(1+  ) 2 d(1+  ) -2 d(1+  ) -4 d(1+  ) 2i-2 d(1+  ) 2i-4 d(1+  ) 2k-6 local algorithms and adj. vars are initially 0

56. CPSC 668Set 13: Clocks56 Execution on Big Ring We cannot rely on  satisfying the clock synch properties: –more than 3 processors –some h/w clock drift rates are out of range –some message delays are out of range However, we can make some deductions about how processors behave in  : –show that pieces of the ring "look like" certain systems in which the algorithm is supposed to be correct.

57. CPSC 668Set 13: Clocks57 Behavior in Big Ring Lemma (13.4): In , for all t: a)|AC i (t) - AC i+1 (t)| ≤  b)(1+  ) -1 HC i (t) ≤ AC i (t) ≤ (1+  )(HC i (t)) Proof: Take p i and p i+1 from big ring and put them in a triangle in which 3rd processor is faulty and acts like the rest of the big ring. Call this execution .

58. CPSC 668Set 13: Clocks58 Triangle Based on Big Ring pipi p i+1 t(1+  ) 1-2i t(1+  ) 1-2(i+1) d(1+  ) 2(i+1)-4 d(1+  ) 2i-4 d(1+  ) 2(i+2)-4 acts like p i-1 toward p i in  acts like p i+2 toward p i+1 in 

59. CPSC 668Set 13: Clocks59 Relationship of Triangle and Ring Claim: p i and p i+1 behave the same in  (the execution on the triangle with the Byzantine processor) as they do in  (the execution on the big ring).

60. CPSC 668Set 13: Clocks60 Scaled Triangle Scale  by (1 +  ) -2i to get  ' : pipi p i+1 t(1+  ) t(1+  ) -1 d(1+  ) -2 d(1+  ) -4 d acts like p i-1 toward p i in  acts like p i+2 toward p i+1 in  ≥ d - u by assump.

61. CPSC 668Set 13: Clocks61 Relating the Three Executions Since  ' is admissible, it satisfies  - clock agreement and  -clock validity for p i and p i+1. By Scaling Lemma (13.2),  also satisfies those conditions for p i and p i+1. Since  and  look the same to p i and p i+1,  also satisfies those conditions for p i and p i+1.

62. CPSC 668Set 13: Clocks62 Finishing the Main Lower Bound Referring back to , AC 0 (t) ≤ AC 1 (t) +  by Lemma 13.4(a) ≤ AC 2 (t) + 2  by Lemma 13.4(a) … ≤ AC k-1 (t) + (k-1)  by Lemma 13.4(a) So AC k-1 (t) ≥ AC 0 (t) - (k-1)  ≥ (1+  ) -1 HC 0 (t) - (k-1)  by Lemma 13.4(b) = (1+  ) -1 (1+  ) 2(k-1) HC k-1 (t) - (k-1) 

63. CPSC 668Set 13: Clocks63 Finishing the Main Lower Bound From previous slide: AC k-1 (t) ≥ (1+  ) -1 (1+  ) 2(k-1) HC k-1 (t) - (k-1)  By Lemma 13.4(b): AC k-1 (t) ≤ (1+  )HC k-1 (t) Combining and rearranging gives: HC k-1 (t) [(1+  ) -1 (1+  ) 2(k-1) - (1+  )] ≤ (k-1)  grows w/o bound positive, by assumption about k constant

64. CPSC 668Set 13: Clocks64 Fault-Tolerant Clock Synchronization Algorithms Continue to focus on maintenance algorithms. Assume clocks are initially close together –different algorithms state this condition differently Processors resynchronize every P time units: –different algorithms have different constraints on P.

65. CPSC 668Set 13: Clocks65 A Fault-Tolerant CS Algorithm [ Welch & Lynch, 1988] Assume adjusted clocks reach clock time 0 within B real time of each other Resynch every P time units; choose P –large enough to avoid confusion between resynchronizations –small enough to prevent skew due to drift from becoming too large

66. CPSC 668Set 13: Clocks66 Code for a Processor when AC = kP (k = 1, 2, …): send AC to all set timer for (1 +  )(B + d) in the future when receive T msg from p j : diff[j] := (T + d – u/2) – AC when timer goes off: adj := adj + midpoint(trim(f,diff)) clear diff array discard f largest and f lowest values

67. CPSC 668Set 13: Clocks67 Explanation of Timer Value Why wait (1 +  )(B + d) time to collect messages? Want to hear from all nonfaulty processors before adjusting. –All nonfaulty procs will reach clock time kP within B time of each other (true for k = 0 by assumption, shown by induction for k > 0) –Maximum msg delay is d –Waiting B + d clock time might not be long enough if your clock is fast. To be safe, wait extra factor of (1 +  )

68. CPSC 668Set 13: Clocks68 Clock Agreement Claim: Nonfaulty clocks reach each kP within B real time of each other. –Proved by induction. Claim: After adjusting their clocks in each resynch period, the new (nonfaulty) clocks reach kP within real time B/2 + u + O(  ) of each other. See figure. –Proved using properties of the trim and midpoint functions: difference is roughly halved.

69. CPSC 668Set 13: Clocks69 Figure for Resynchronization real time kP kP+(B+d)(1+  ) (k+1)P (k+1)P+(B+d)(1+  ) at most B at most B/2 + u + O(  ) AC i AC j

70. CPSC 668Set 13: Clocks70 Clock Agreement Due to drift, new clocks reach (k+1)P (start of next resynch) within real time B/2 + u + 2  P of each other. B/2 + u + 2  P ≤ B implies B ≥ 2u + 4  P = 2u + O(  ) So B cannot be any smaller than 2u plus terms of order .

71. CPSC 668Set 13: Clocks71 Clock Agreement Claim: The algorithm achieves  -clock agreement, where  = B + u/2 + O(  ) Using the smallest possible B, the best this algorithm gives is  = 5u/2 + O(  ).

72. CPSC 668Set 13: Clocks72 Clock Validity Paper analyzes drift of adjusted clocks with respect to real time, not hardware clock time. Adjusted clock drift rate is calculated to be  + O(1/P), as opposed to  for the hardware clocks. –The more frequently the processors resynchronize, the more they degrade the drift rate (tradeoff with Clock Agreement) Careful analysis for the version of clock validity given in textbook is open.