1. Distributed Operating Systems CS551 Colorado State University at Lockheed-Martin Lecture 7 -- Spring 2001

2. 28 March 2001CS-551, Lecture 72 CS551: Lecture 7 n Topics – Distributed Synchronization (Chapter 10) n Global Time n Physical Clocks n Network Time Protocol (NTP) n Logical Clocks

3. 28 March 2001CS-551, Lecture 73 Global Clocks n Distributed systems have no global clock n Each processor in the system is autonomous n Each processor has its own clock n Impossible to have the processes across the system synchronized exactly n Cannot know the true time order of any two events

4. 28 March 2001CS-551, Lecture 74 Ordering of Events n It is impossible to know which of two events happens first n This has an impact on scheduling n This makes the distributed system harder to debug

5. 28 March 2001CS-551, Lecture 75 Global Time via Shared Memory? n Distributed systems have no shared memory n Thus it is hard (impossible) to get an up-to- date state of the entire system n A global state would give us – A view of all local states – The contents of all messages currently in transit

6. 28 March 2001CS-551, Lecture 76 Definitions n Drifting: “the gradual misalignment of once synchronized clocks caused by the slight inaccuracies of the time-keeping mechanisms” (Galli) n Clock Skew: “the difference in time between two clocks due to drifting” (Galli)

7. 28 March 2001CS-551, Lecture 77 Global Time via Physical Clocks? n Universal Time Coordinator (UTC) – A consensus – Accurate to +/- 1 second per 20,000,000 years n about 1 part in 10^12 – Sources: n Geostationary Operational Environmental Satellites (GEOS) – GPS devices n WWV: a Fort Collins radio station

8. 28 March 2001CS-551, Lecture 78 Global Time via WWV A Fort Collins shortwave radio station n Transmits UTC signal n Low-frequency => less atmospheric disturbance n 2000 mile radius n Sends signals once a day to clocks/watches n Transmission delay is 24000 microseconds at the extreme range – Less than 0.1 second – Can be corrected for

9. 28 March 2001CS-551, Lecture 79 UTC Time Providers n Time Provider: “a commercial device that is capable of directly receiving information from a UTC server and making appropriate adjustments due to communication delays” (Galli) n Such devices are currently installed in watches, clocks, and computers

10. 28 March 2001CS-551, Lecture 710 Figure 10.1 Network Delays when Communicating Time. (Galli,p.239)

11. 28 March 2001CS-551, Lecture 711 Correcting for Transmitted Time n A UTC signal is sent out n Transmit time varies depending on – Atmospheric conditions – Humidity n Receiving clock must make compensation for transmit time n However, once reset, clock will start drifting again

12. 28 March 2001CS-551, Lecture 712 Figure 10.2 Forward Adjustment of a Clock. (Galli,p.240)

13. 28 March 2001CS-551, Lecture 713 Physical Time Services n Centralized – Broadcast-based n UTC (previous slides) n Berkeley Unix algorithm – Request-driven n Distributed n Notice: clocks cannot be moved backward

14. 28 March 2001CS-551, Lecture 714 Berkeley Unix Algorithm n Gusella & Zatti (1989) n Synchronizes clocks for processors running Berkeley Unix 4.3 n Does not require UTC n Centralized server broadcasts time periodically

15. 28 March 2001CS-551, Lecture 715 Figure 10.3 The Berkeley Algorithm for Physical Clock Synchronization. (Galli,p.241)

16. 28 March 2001CS-551, Lecture 716 Figure 10.4 Request-Driven Physical Clock Synchronization. (Galli,p.242)

17. 28 March 2001CS-551, Lecture 717 Distributed Physical Time Services n Each processor broadcasts its current time at regular intervals n Then starts a timer n Timestamps each response n Does so until timer runs out n Then adjusts its own time accordingly

18. 28 March 2001CS-551, Lecture 718 Figure 10.5 Fault-Tolerant Threshold Method. (Galli,p.243)

19. 28 March 2001CS-551, Lecture 719 Figure 10.6 Discard m Highest and Lowest Values. (Galli,p.243)

20. 28 March 2001CS-551, Lecture 720 Network Time Protocol (NTP) n Currently version 3 n RFC 1305 (request for comments) n Design goals – Allow accurate UTC synchronization – Enable survival despite significant losses of connectivity – Allow frequent resynchronization – Protect against malicious or accidental interference

21. 28 March 2001CS-551, Lecture 721 Request-Driven Time Service n Cristian (1989) n Uses a threshold to remove bad times caused by slow/faulty message transmission n Threshold matched against difference of times in current processor and received from server n Considers transmit time and interrupt time

22. 28 March 2001CS-551, Lecture 722 Figure 10.7 Strata in the NTP Architecture. (Galli,p.245)

23. 28 March 2001CS-551, Lecture 723 Simple Network Time Protocol (SNTP) n An adaptation of NTP n Simplifies access to an NTP server n RFC 1769 n Clients located only at the highest strata n SNTP servers do not implement fault tolerance

24. 28 March 2001CS-551, Lecture 724 Logical Time n Because of clock skew, physical clocks do not provide absolute time ordering of events n Instead we use the concept of virtual time to order certain events n There are a great number of algorithms that attempt to provide logical time and some event ordering – E.g. Lamport’s paper

25. 28 March 2001CS-551, Lecture 725 Ordering Events n What is an event? – Sending a message – Receiving a message – Execution within a process n Most events happen asynchronously – Non-instantaneous communication – Interruptions n There is no global state

26. 28 March 2001CS-551, Lecture 726 Assumptions n Assume all processes are sequential n Assume that the sending of a message always precedes the receiving of said message n Need to define a relationship that combine this information – Lamport’s  “happens before” relation

27. 28 March 2001CS-551, Lecture 727 Properties of  1. a  b is defined as i. If a and b are in the same process, then a happens before b happens ii. If a is sending a message, then b is receiving the same message 2. Transitive: If a  b and b  c, then a  c 3. If there is no ordering between a and b, !(a  b) and !(b  a), then a and b are concurrent (disjoint)

28. 28 March 2001CS-551, Lecture 728 Figure 10.8 Happen-Before Relationship Examples. (Galli,p.252)

29. 28 March 2001CS-551, Lecture 729 Properties of a Logical Clock n Let Ci be associated with the process Pi, for all processes Pi n Clock condition: – if a  b, then C(a)  C(b) n Subconditions: – If a  b in process Pi, then Ci(a) < Ci(b) – If a sends message m and b receives m, then Ci(a) < Cj(b)

30. 28 March 2001CS-551, Lecture 730 Logical clock conditions, cont. n In order to achieve these conditions, – Pi increments Ci between any two events related to Pi – If a is sending message m from Pi, put a timestamp, Tm = Ci(a), on the message m – When m is received by b in Pj, Pj sets Cj to be the maximum value of Cj + d or Tm + d for some increment d

31. 28 March 2001CS-551, Lecture 731 Definition of precede n Def: Event a in Pi precedes event b in Pj if and only if (system-wide) – 1. Ci(a) < Cj(b) OR – 2. Ci(a) = Cj(b) and Pi < Pj n Assume that each process Pi is ordered by a unique value of i n This relation is written as a  b

32. 28 March 2001CS-551, Lecture 732 Example t P1P2P3 e1 e8 e4 e5 e3 e2 e6 e7 e1  e4  e7 (P1) e1  e3 (m) e5  e8 (m) e5  e8 (why?) A partial ordering

33. 28 March 2001CS-551, Lecture 733 Example, continued t P1P2P3 e1 e8 e4 e5 e3 e2 e6 e7 1.1 2.2 2.3 2.1 3.1 3.2 1.2 1.3 A DAG Becomes a total ordering on events

34. 28 March 2001CS-551, Lecture 734 Total ordering of events n Any total ordering on events must be consistent with the existent partial order n One solution: a topological sort on the partial order – after the fact n Lamport: Uses an event number and a time stamp on all events n Further, a timestamp is attached to all messages

35. 28 March 2001CS-551, Lecture 735 Figure 10.9 Logical Ordering of Events Using Counters. (Galli,p.253)

36. 28 March 2001CS-551, Lecture 736 Figure 10.10 Logical Ordering of Events Using Physical Clocks. (Galli,p.254)

37. 28 March 2001CS-551, Lecture 737 Causal Events n Causal: – “1. Expressing or indicting cause; – 2. Relating to or acting as cause” – (Merriam-Webster) n Causal events: – If e1  e2, then C(e1)  C(e2) – Two events may have the same timestamp n Just include the i of Pi as part of the timestamp

38. 28 March 2001CS-551, Lecture 738 Definition n For each process p, – Initialize the timestamp, p.TS, to zero – On each event, n If e is receipt of message m – p.TS = max (m.TS, p.TS); n p.TS ++; n e.TS = p.TS; n If e is sending message m – m.TS = p.TS;

39. 28 March 2001CS-551, Lecture 739 Limitation of Lamport’s Clocks n If a  b, then C(a) < C(b) n But the reverse is not necessarily true – if the events occur in different processes – We can’t tell how a and b are related – Each clock can independently advance based on its local events – We need messages exchanges to synchronize between a pair of processes

40. 28 March 2001CS-551, Lecture 740 Concurrent events n There is an arbitrary ordering of concurrent events n This can lead to a causality violation: – When distributed objects are mobile, i.e. they can move freely among processes – This may happen when load balancing occurs

41. 28 March 2001CS-551, Lecture 741 Causality Violation P1P2P3 Migrate T to P2 On P2 I don’t know Where is T? Error? Not transitive Where is T? m1 m3 m4 m2 m0

42. 28 March 2001CS-551, Lecture 742 Causality Violation n Message m0 arrives late to P2 n Message m3 arrives at P2 before P2 knows that T is migrating there n To fulfill the transitivity condition, m3 should have arrived at P2 after m0 arrived at P2

43. 28 March 2001CS-551, Lecture 743 Notation n s(m) = the event of sending message m n r(m) = the event of receiving message m n m1 <c m2 if s(m1)  s(m2) n A causality violation happens if m1 <c m2, but r(m2) <p r(m1) n Need a comparison function f such that – e  e’ iff f(e) < f(e’) – Idea – vector timestamps

44. 28 March 2001CS-551, Lecture 744 Causality Violation, relabelled P1P2P3 Migrate T to P2 (1,0,0) On P2 I don’t know m3 Where is T? Error? Not transitive Where is T? (3,0,3) (3,0,2) (3,2,3) (0,0,1) (2,0,1) (3,0,1) (3,1,3) (3,3,3) (3,2,4) m2 m1 m4

45. 28 March 2001CS-551, Lecture 745 Vector Clocks n Each Pi keeps a clock vector Ci[k], k=1,..,n n The kth entry is Pi’s best guess of what process Pk has for its clock values n A message carries a timestamp vector of the clock vector of the sender n A receiver updates its clock vector using the timestamp vector from the message

46. 28 March 2001CS-551, Lecture 746 Vector Clocks, continued n The vector clocks provide a partial ordering of the timestamps – Using a vector comparison (all elements must be =, pairwise – If ta tb, then a and b are causally related – Otherwise a and b must be concurrent

47. 28 March 2001CS-551, Lecture 747 Causal Ordering of Messages n For M processes, – Initialize p.VT = [0, 0, … 0] – On event e, n If e is receipt of message m – For i=1 to M P.VT[i] = max (p.VT[i], m.VT[i]) n p.VT(self) ++ ; n e.VT = p.VT ; n If e is sending message m – m.VT = p.VT;