Distributed Web Systems Time and Global State Lecturer Department University

Outline Synchronizing physical clocks Algorithms and case studies (NTP) Logical clocks and global state

Motivation Need to measure time accurately: –to know the time an event occurred at a computer –to do this we need to synchronize its clock with an authoritative external clock Algorithms for clock synchronization useful for –concurrency control based on timestamp ordering (e.g. in distributed transactions) –authenticity of requests e.g. in Kerberos –auditing purposes e.g. in e-commerce scenarios –ordering of events - also useful for maintaining consistency of data (e.g. in replication)

Time… Special Theory of Relativity Black holes Parallel worlds Space travel

Events and process states A distributed system is defined as a collection of N processes p i, i = 1,2,… N –Each process p i has a state s i consisting of its variables (which it transforms as it executes) –Processes communicate only by messages (via a network) –Actions of processes: Send, Receive, Change own state –Event: the occurrence of a single action that a process carries out as it executes e.g. Send, Receive, Change state “Happened-before” relation: –Events at a single process p i, can be placed in a total ordering: e  e’ means e happened before e’ –a history of process is a series of ordered events: history(p i ) =

Clocks and timestamps Hardware clock: –electronic device counting physical events occurring at a definite frequency (e.g. oscillations in a quartz crystal) Software clock: –At real time, t, the OS reads the time on the computer’s hardware clock H i (t) –It calculates the time on its software clock C i (t)=  H i (t) +  Clock resolution: –period between updates of the clock value: successive events will correspond to different timestamps only if the clock resolution is smaller than the time interval between the events scaling offset

Clock skew and drift Computer clocks are not generally in perfect agreement –Skew: the difference between the times on two clocks (at any instant) Computer clocks are subject to clock drift (they count time at different rates) –Clock drift rate: the difference per unit of time from some ideal reference clock –Ordinary quartz clocks drift by about 1 sec in days. (10 -6 secs/sec). High precision quartz clocks drift rate is about or secs/sec © Pearson Education 2005

Coordinated Universal Time International Atomic Time is based on very accurate physical clocks (drift rate secs/sec) UTC is an international standard for time keeping –It is based on atomic time, but occasionally adjusted to astronomical time –It is broadcast from radio stations on land and satellite (e.g. GPS) Computers with receivers can synchronize their clocks with these timing signals –Signals from land-based stations are accurate to about millisecond –Signals from GPS are accurate to about 1 microsecond

Synchronizing physical clocks External synchronization –Clocks C i of a set of N computers are synchronized with an external authoritative time source S if: –|S(t) - C i (t) | < D for i = 1, 2, … N for all t in an interval of real time –The clocks C i are accurate to within the bound D. Internal synchronization –The clocks C i of a set of N computers are synchronized with one another if: –| C i (t) - C j (t) | < D for i,j = 1, 2, … N for all t in an interval of real time –The clocks C i agree within the bound D. Internally synchronized clocks are not necessarily externally synchronized, as they may drift collectively – if the set of processes P is synchronized externally within bound D, it is also internally synchronized within bound 2D

Clock correctness A hardware clock, H is said to be correct if its drift rate is within a known bound  > 0. (e.g secs/ sec) –the error in measuring the interval between real times t and t’ is bounded: (1 -  (t’ - t) ≤ H(t’) - H(t) ≤ (1 +  (t’ - t) (where t’>t) –forbids jumps in time readings of hardware clocks For software clocks, weaker condition of monotonicity –t' > t  C(t’) > C(t) –e.g. required by Unix make –can achieve monotonicity with a hardware clock that runs fast by adjusting the values of  and  in C i (t)=  H i (t) +  a faulty clock is one that does not obey its correctness condition –crash failure - a clock stops ticking –arbitrary failure - any other failure e.g. jumps in time, Y2K

Synchronization in a synchronous system Internal synchronization in a synchronous system: One process p1 sends its local time t to process p2 in a message m, p2 could set its clock to t + T trans where T trans is the time to transmit m T trans is unknown but min ≤ T trans ≤ max uncertainty u = max-min. Set clock to t + (max - min)/2 then skew ≤ u/2 Recall: a synchronous distributed system is one in which the following bounds are defined: –the time to execute each step of a process has known lower and upper bounds –each message transmitted over a channel is received within a known bounded time –each process has a local clock whose drift rate from real time has a known bound

Cristian’s method for an asynchronous system A time server S receives signals from a UTC source –Process p requests time in m r and receives t in m t from S –p sets its clock to t + T round /2 –Accuracy ± (T round /2 - min) : because the earliest time S puts t in message m t is min after p sent m r. the latest time was min before m t arrived at p the time by S’s clock when m t arrives is in the range [t+min, t + T round - min] T round is the round trip time recorded by p min is an estimated minimum transmission time m r m t p Time server,S © Pearson Education 2005

Berkeley algorithm Cristian’s algorithm - –a single time server might fail, so they suggest the use of a group of synchronized servers –it does not deal with faulty servers Berkeley algorithm –An algorithm for internal synchronization of a group of computers –A master polls to collect clock values from the others (slaves) –The master uses round trip times to estimate the slaves’ clock values –It takes an average (eliminating any above some average round trip time or with faulty clocks) –It sends the required adjustment to the slaves (better than sending the time which depends on the round trip time) –Measurements 15 computers, clock synchronization millisecs drift rate < 2x10-5 If master fails, can elect a new master to take over (not in bounded time)

Network time protocol (NTP) A time service for the Internet - synchronizes clients to UTC Primary servers are connected to UTC sources Secondary servers are synchronized to primary servers Synchronization subnet - lowest level servers in users’ computers

Synchronization of NTP servers The synchronization subnet can reconfigure if failures occur –a primary that loses its UTC source can become a secondary –a secondary that loses its primary can use another primary Modes of synchronization: –Multicast A server within a high speed LAN multicasts time to others which set clocks assuming some delay (not very accurate) –Procedure call A server accepts requests from other computers (like Cristiain’s algorithm). Higher accuracy. Useful if no hardware multicast. –Symmetric Pairs of servers exchange messages containing time information Used where very high accuracies are needed (e.g. for higher levels)

Messages exchanged between NTP peers All modes use UDP Each message bears timestamps of recent events: –Local times of Send and Receive of previous message –Local times of Send of current message Recipient notes the time of receipt T i ( we have T i-3, T i-2, T i-1, T i ) In symmetric mode there can be a non-negligible delay between messages T i T i-1 T i-2 T i-3 Server B Server A Time mm' Time

Accuracy of NTP For each pair of messages between two servers, NTP estimates an offset o, between the two clocks and a delay d i (total time for the two messages, which take t and t’) T i-2 = T i-3 + t + o and T i = T i-1 + t’ - o This gives us (by adding the equations) : d i = t + t’ = T i-2 - T i-3 + T i - T i-1 Also (by subtracting the equations) o = o i + (t’ - t )/2 where o i = (T i-2 - T i-3 + T i-1 - T i )/2 Using the fact that t, t’>0 it can be shown that o i - d i /2 ≤ o ≤ o i + d i /2. –Thus o i is an estimate of the offset and d i is a measure of the accuracy NTP servers filter pairs, estimating reliability from variation, allowing them to select peers Accuracy of 10s of millisecs over Internet paths (1 on LANs)

Logical time and logical clocks Instead of synchronizing clocks, event ordering can be used: 1.If two events occurred at the same process p i (i = 1, 2, … N) then they occurred in the order observed by p i, (recall happened-before relation) 2.when a message, m is sent between two processes, send(m) happened before receive(m) 3.The happened before relation is transitive Not all events are related in this way! (consider “a” and “e” below)

Lamport’s logical clocks A logical clock is a monotonically increasing software counter. It need not relate to a physical clock. Each process p i has a logical clock, L i which can be used to apply logical timestamps to events – LC1: L i is incremented by 1 before each event at process p i – LC2: (a) when process p i sends message m, it piggybacks t = L i (b) when pj receives (m,t) it sets L j := max(L j, t) and applies LC1 before timestamping the event receive (m)

Vector clocks A problem with Lamport’s clocks: cannot tell if events are ordered by happened-before by looking at the logical clock timestamps…  Vector clock V i at process p i is an array of N integers  VC1: initially V i [j] = 0 for i, j = 1, 2, …N  VC2: before p i timestamps an event it sets V i [i] := V i [i] +1  VC3: p i piggybacks t = V i on every message it sends  VC4: when p i receives (m,t) it sets V i [j] := max(V i [j], t[j]) j = 1, 2, …N ( then before next event adds I to own element using VC2)

Need for global state Finding out whether a particular property in the system is true (i.e. whether the system in a particular global state) –Distributed garbage collection –Distributed deadlock detection –Distributed debugging Problem: absence of global time –use the idea of process histories instead! history(p i ) = –Global state corresponds to a set of initial prefixes of the individual process histories

Detecting global properties p 2 p 1 message garbage object object reference a. Garbage collection p 2 p 1 wait-for b. Deadlock © Pearson Education 2005

Consistent global states Not all combinations of history prefixes are meaningful: Inconsistent cut Consistent cut See the Chandy and Lamport’s ‘snapshot’ algorithm for determining global states of distributed systems…

Summary Knowing when an event occurred at a computer is important for many functions in distributed systems No global time – instead synchronisation of clocks within certain accuracy bounds Sometimes, the exact time in not important, but the order of events is – use logical time. Also useful for determining global state of a system. Questions