2. Chapter 5  Synchronization 1 Synchronization Chapter 5

3. Chapter 5  Synchronization 2 Synchronization  Multiple processes must not simultaneously access shared resource  Ordering may be important o Such as, msg 1 must come before msg 2  Time o Absolute time vs relative time  May want one process to coordinate o Election algorithms

4. Chapter 5  Synchronization 3 Synchronization  Special topics…  Distributed mutual exclusion o Protect shared resources from simultaneous access  Distributed transactions o Similar, but try to optimize access thru “advanced concurrency control”

5. Chapter 5  Synchronization 4 What Time is It?  Easy to answer in a non-dist system o Spse A asks for time, then B o B’s time will be later than A’s o In dist system, this may not be true  Spse A checks time, then B  B’s time might not be later than A’s o That is, time on A and B might not agree o If time comes from a central location, network communication variation is a problem

6. Chapter 5  Synchronization 5 What Time is It?  Why do we care about time?  Consider make example  Make used to compile and link multiple source files into one executable file  If file.o was last modified before file.c, then file.c must be recompiled  If file.o was last modified after file.c, then no need to recompile file.c  This breaks if time is not the same in distributed system

7. Chapter 5  Synchronization 6 Clock Synchronization  Both machines have their own clock o Clocks differ by “2”  What will make do with output.c?  Oops!

8. Chapter 5  Synchronization 7 Time  With single processor system o Doesn’t matter if time is incorrect o Relative time is what’s important  If more than one processor o Clock skew is inevitable  Multiple clock problems o How to synchronize with “real” clock? o How to synchronize clocks with each other?  But first we digress…

9. Chapter 5  Synchronization 8 Physical Clocks  Time between 2 transits of the sun o Solar day  Solar second is 1/86400th solar day

10. Chapter 5  Synchronization 9 Physical Clocks  Period of earth rotation not constant o Earth is slowing due to drag o Days are getting longer  Atomic clock invented 1948  Official second is now o 9,192,631,770 transitions of cesium 133  International Atomic Time (TAI)  Today, 86,400 TAI seconds is about 3 msec less than mean solar day!

11. Chapter 5  Synchronization 10 Physical Clocks  Solar seconds are not of constant length  TAI seconds are of constant length o Leap seconds are used to keep in phase with sun o Add leap second when discrepancy > 800 msec  Otherwise noon would eventually be before breakfast  might cause riots!

12. Chapter 5  Synchronization 11 Physical Clocks  TAI with leap seconds is known as o Universal Coordinated Time (UTC)  UTC replaces Greenwich Mean Time (GMT)  NIST operates radio WWV from Colorado o Sends out pulse at start of each UTC second o But only accurate to within  1 msec o Do to atmospheric effects, can vary by  10 msec  Some satellites offer similar service  In any case, must know relative position o To compensate for propagation delay

13. Chapter 5  Synchronization 12 Clock Sync. Algorithms  Suppose one machine monitor WWV  How to keep other clocks in sync? o Let t be UTC time o Let C p (t) be time on machine p  Ideally, want C p (t) = t o We’ll be happy if dC p /dt = 1

14. Chapter 5  Synchronization 13 Clock Sync. Algorithms  Clocks drift  Suppose  One clock is slow and one is fast…  Drift apart at twice the drift rate

15. Chapter 5  Synchronization 14 Clock Sync. Algorithms  Let C p (t) be time on machine p  Ideally, want C p (t) = t o Or dC p /dt = 1  But processor clocks can drift o If maximum rate of drift is  o After  t, two clocks could be 2    t apart  If you want clocks to differ by less than  o Must synchronize clocks every  / 2  seconds  How to synchronize?

16. Chapter 5  Synchronization 15 Clock Sync. Algorithms  How to synchronize clocks?  Cristian’s algorithm o Pull protocol  Berkeley algorithm o Push protocol  Averaging algorithms o Decentralized approach  Network Time Protocol (NTP)  Multiple external time sources

17. Chapter 5  Synchronization 16 Cristian's Algorithm  Spse time server has WWV time  Clients want to stay within  of others  Every  / 2  seconds or less… o Client asks time server for time  Somebody got an algorithm named after themselves for that?  See next slide

18. Chapter 5  Synchronization 17 Cristian's Algorithm  What are the potential problems? o Time cannot run backwards o Takes (variable) time to get reply

19. Chapter 5  Synchronization 18 Cristian's Algorithm  Time cannot run backwards o If clock is fast… o Increment time more slowly than usual  Must account for time to get reply o How to do this? o Educated guess! Roundtrip time divided by 2 o Account for time server takes to process, multiple roundtrip measurements, etc., etc.

20. Chapter 5  Synchronization 19 Berkeley Algorithm  Cristian’s “algorithm” o Time server is passive  Berkeley algorithm o Time server is aggressive o Does not require server to know UTC o Server polls clients o Computes average time o Pushes result to clients

21. Chapter 5  Synchronization 20 Berkeley Algorithm a) Server asks others for their clock values b) Machines answer c) Server tells others how to adjust their clock

22. Chapter 5  Synchronization 21 Averaging Algorithms  Cristian’s and Berkeley are centralized  Averaging (decentralized) approach… o All machines broadcast time o Everybody computes average o The usual refinements apply  When to broadcast?  Only practical on a LAN

23. Chapter 5  Synchronization 22 Network Time Protocol  According to book, NTP uses o “advanced clock synchronization algorithms” o Accuracy range of 1 to 50 msec  But NTP is not very secure  NTP actually uses Marzullo’s Algorithm o Aka the Intersection Algorithm  Have a collection of times intervals o Example: time of 10  2 gives interval [8,12]

24. Chapter 5  Synchronization 23 Network Time Protocol  Given collection of times intervals o Of the form [a,b]  Marzullo’s algorithm finds consistent interval o Efficient: linear in time and space o If no consistent interval, finds interval(s) consistent with the most sources  Marzullo takes center of resulting interval  Intersection Algorithm refines this o Use statistical info on confidence intervals o Selected time not necessarily midpoint

25. Chapter 5  Synchronization 24 Multiple External Time Sources  Suppose very accurate time needed  Multiple UTC sources?  But these will not agree  So need to average (or similar) o Network delays o Processing delays, etc.  Not clear that this helps very much!

26. Chapter 5  Synchronization 25 Use of Synchronized Clocks  Today, computers can be at or near UTC  How to make use of this?  To enforce “at most once delivery”  Traditional approach o Server keeps track of msg numbers o Checks list against incoming msg numbers o How long to keep list? What if server crashes?  Alternative is to use timestamps o We discuss other apps in later sections

27. Chapter 5  Synchronization 26 Logical Clocks  Usually good enough to agree on time o Even if it’s not the actual time  Often sufficient to agree on order o Recall make example  Lamport time o Synchronize logical clocks  Vector timestamps o Extension of Lamport’s algorithm

28. Chapter 5  Synchronization 27 Lamport Timestamps  “Happens before”: a  b  According to Tanenbaum: a  b if all processes agree that a came before b  Lamport actually defines “  ” as the “smallest” relation satisfying o If a occurs before b on same processor then a  b o If a == send, b == receive, a  b o Transitive: a  b and b  c implies a  c

29. Chapter 5  Synchronization 28 Lamport Timestamps  “Happens before”: a  b  Does “happens before” equal “really happened before”?  If a and b are on same process and a occurs before b, then a  b  If a == msg sent, b == (same) msg received, then a  b o It takes time for message to be sent  If a  b and b  a, msgs are concurrent //

30. Chapter 5  Synchronization 29 Lamport Timestamps  For event a, want timestamp C(a) o If a  b then C(a) < C(b) o C is a non-decreasing function o Time cannot go backwards!  Lamport’s solution o Each msg carries timestamp with it o If local time is less than timestamp, set local time to timestamp + 1 o Advance clock between any two events  Illustrated on next slide…

31. Chapter 5  Synchronization 30 Lamport Timestamps a) Three processes with different clocks 60 54 48 42 36 30 24 18 12 6 0 80 72 64 56 48 40 32 24 16 8 0 100 90 80 70 60 50 40 30 20 10 0 A B C D 76 70 48 42 36 30 24 18 12 6 0 85 77 69 61 48 40 32 24 16 8 0 100 90 80 70 60 50 40 30 20 10 0 A B C D b) Lamport's algorithm corrects the clocks

32. Chapter 5  Synchronization 31 Lamport Timestamps  Can also insure that no two events ever occur at exactly the same time o 40.1 for process 1 o 40.2 for process 2, etc.  With this refinement, we have a total ordering on all events in the system o If a  b on same process then C(a) < C(b) o If a == msg sent, b == msg received, then we have C(a) < C(b) o If a  b then C(a)  C(b)

33. Chapter 5  Synchronization 32 Totally-Ordered Multicast  Consider replicated database o Spse replica in San Jose and in New York o Query goes to nearest copy  Updates are tricky o Must have updates in same order at replicas o For example: Interest calculation and deposit  For consistency, no “right” order o Just want updates to happen in same order  Correctness is a different story…

34. Chapter 5  Synchronization 33 Non-Totally-Ordered Multicast  Assumptions o $1000 in acct, deposit is $1000, interest rate is 10%  On left, $2200, on right $2100  Inconsistent! Deposit Interest

35. Chapter 5  Synchronization 34 Totally-Ordered Multicast  Assume msgs received in order and no loss  Using Lamport timestamps… o Msgs timestamped with sender’s logical time o Multicast sent to all sites, including sender o Msgs go into local queue in timestamp order o Multicast ACK msgs (to yourself too)  Message only removed from queue if o It is at head of queue and o It has been ACKed  Does this work? See next slide…

36. Chapter 5  Synchronization 35 Totally-Ordered Multicast  $1000 in acct, deposit is $1000, interest rate 10%  What happens in this case? DepositInterest Deposit 91 90 46 45 20 10 0 120 105 90 75 60 45 30 Interest ACK(D) ACK(I) Later… Interest: 10 Deposit: 45 ACK(I): 90 ACK(D): 105 Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 After 45… Deposit: 45 After 10… Interest: 10

37. Chapter 5  Synchronization 36 Totally-Ordered Multicast  When is interest calculation done?  When is deposit made? DepositInterest Deposit 91 90 46 45 20 10 0 120 105 90 75 60 45 30 Interest ACK(D) ACK(I) Later… Interest: 10 Deposit: 45 ACK(I): 90 ACK(D): 105 Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 After 45… Deposit: 45 After 10… Interest: 10

38. Chapter 5  Synchronization 37 Scalar Timestamps  Scalar timestamps (such as Lamport timestamps) give total ordering using C(a)  But C(a) < C(b) does not mean that event a really happened before b P1P1 P2P2 P3P3 123 89 10 54 3 1 1 567 11 2 4 79  The “4” at P 2 occurs before the “3” at P 1

39. Chapter 5  Synchronization 38 Vector Timestamps  Lamport timestamps don’t reflect causality o Local events are causally ordered  Example: multicast news posting o “Happens after” not necessarily “response to”  Vector timestamps do reflect causality  Must specify o Local data structures to represent logical time o Update mechanism/protocol  Tanenbaum’s description is confusing!

40. Chapter 5  Synchronization 39 Vector Timestamps  Want vector timestamp such that o If VT(a) < VT(b) then a causally precedes b  Process P i maintains vector V i o V i [i] is incremented for each event at i o V i [j] is P i ’s current view of the number of events that have occurred at process P j  V i [i] is easy to maintain  V i [j] is obtained from info sent with msgs o Each message includes vector timestamp

41. Chapter 5  Synchronization 40 Vector Timestamps  Suppose P j received msg m from P i  P i includes it’s vector timestamp, vt  Then P j adjusts its values according to vt  P j then knows the number of events on which m can depend  Tanenbaum claims… o P j knows no. of messages it must receive before it has seen everything that m could depend on o Not true! Event  msg!

42. Chapter 5  Synchronization 41 Vector Timestamps P1P1 P2P2 P3P3 1 0 0 0 1 0 0 0 1 2 0 0 3 0 0 4 3 4 5 3 4 2 2 0 2 3 0 2 4 0 5 5 4 5 6 4 2 3 4 2 3 3 2 3 2 2 3 0 2 0 0 5 3 4 2 3 4

43. Chapter 5  Synchronization 42 Vector Timestamp  Modified (useful) form of VT  Suppose V i [i] counts msgs sent by P i  Now consider multicast newsgroup  Suppose P i post a message o Includes vector vt(a)  Suppose P j posts a response o Includes vector vt(b)

44. Chapter 5  Synchronization 43 Vector Timestamp  P i posts a and includes vt(a)  P j posts response b with vector vt(b)  Suppose P k receives b before a  P k waits to deliver msg until o vt(b)[j] == V k [j] + 1  This is the next msg expected from P j o vt(b)[i] <= V k [i], all i  j  Ensures that P k must have seen msg a

45. Chapter 5  Synchronization 44 Vector Timestamp  Example

46. Chapter 5  Synchronization 45 Global State  Global state of distributed system o All local states plus msgs in transit o Definition of “state” can vary  Useful to know global state to o Know that computation is finished o Detect deadlock  How to record global state? o Distributed snapshot

47. Chapter 5  Synchronization 46 Global State  Distributed snapshot o A consistent state “in which the system might have been” o For example, if Q received msg from P then must show that P sent the msg o P sent msg Q has not yet received is OK  Global state represented by a cut  Next slide…

48. Chapter 5  Synchronization 47 Global State a) Consistent cut b) Inconsistent cut

49. Chapter 5  Synchronization 48 Global State  Assume distributed system uses point-to- point unidirectional communication  Any process can initiate snapshot  Suppose P starts snapshot o P records its state o P sends “marker” to neighbors  When Q receives marker o First marker on any channel: Q records state o Record incoming messages until… o …Q has received marker on all incoming channels, then Q is done

50. Chapter 5  Synchronization 49 Global State  This figure does not match algorithm!  See next few slides…

51. Chapter 5  Synchronization 50 Global State  Consider the following example  Bank has 3 branches, A, B, C  Each branch connected to others o Point-to-point links  State consists of o Money in branch and… o …money in transit

52. Chapter 5  Synchronization 51 Global State  Note that no messages are in transit  Global state: (S A,S B,S C ) A B C Begin: S A M1M1 M2M2 SBSB SCSC M6M6 M5M5 M3M3 M4M4 Done: S A Done: S B Done: S C

53. Chapter 5  Synchronization 52 Global State  Note that no messages are in transit  Global state: (S A,T,S B,S C ) A B C Begin: S A M1M1 M2M2 SBSB SCSC M6M6 M5M5 M3M3 M4M4 Done: ( S A,T) Done: S B Done: S C T (S A,T)

54. Chapter 5  Synchronization 53 Global State EExample: Termination detection PProcess Q received marker 1st time oPoProcess that sent it is Q’s predecessor oWoWhen Q completes its part… o…o…Q sends DONE msg to its predecessor WWhen is snapshot DONE? oWoWhen initiator of snapshot received DONE from all of its successors

55. Chapter 5  Synchronization 54 Global State  Problem: if DONE and msgs in transit, then computation may not really be done  Are msgs part of snapshot or computation?  Modification: send DONE provided o All of Q’s successors returned DONE and o Q has not received any msg between time state was recorded and marker(s) received  Otherwise send CONTINUE msg  DONE when initiator receives all DONEs o If CONTINUEs, must do it again

56. Chapter 5  Synchronization 55 Election Algorithms  May want one process to coordinate o We don’t care which process  How to choose coordinator?  Have an election! o Assume each process has a unique number o All processes know everybody else’s number o But some processes may be down o Want to elect (live) process with highest number  We’ll consider two election algorithms o Bully algorithm and ring algorithm

57. Chapter 5  Synchronization 56 Bully Algorithm  P notices coordinator not responding o P sends ELECTION msg to all processes with higher number than P’s o If no one responds, P becomes coordinator o If a higher number responds, P is done  Process receives ELECTION from lower no. o Responds with OK o If not already doing so, it initiates an election  Eventually, everybody gives up… o Except for the biggest bully

58. Chapter 5  Synchronization 57 Bully Algorithm  Process 7 was coordinator until he died  Process 4 is first to notice, so holds an election  5 and 6 respond, 4 gives up (why not stop here?)  Now 5 and 6 each hold an election

59. Chapter 5  Synchronization 58 Bully Algorithm d) Process 6 tells 5 to give up e) Process 6 wins, then tells everyone

60. Chapter 5  Synchronization 59 Ring Algorithm  Assume processes are ordered o Everyone knows their successor o Note that no “token” involved  Spse P notices coordinator has died o P sends ELECTION msg to its successor with P’s number attached o If no response, sends msg to P’s successor’s successor, and so on o Each guy in chain appends its number o When msg gets back to P, it selects highest number on list and sends COORDINATOR msg

61. Chapter 5  Synchronization 60 Ring Algorithm  5 and 2 both initiate ELECTION  What will happen?

62. Chapter 5  Synchronization 61 Mutual Exclusion  Critical region  a place where mutual exclusion is required o Example: update to a shared data structure  For single processor system o Use semaphore, monitors, etc.  Possible istributed system approaches o Imitate single processor approach o Distributed approach o Token ring approach

63. Chapter 5  Synchronization 62 Centralized Algorithm  Elect a coordinator  If P want to enter critical region o Checks with coordinator  How does coordinator deny request? o Either explicit denial or no response o Queues any pending requests  Fair, efficient, etc. o No starvation?  But it’s centralized and we hate that!

64. Chapter 5  Synchronization 63 Centralized Algorithm a) Process 1 OK to enter a critical region b) Process 2 asks permission to enter the same critical region, but no reply c) Process 1 exits, coordinator replies to 2

65. Chapter 5  Synchronization 64 Distributed Algorithm  For this, we need a total ordering on events o We know how to do this, right?  P wants to enter critical region o Send request msg (with timestamp) to everybody o Including itself  When request is received o Receiver not in critical region? Send OK o Receiver in critical region? No reply, queue request o Receiver wants to enter critical region but has not yet? Check timestamps, low one wins  After OKed by everybody, enter critical region

66. Chapter 5  Synchronization 65 Distributed Algorithm a) Processes 0 and 2 want to enter critical region b) Process 0 has the lowest timestamp, it wins c) When process 0 is done, 2 gets its turn

67. Chapter 5  Synchronization 66 Token Ring Algorithm AA logical ring with a token TToken passed around ring PProcess can only enter critical region when it has the token EEasy to see that this works! UUsual token ring problems apply

68. Chapter 5  Synchronization 67 Token Ring Algorithm a) Unordered group of processes b) Logical ring (also need a token)

69. Chapter 5  Synchronization 68 Comparison of Mutual Exclusion Algorithms  ????? Lost token, process crash 0 to n – 1 1 to  Token ring Crash of any process 2 ( n – 1 ) Distributed Coordinator crash23Centralized Problems Delay before entry (in message times) Messages per entry/exit Algorithm

70. Chapter 5  Synchronization 69 Distributed Transactions  Blah

71. Chapter 5  Synchronization 70 Transaction Model  Updating master tape is fault tolerant

72. Chapter 5  Synchronization 71 Transaction Model  Primitives for transactions PrimitiveDescription BEGIN_TRANSACTIONMake the start of a transaction END_TRANSACTIONTerminate the transaction and try to commit ABORT_TRANSACTIONKill the transaction and restore the old values READRead data from a file, a table, or otherwise WRITEWrite data to a file, a table, or otherwise

73. Chapter 5  Synchronization 72 The Transaction Model a) Transaction to reserve 3 flights commits b) Aborts when 3rd flight unavailable BEGIN_TRANSACTION reserve WP -> JFK; reserve JFK -> Nairobi; reserve Nairobi -> Malindi; END_TRANSACTION (a) BEGIN_TRANSACTION reserve WP -> JFK; reserve JFK -> Nairobi; reserve Nairobi -> Malindi full => ABORT_TRANSACTION (b)

74. Chapter 5  Synchronization 73 Distributed Transactions a) A nested transaction b) A distributed transaction

75. Chapter 5  Synchronization 74 Private Workspace a) File index and disk blocks of 3-block file b) After transaction modified 0, appended block 3 c) After committing

76. Chapter 5  Synchronization 75 Writeahead Log a) A transaction b) – d) Log before statement is executed x = 0; y = 0; BEGIN_TRANSACTION; x = x + 1; y = y + 2 x = y * y; END_TRANSACTION; (a) Log [x = 0 / 1] (b) Log [x = 0 / 1] [y = 0/2] (c) Log [x = 0 / 1] [y = 0/2] [x = 1/4] (d)

77. Chapter 5  Synchronization 76 Concurrency Control  Managers for handling transactions

78. Chapter 5  Synchronization 77 Concurrency Control  Managers for distributed transactions

79. Chapter 5  Synchronization 78 Serializability a) – c) Transactions T 1, T 2, and T 3 d) Possible schedules BEGIN_TRANSACTION x = 0; x = x + 1; END_TRANSACTION (a) BEGIN_TRANSACTION x = 0; x = x + 2; END_TRANSACTION (b) BEGIN_TRANSACTION x = 0; x = x + 3; END_TRANSACTION (c) Schedule 1x = 0; x = x + 1; x = 0; x = x + 2; x = 0; x = x + 3Legal Schedule 2x = 0; x = 0; x = x + 1; x = x + 2; x = 0; x = x + 3;Legal Schedule 3x = 0; x = 0; x = x + 1; x = 0; x = x + 2; x = x + 3;Illegal (d)

80. Chapter 5  Synchronization 79 Two-Phase Locking  Two-phase locking (duh!)

81. Chapter 5  Synchronization 80 Two-Phase Locking  Strict two-phase locking

82. Chapter 5  Synchronization 81 Pessimistic Timestamp Ordering  Concurrency control using timestamps