1. Page 1 Lecturer: Roohollah Abdipour Acknowledgment: Many of the slides are based on slides by Prof. Paul Krzyzanowski at Rutgers University, pxk@cs.rutgers.edu 1

2. Page 2 2

3. Page 3 Logical clock keeps track of event ordering  among related (causal) events Physical clocks keep time of day  Consistent across systems 3

4. Page 4  1880: Piezoelectric effect  Curie brothers  Squeeze a quartz crystal & it generates an electric field  Apply an electric field and it bends  1929: Quartz crystal clock  Resonator shaped like tuning fork  Laser-trimmed to vibrate at 32,768 Hz  Standard resonators accurate to 6 parts per million at 31° C  Watch will gain/lose < ½ sec/day  Stability > accuracy: stable to 2 sec/month  Good resonator can have accuracy of 1 second in 10 years  Frequency changes with age, temperature, and acceleration 4

5. Page 5 Real-time Clock: CMOS clock (counter) circuit driven by a quartz oscillator  battery backup to continue measuring time when power is off OS generally programs a timer circuit to generate an interrupt periodically  e. g., 60, 100, 250, 1000 interrupts per second (Linux 2.6+ adjustable up to 1000 Hz)  Programmable Interval Timer (PIT) – Intel 8253, 8254  Interrupt service procedure adds 1 to a counter in memory 5

6. Page 6 Getting two systems to agree on time  Two clocks hardly ever agree  Quartz oscillators oscillate at slightly different frequencies Clocks tick at different rates  Create ever-widening gap in perceived time  Clock Drift Difference between two clocks at one point in time  Clock Skew 6

7. Page 7 Sept 18, 2006 8:00:00 7

8. Page 8 Oct 23, 2006 8:00:00 8:01:248:01:48 Skew = +84 seconds +84 seconds/35 days Drift = +2.4 sec/day Skew = +108 seconds +108 seconds/35 days Drift = +3.1 sec/day 8

9. Page 9 UTC time, t Computer’s time, C 9

10. Page 10 UTC time, t Computer’s time, C skew 10

11. Page 11 UTC time, t Computer’s time, C skew 11

12. Page 12 Assume we set computer to true time Not good idea to set clock back  Illusion of time moving backwards can confuse message ordering and software development environments 12

13. Page 13 Go for gradual clock correction If fast: Make clock run slower until it synchronizes If slow: Make clock run faster until it synchronizes 13

14. Page 14 OS can do this: Change rate at which it requests interrupts e.g.: if system requests interrupts every 17 msec but clock is too slow: request interrupts at (e.g.) 15 msec Or software correction: redefine the interval Adjustment changes slope of system time: Linear compensating function 14

15. Page 15 UTC time, t Computer’s time, C Linear compensating function applied Clock synchronized skew 15

16. Page 16 UTC time, t Computer’s time, C 16

17. Page 17  Attach GPS receiver to each computer ± 1 msec of UTC  Attach WWV radio receiver Obtain time broadcasts from Boulder or DC ± 3 msec of UTC (depending on distance)  Attach GOES receiver ± 0.1 msec of UTC Not practical solution for every machine  Cost, size, convenience, environment 17

18. Page 18 Synchronize from another machine  One with a more accurate clock Machine/service that provides time information: Time server 18

19. Page 19 Simplest synchronization technique  Issue RPC to obtain time  Set time Does not account for network or processing latency clientserver what’s the time? 3:42:19 19

20. Page 20 Compensate for delays  Note times:  request sent: T 0  reply received: T 1  Assume network delays are symmetric server client time requestreply T0T0 T1T1 T server 20

21. Page 21 Client sets time to: server client time requestreply T0T0 T1T1 T server = estimated overhead in each direction 21

22. Page 22 If minimum message transit time (T min ) is known: Place bounds on accuracy of result 22

23. Page 23 server client time requestreply T0T0 T1T1 T server T min Earliest time message arrives Latest time message leaves range = T 1 -T 0 -2T min accuracy of result = 23

24. Page 24  Send request at 5:08:15.100 ( T 0 )  Receive response at 5:08:15.900 ( T 1 )  Response contains 5:09:25.300 ( T server )  Elapsed time is T 1 - T 0 5:08:15.900 - 5:08:15.100 = 800 msec  Best guess: timestamp was generated 400 msec ago  Set time to T server + elapsed time 5:09:25.300 + 400 = 5:09.25.700 24

25. Page 25 If best-case message time=200 msec server client time requestreply T0T0 T1T1 T server 200 800 Error = T 0 = 5:08:15.100 T 1 = 5:08:15.900 T s = 5:09:25:300 T min = 200msec 25

26. Page 26  Gusella & Zatti, 1989  Assumes no machine has an accurate time source  Obtains average from participating computers  Synchronizes all clocks to average 26

27. Page 27  Machines run time dæmon  Process that implements protocol  One machine is elected (or designated) as the server ( master )  Others are slaves 27

28. Page 28  Master polls each machine periodically  Ask each machine for time  Can use Cristian’s algorithm to compensate for network latency  When results are in, compute average  Including master’s time  Hope: average cancels out individual clock’s tendencies to run fast or slow  Send offset by which each clock needs adjustment to each slave  Avoids problems with network delays if we send a time stamp 28

29. Page 29 Algorithm has provisions for ignoring readings from clocks whose skew is too great  Compute a fault-tolerant average If master fails  Any slave can take over 29

30. Page 30 30

31. Page 31 3:252:509:103:00 1. Request timestamps from all slaves 3:25 2:50 9:10 31

32. Page 32 3:252:509:103:00 2. Compute fault-tolerant average: 3:25 2:50 9:10 32

33. Page 33 3:252:509:103:00 3. Send offset to each client -0:20 +0:15 -6:05 +0.15 33

34. Page 34 Logical Clocks 34

35. Page 35 Assign sequence numbers to messages  All cooperating processes can agree on order of events  vs. physical clocks : time of day Assume no central time source  Each system maintains its own local clock  No total ordering of events  No concept of happened-when 35

36. Page 36 Lamport’s “happened-before” notation a  b event a happened before event b e.g.: a : message being sent, b : message receipt Transitive: if a  b and b  c then a  c 36

37. Page 37 Assign “clock” value to each event  if a  b then clock( a ) < clock( b )  since time cannot run backwards If a and b occur on different processes that do not exchange messages, then neither a  b nor b  a are true  These events are concurrent 37

38. Page 38  Three systems: P 0, P 1, P 2  Events a, b, c, …  Local event counter on each system  Systems occasionally communicate 38

39. Page 39 ab hi k P1P1 P2P2 P3P3 12 13 21 df g 3 c 2 4 6 e 5 j 39

40. Page 40 ab i k j P1P1 P2P2 P3P3 12 13 21 df g 3 c 2 4 6 Bad ordering: e  h f  k h e 5 40

41. Page 41  Each message carries a timestamp of the sender’s clock  When a message arrives:  if receiver’s clock < message timestamp set system clock to (message timestamp + 1)  else do nothing  Clock must be advanced between any two events in the same process 41

42. Page 42 Algorithm allows us to maintain time ordering among related events  Partial ordering 42

43. Page 43 ab i k j P1P1 P2P2 P3P3 12 17 21 df g 3 c 2 4 6 6 7 h e 5 43

44. Page 44 44

45. Page 45  Algorithm needs monotonically increasing software counter  Incremented at least when events that need to be timestamped occur  Each event has a Lamport timestamp attached to it  For any two events, where a  b: L(a) < L(b) 45

46. Page 46 46

47. Page 47 Techniques to coordinate execution among processes  One process may have to wait for another  Shared resource (e.g. critical section) may require exclusive access 47

48. Page 48 Mutual exclusion via:  Test & set in hardware  Semaphores  Messages  Condition variables 48

49. Page 49  Assume there is agreement on how a resource is identified  Pass identifier with requests  Create an algorithm to allow a process to obtain exclusive access to a resource. 49

50. Page 50 50

51. Page 51  Mimic single processor system  One process elected as coordinator P C request(R) grant(R) 1. Request resource 2. Wait for response 3. Receive grant 4. access resource 5. Release resource release(R) 51

52. Page 52 If another process claimed resource:  Coordinator does not reply until release  Maintain queue  Service requests in FIFO order P0P0 C request(R) grant(R) release(R) P1P1 P2P2 request(R) Queue P1P1 request(R) P2P2 grant(R) 52

53. Page 53 Benefits  Fair  All requests processed in order  Easy to implement, understand, verify Problems  Process cannot distinguish being blocked from a dead coordinator  Centralized server can be a bottleneck 53

54. Page 54 Assume known group of processes  Some ordering can be imposed on group  Construct logical ring in software  Process communicates with neighbor P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 54

55. Page 55  Initialization  Process 0 gets token for resource R  Token circulates around ring  From P i to P (i+1) mod N  When process acquires token  Checks to see if it needs to enter critical section  If no, send ring to neighbor  If yes, access resource  Hold token until done P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 token(R) 55

56. Page 56  Only one process at a time has token  Mutual exclusion guaranteed  Order well-defined  Starvation cannot occur  If token is lost (e.g. process died)  It will have to be regenerated  Does not guarantee FIFO order  sometimes this is undesirable 56

57. Page 57  Distributed algorithm using reliable multicast and logical clocks  Process wants to enter critical section:  Compose message containing:  Identifier (machine ID, process ID)  Name of resource  Timestamp (totally-ordered Lamport)  Send request to all processes in group  Wait until everyone gives permission  Enter critical section / use resource 57

58. Page 58  When process receives request:  If receiver not interested:  Send OK to sender  If receiver is in critical section  Do not reply; add request to queue  If receiver just sent a request as well:  Compare timestamps: received & sent messages  Earliest wins  If receiver is loser, send OK  If receiver is winner, do not reply, queue  When done with critical section  Send OK to all queued requests 58

59. Page 59 59

60. Page 60  N points of failure  A lot of messaging traffic  Demonstrates that a fully distributed algorithm is possible 60

61. Page 61 61

62. Page 62  Need one process to act as coordinator  Processes have no distinguishing characteristics  Each process can obtain a unique ID 62

63. Page 63  Select process with largest ID as coordinator  When process P detects dead coordinator:  Send election message to all processes with higher IDs.  If nobody responds, P wins and takes over.  If any process responds, P’s job is done.  Optional: Let all nodes with lower IDs know an election is taking place.  If process receives an election message  Send OK message back  Hold election (unless it is already holding one) 63

64. Page 64  A process announces victory by sending all processes a message telling them that it is the new coordinator  If a dead process recovers, it holds an election to find the coordinator. 64

65. Page 65 65

66. Page 66  Ring arrangement of processes  If any process detects failure of coordinator  Construct election message with process ID and send to next process  If successor is down, skip over  Repeat until a running process is located  Upon receiving an election message  Process forwards the message, adding its process ID to the body 66

67. Page 67 Eventually message returns to originator  Process sees its ID on list  Circulates (or multicasts) a coordinator message announcing coordinator  E.g. lowest numbered process 67

68. Page 68 68

69. Page 69 69