1 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Synchronization Algorithms and Concurrent Programming Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld Version: June 2014 Chapter 4 Blocking and Non-blocking Synchronization Chapter 4 Blocking and Non-blocking Synchronization

2 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 A note on the use of these power-point slides: I am making these slides freely available to all (faculty, students, readers). They are in PowerPoint form so you can add, modify, and delete slides and slide content to suit your needs. They obviously represent a lot of work on my part. In return for use, I only ask the following:  That you mention their source, after all, I would like people to use my book!  That you note that they are adapted from (or perhaps identical to) my slides, and note my copyright of this material. Thanks and enjoy! Gadi Taubenfeld All material copyright 2014 Gadi Taubenfeld, All Rights Reserved A note on the use of these power-point slides: I am making these slides freely available to all (faculty, students, readers). They are in PowerPoint form so you can add, modify, and delete slides and slide content to suit your needs. They obviously represent a lot of work on my part. In return for use, I only ask the following:  That you mention their source, after all, I would like people to use my book!  That you note that they are adapted from (or perhaps identical to) my slides, and note my copyright of this material. Thanks and enjoy! Gadi Taubenfeld All material copyright 2014 Gadi Taubenfeld, All Rights Reserved Synchronization Algorithms and Concurrent Programming ISBN: , 1 st edition Synchronization Algorithms and Concurrent Programming ISBN: , 1 st edition To get the most updated version of these slides go to:

3 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © Synchronization Primitives 4.2 Collision Avoidance using Test-and-set Bits 4.3 The Ticket Algorithm using RMW Objects 4.4 Local Spinning using Strong Primitives 4.5 Concurrent Data Structures -- Progress Conditions -- Consistency Conditions -- Non-blocking Queue -- Memory Barriers 4.6 Semaphores 4.7 Monitors 4.8 Fairness of Shared Objects 4.1 Synchronization Primitives 4.2 Collision Avoidance using Test-and-set Bits 4.3 The Ticket Algorithm using RMW Objects 4.4 Local Spinning using Strong Primitives 4.5 Concurrent Data Structures -- Progress Conditions -- Consistency Conditions -- Non-blocking Queue -- Memory Barriers 4.6 Semaphores 4.7 Monitors 4.8 Fairness of Shared Objects Chapter 4 Blocking and Non-blocking Synchronization Chapter 4 Blocking and Non-blocking Synchronization These topics are covered in this presentation

4 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Concurrent Data Structures P1 P2P3 data structure Without locks data structure Using locks P1 P2P3

5 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Concurrent Data Structures P1 P2P3 data structure Without locks data structure Using locks P1 P2P3 false conflicts fault-free solutions only sequential bottleneck simple programming model often complex memory consuming sometime -- weak progress cond. resilient to failures, etc.

Progress Conditions Section Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld ©

7 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Progress Conditions P1 P2P3 data structure Using locks lock-freelock-free data structure P1 P2P3 starvation freedom deadlock (livelock) freedom FIFO (+ df ) no failures wait freedom obstruction freedom non-blocking

8 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Obstruction-freedomObstruction-freedom P1P4P3 Done P2

9 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Non-blockingNon-blocking P1P4P2 Done P3

10 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Wait-freedomWait-freedom P1P4P2P3

11 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Wait-freedomWait-freedom P1P4P2P3 Done

12 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Obstruction-freedomWait-freedomNon-blocking Lock-free Data Structures too weak progress condition not complex strong/desirable complex/less efficient strong enough not so complex

Consistency Conditions Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld ©  Linearizability  Sequential Consistency  …  Linearizability  Sequential Consistency  … How do we define the correctness of a concurrent object ?

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Why are consistency conditions important  Formal specification of concurrent objects  Help to understand possible behaviors of an implementation of a concurrent object and to reason about correctness  Clarifies what optimizations are acceptable 14

Example: Concurrent Queue q.enq( ) q.deq( ) Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld ©

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq q.enq q.deq Is this execution reasonable? Yes Sequential Execution P1 P2 16

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq q.enq q.deq Is this execution reasonable? ??? Concurrent Execution P1 P2 17

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq q.enq q.deq Is this execution reasonable? Yes Concurrent Execution P1 P2 18

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq Is this execution reasonable? Yes q.deq Concurrent Execution P1 P2 19

q.deq Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq q.enq Is this execution reasonable? Yes Concurrent Execution P1 P2 20

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Sequential Specification of an object Sequential specification Queue: In sequential executions, the enqueue operation inserts a value to the queue and the dequeue operation returns and deletes the oldest value in the queue. 21

Chapter 4 Sequential executions concurrent executions Linearizability for an object Each sequential execution is included in the sequential specification Each concurrent execution is “equivalent’’ to an execution in the sequential specification Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 All executions 22

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Linearizability for an object  Need to preserve real time order  Each operation should appear to “take effect” instantaneously at some moment between its invocation and response  Object is correct if this “sequential” behavior is correct q.enq q.deq q.enq q.deq P1 P2 23

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Linearizability  Permits programmers to specify and reason about concurrent objects using known techniques from the sequential domain  provides the illusion that each operation applied by concurrent processes takes effect instantaneously between its invocation and its response 24

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Terminology Linearization Point Invocation Response q.enq 25

q.deq Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq q.enq Is this execution linearizable? Yes Linearizability P1 P2 26

q.deq Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq q.enq Is this execution linearizable? Yes X 2 Linearizability P1 P2 27

q.enq q.deq Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq Is this execution linearizable? No, BUT … Linearizability P1 P2 28

q.enq q.deq Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq Is this execution linearizable? No, BUT … Linearizability P1 P2 29

q.enq q.deq Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq Is this execution linearizable? No, BUT … Linearizability Sequential Consistency P1 P2 30

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Sequential Consistency  No need to preserve real time order – Cannot re-order operations done by the same thread – Can re-order non-overlapping operations done by different threads  Each operation should “take effect” instantaneously between invocation and response events  Object is correct if this “sequential” behavior is correct 31

q.enq Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq Is this execution sequential consistent? Yes Sequential Consistency P1 P2 32

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time Is this execution sequential consistent? Yes P1 P2 p.enq p.deq Sequential Consistency Its also linearizable! 33

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time Is this execution sequential consistent? P1 P2 p.enq p.deq Sequential Consistency 34

q.enq Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time q.enq q.deq Is this execution sequential consistent? No !!! P1 P2 p.enq p.deq Sequential Consistency 35

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 time Is this execution sequential consistent? No !!! P1 P2 q.enq p.enq q.enq q.deq p.enq p.deq Sequential Consistency 36

Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Theorem: Sequential consistency is not compositional. Compositionality Proof: The example from the previous slide. Theorem: Linearizability is compositional. Proof:... 37

A Non-blocking Concurrent Queue Algorithm Section Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

39 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Implementing a Non-blocking Queue enqueuedequeue

40 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Implementing a Non-blocking Queue head tail Empty queue Dummy Node

41 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail lnodeEnqueueEnqueue

42 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail 2 concurrent enqueue operations 2 concurrent enqueue operations What can go wrong when using registers?

43 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail DequeueDequeue

44 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail 2 concurrent dequeue operations 2 concurrent dequeue operations What can go wrong when using registers?

45 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail Enqueue + Dequeue Enqueue + Dequeue (Non-empty queue)

46 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail Enqueue + Dequeue Enqueue + Dequeue (Empty queue)

47 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail Enqueue + Dequeue Enqueue + Dequeue (Empty queue) Lost tail !!! The dequeuer must advance tail before redirecting head

48 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Impossibility Result There is NO wait-free or non-blocking implementation of a queue for two processes or more, from atomic registers. Theorem: There is NO wait-free (or non-blocking) consensus algorithm for 2 processes, using only atomic registers. Theorem: There is a wait-free consensus algorithm for 2 processes, using queues and atomic registers.

49 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Compare & swap (CAS) CAS ( A, B, C ) if A=B then A:=C; return(true) else return(false) shared register oldnew Supported by Sun, Intel, AMD, …

50 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 CAS: The ABA problem if A=B then A:=C CAS (A, local, 100) local:= A CAS ( A, B, C ) shared register oldnew A: valuetag CAS (A, local, ) In the following, we will ignore the ABA problem and assume that it is resolved using additional tag fields. local: valuetag

51 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 A Non-blocking Queue  The algorithm is due to M. M. Michael and M. L. Scott (1996).  The algorithm is included in the Standard Java Concurrency Package.  Every process must be prepared to encounter a half-finished enqueue operation, and help to finish it.  Invariant: The tail refers to either the last node or to the node just before the last node. Section 4.5

52 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail ltail lnext lnode ltail  tail lnext  ltail.next if ltail = tail then if CAS (ltail.next, lnext, lnode) then done  true fi CAS (tail, ltail, lnode) done false done true repeat until done = true else CAS (tail, ltail, lnext) fi fi EnqueueEnqueue try to link lnode to the end of the queue enqueue failed: try to swing tail to a node that was inserted by some other process enqueue done: try to swing tail to the inserted node. if lnext = NULL then

53 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail ltail lnext lnode ltail  tail lnext  ltail.next if ltail = tail then if CAS (ltail.next, lnext, node) then done  true fi CAS (tail, ltail, lnode) done false done true repeat until done = true else CAS (tail, ltail, lnext) fi fi EnqueueEnqueue Question: Would the enqueue operation be correct if the last line is omitted? Answer: Yes, but… if lnext = NULL then

54 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 head tail ltail lnext lnode ltail  tail lnext  ltail.next if ltail = tail then if CAS (ltail.next, lnext, node) then done  true fi CAS (tail, ltail, lnode) done false done true repeat until done = true else CAS (tail, ltail, lnext) fi fi EnqueueEnqueue Question: Would the enqueue operation be correct if this line is omitted? No, when it is assumed that cells can be released and then reallocated. Otherwise, yes. if lnext = NULL then

55 Chapter 4 head tail ltail lnext ltail  tail lnext  lhead.next if lhead = head then done false done true repeat until done = true CAS (tail, ltail, lnext) /* try to advance tail DequeueDequeue lhead lhead  head if lhead = ltail then /* empty or tail behind? if lnext = NULL then return(NULL) fi ; /* empty else lvalue  lnext.value /* no need to deal with tail if CAS (head, lhead, lnext) then done  true fi fi fi lvalue free(lhead) ; return(lvalue) Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

56 Chapter 4 head tail ltail lnext ltail  tail lnext  lhead.next if lhead = head then done false done true repeat until done = true CAS (tail, ltail, lnext) /* try to advance tail DequeueDequeue lhead lhead  head if lhead = ltail then /* empty or tail behind? if lnext = NULL then return(NULL) fi ; /* empty else lvalue  lnext.value /* no need to deal with tail if CAS (head, lhead, lnext) then done  true fi fi fi lvalue free(lhead) ; return(lvalue) Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Question: Would the dequeue operation be correct if this line is omitted?

Memory Barriers Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 A note about

58 Example: Using Flags Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 x and y : atomic bits, initially 0 Q: Is it possible that both processes read the value 0 ? 0 0 x x 0 0 y y Process A write.x(1) read.y write.x(1) read.y Process B write.y(1) read.x write.y(1) read.x

59 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Example: Using Flags x and y : atomic bits, initially x x 0 0 y y Process A write.x(1) read.y write.x(1) read.y Process B write.y(1) read.x write.y(1) read.x Fact: Many hardware architectures do not support sequential consistency because they think it is too strong 

60 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Example: Using Flags x and y : atomic bits, initially x x 0 0 y y Process A write.x(1) read.y write.x(1) read.y Process B write.y(1) read.x write.y(1) read.x Solution: Memory barriers

Binary Semaphores Section 4.6 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

62 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Binary Semaphores unfair semaphore weak semaphore strong semaphore 1 down(S) up(S) Types Operations Section 4.6 down(S) m if S > 0, then S = 0 otherwise, the process is blocked until the value becomes greater than 0. m Testing and decrementing the semaphore are executed atomically without interruption. up(S) m S = 1

63 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Semaphores 1 down(S) critical section up(S) Deadlock-free Mutual exclusion Algorithm

64 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © queue 1 S.0 1 S.1 Weak semaphores 1 Shared bits S.0 1 S.1 0 queue false empty local bits myqueue otherqueue 00 1 critical section false empty Semaphores 1 true doorkeeper Constant Space Starvation-free Algorithm

65 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 queue 1 S.0 S.1 Weak semaphores 1 Shared bits S.0 1 S.1 0 queue false empty local bits myqueue otherqueue 00 1 critical section emptySemaphores Constant Space Starvation-free Algorithm truefalse

66 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 queue 1 S.0 S.1 Weak semaphores 1 Shared bits S.0 1 S.1 0 queue false empty local bits myqueue otherqueue 0 1 critical section emptySemaphores Constant Space Starvation-free Algorithm truefalse

67 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 true queue 1 S.0 S.1 Weak semaphores 1 Shared bits S.0 1 S.1 0 queue false empty local bits myqueue otherqueue 00 1 critical section emptySemaphores Constant Space Starvation-free Algorithm false 1

68 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 queue 1 S.0 S.1 Weak semaphores 1 Shared bits S.0 1 S.1 0 queue false empty local bits myqueue otherqueue 0 1 critical section emptySemaphores Constant Space Starvation-free Algorithm truefalse true

69 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 true queue 1 S.0 S.1 Weak semaphores 1 Shared bits S.0 1 S.1 0 queue false empty local bits myqueue otherqueue 00 1 critical section emptySemaphores Constant Space Starvation-free Algorithm false 1

70 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 queue 1 S.0 S.1 Weak semaphores 1 Shared bits S.0 1 S.1 0 queue false empty local bits myqueue otherqueue 00 1 critical section emptySemaphores 1 true Constant Space Starvation-free Algorithm

71 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Semaphores myeuque = queue down (S.myqueue) if queue = myqueue then otherqueue = 1 – myqueue down(S.otherqueue) queue = otherqueue repeat empty = true up(S.myqueue) down(S.myqueue) until empty critical section up(S.otherqueue) else empty = false critical section fi up(S.myqueue) myeuque = queue down (S.myqueue) if queue = myqueue then otherqueue = 1 – myqueue down(S.otherqueue) queue = otherqueue repeat empty = true up(S.myqueue) down(S.myqueue) until empty critical section up(S.otherqueue) else empty = false critical section fi up(S.myqueue) Weak semaphores 1 Shared bits S.0 1 S.1 0 queue false empty local bits myqueue otherqueue Constant Space Starvation-free Algorithm

72 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © queue 1 S.0 1 S critical section false empty Semaphores 1 true Yes Question: Is it possible that some process will be in its CS, while no process is a doorkeeper? Question: Is it possible that some process will be in its CS, while no process is a doorkeeper? Constant Space Starvation-free Algorithm

73 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Properties of the Algorithm  Satisfies mutex & deadlock-freedom.  Satisfies 2-bounded-waiting: A process can enter its CS ahead of anoter only twice.  There is no need to know the # of processes  System response time is a constant  Two weak semaphores and two atomic bits are used.

74 Chapter 4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 The End