1 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Synchronization Algorithms and Concurrent Programming Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld Chapter 9 Consensus Chapter 9 Consensus Version: June 2014

2 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 A note on the use of these ppt slides: I am making these slides freely available to all (faculty, students, readers). They are in PowerPoint (2003) form so you can add, modify, and delete slides and slide content to suit your needs. They obviously represent a lot of work on my part. In return for use, I only ask the following:  That you mention their source, after all, I would like people to use my book!  That you note that they are adapted from (or perhaps identical to) my slides, and note my copyright of this material. Thanks and enjoy! Gadi Taubenfeld All material copyright 2014 Gadi Taubenfeld, All Rights Reserved A note on the use of these ppt slides: I am making these slides freely available to all (faculty, students, readers). They are in PowerPoint (2003) form so you can add, modify, and delete slides and slide content to suit your needs. They obviously represent a lot of work on my part. In return for use, I only ask the following:  That you mention their source, after all, I would like people to use my book!  That you note that they are adapted from (or perhaps identical to) my slides, and note my copyright of this material. Thanks and enjoy! Gadi Taubenfeld All material copyright 2014 Gadi Taubenfeld, All Rights Reserved Synchronization Algorithms and Concurrent Programming ISBN: , 1 st edition Synchronization Algorithms and Concurrent Programming ISBN: , 1 st edition To get the most updated version of these slides go to:

3 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © The Problem 9.2 Three Simple Consensus Algorithms 9.3 Consensus without Memory Initialization 9.4 Reaching Consensus Using a Shared Queue 9.5 Impossibility of Consensus with One Faulty Process 9.6 The Relative Power of Synchronization Primitives 9.7 The Universality of Consensus 9.1 The Problem 9.2 Three Simple Consensus Algorithms 9.3 Consensus without Memory Initialization 9.4 Reaching Consensus Using a Shared Queue 9.5 Impossibility of Consensus with One Faulty Process 9.6 The Relative Power of Synchronization Primitives 9.7 The Universality of Consensus Chapter 9 Consensus Consensus

4 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 ConsensusConsensus Each process p i has input value in i  agreement: All non-faulty processes eventually decides on the same value v.  validity: v  {in 1, in 2, …, in n }. -- binary consensus ; multi-valued consensus -- we assume crash (fail-stop) failures p1p1 p2p2 p3p3 p4p4 p5p5 p6p6 pnpn … Section 9.1

5 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 ConsensusConsensus There is a trivial solution when there are no faults, but what happens when there are faults? p1p1 p2p2 p3p3 p4p4 p5p5 p6p6 pnpn …

6 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Impossibility of Consensus with One Faulty Process What can be done?  Strong primitives.  Timing assumptions.  Randomization. Theorem: There is no (asynchronous) algorithm that solves the consensus problem using atomic read/write registers in the presence of a single faulty process. We give a detailed proof of this important result in Chapter 9.5 of the book.  Same result holds also for message passing systems

Three Simple Consensus Algorithms Section 9.2 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld ©

8 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: n processes one 3-valued register (trivial) The Algorithm  The first process to access z sets z to its input.  All the processes adopt this value as the decision value. 3-valued RMW register z  The algorithm is wait-free (i.e., can tolerate any number of failures) 

9 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: Two processes three bits (trivial) The Algorithm 1. Each process p i uses one bit r i to announce its input. 2. Then, each process tries to set z from 0 to The decision value is the input of the process that succeeds in step 2.  The algorithm is wait-free atomic r/w bit r2r2 RMW bit z 0 atomic r/w bit r1r1

10 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: Two processes two bits (not trivial) x RMW bit y  The algorithm is wait-free 0 RMW bit 0

11 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: Two processes two bits (not trivial) Algorithm for process p i with input in i 1. If x =1 then p i decides 1 and halts, otherwise p i sets x to in i and continues. 2. If y =0 then p i sets y to 1, decides in i and halts, otherwise p i continues. 3. If x =0 then p i decides 0 and halts, otherwise p i decides 1-in i. x RMW bit y  The algorithm is wait-free 0 RMW bit 0

12 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: Two processes two bits (not trivial) Is the following modified version of the algorithm correct? Algorithm for process p i with input in i 1. If x =1 then p i decides 1 and halts, otherwise p i sets x to in i if in i =1 than p i decides 1 and halts else p i continues. 2. If y =0 then p i sets y to 1, decides in i and halts, otherwise p i continues. 3. If x =0 then p i decides 0 and halts, otherwise p i decides 1-in i. x RMW bit y 0 0 NO

13 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: Three processes two bits x RMW bit y 0 0 IMPOSSIBLE! There is NO wait-free algorithm!!!

14 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Three processes many RMW bits and atomic register IMPOSSIBLE! There is NO wait-free algorithm!!! RMW bits atomic read/write registers Theorem: There is NO consensus algorithm for n processes that can tolerate 2 faults, for any n.

15 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Two processes two atomic read/write registers xy 0 atomic register 0 IMPOSSIBLE! There is NO wait-free algorithm!!! atomic register

16 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Two processes many atomic register IMPOSSIBLE! There is NO wait-free algorithm!!! atomic read/write registers Theorem: There is NO consensus algorithm for n processes that can tolerate one fault, for any n. We will prove this very interesting theorem later.

17 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: many processes, four bits, at most one fault x y 0 RMW bits 0 decision 00 finish

18 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: n processes, four bits, at most one fault x y 0 RMW bits 0 decision 00 finish The Algorithm 1. Process 1 and process 2 execute the algorithm for two processes using x and y. 2. Then, they write the decision value into the decision bit and set finish to Each other process busy waits until finish=1, and then decides on the value of the decision bit.

19 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 RMW: n processes, four bits, at most one fault Is it important that all the four bits are initially 0 ? x y 0 RMW bits 0 decision 00 finish No, the initial value of the decision bit is immaterial.

20 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Important Conclusion It is not possible to implement a single RMW bit for two processes or more, from atomic read/write registers, that can tolerate one fault. Theorem: There is NO consensus algorithm for n processes that can tolerate one fault, for any n. Theorem: There is a consensus algorithm for n processes that can tolerate one fault, for any n. RMW bits read/write registers

21 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Question Question Crash-safe semaphores Why is it not possible to implement a semaphore S for three processes or more, from RMW bits (or test-and- set bits) and atomic read/write registers (no kernel calls), which supports the usual up(S) and down(S) operations, and in addition, it is possible to infer which process holds the semaphore S ? (Semaphores are defined in Section 4.6, page 176.)

Consensus without Memory Initialization Section 9.3 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

23 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Consensus without Memory Initialization Solve the consensus problem in the following model:  There are n processes that communicate via a single “large enough” RMW register.  The initial value of the register is not a priori known.  As always, processes are asynchronous.  Processes may crash. With a known initial value, it can be done with a 3- valued register. How would we do it in a fault-free model? RMW register (not bit) ? Impossible if n/2 processes may crash! We assume that at most n/2 - 1 processes may crash. Q: How big? A: 3n values.

24 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld ©  n/2  circle b = 0 n 0  n/2  circle b = 1 n Consensus without Memory Initialization ??  1.5n -1  bit bc A single RMW register with 3n values B = {0,1} C = {0,  1.5n -2 

25 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld ©  n/2  circle b = 0 n 0  n/2  circle b = 1 n Consensus without Memory Initialization 0 bc Example: n = n = 10 B = {0,1} C = {0,…,13}

26 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 The Algorithm:  Each process reads the value of the shared register, stores it, and increments c by one (modulo |C|).  A process becomes the master if it learns (by inspecting c) that more than half of the processes have waked up and the value of b has not changed. Consensus without Memory Initialization 04 bc n =10 0  n/2  circle b = 0 n 0  n/2  circle b = 1 n

27 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 The master:  The master decides on its own input value, say v ;  if v = 0 then c  0 else c   n/2  ;  if b = 0 then b  1 else b  0 ; Consensus without Memory Initialization 011 bc n = interval 0-interval 1-interval 0-interval 510 one atomic action Example: v = 0 Example: v = 1

28 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 The master:  continues forever setting c to the beginning of the interval corresponding to the decision. (i.e., if v = 0 then c  0 else c   n/2  ) Consensus without Memory Initialization 15 bc n = interval 0-interval 1-interval 0-interval 510

29 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Every other process:  find outs that a decision has been made:  b changed value, or  c has been incremented by more than n (modulo |C|) ;  makes a decision according to the interval in which c lies ; Consensus without Memory Initialization 1 bc n = interval 0-interval 1-interval 0-interval 510 5

30 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Every process:  after making its decision, continues forever setting c to the beginning of the interval corresponding to its decision. Consensus without Memory Initialization bc n = interval 0-interval 1-interval 0-interval

31 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Comments:  There is a fault-free solution that uses only 5 values. (Based on the See-Saw Barrier, page 215.)  The known lower bound (when there are faults) is Ω(n 0.63 ) values. Consensus without Memory Initialization

Reaching Consensus using a Shared Queue Section 9.4 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

33 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Reaching Consensus using a Shared Queue Observations:  With a peek operation – trivial for many processes.  Three processes – Impossible! (without peek)  Two processes with initialized queue – trivial (without peek)  Two processes with empty queue – not trivial (without peek) Requirements:  Solution must be wait-free.  Can use many queues & atomic registers

34 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Reaching Consensus using a Shared Queue if in i = 0 then enqueue(Q,0) if R=0 then decide(0) else if dequeue(Q) = empty then decide(0) else decide (1) fi fi else R:=1 if dequeue(Q) = empty then decide(1) else decide (0) fi fi if in i = 0 then enqueue(Q,0) if R=0 then decide(0) else if dequeue(Q) = empty then decide(0) else decide (1) fi fi else R:=1 if dequeue(Q) = empty then decide(1) else decide (0) fi fi Queue, initially empty Atomic bit Q 0 R Program for process p i with input in i

Impossibility of Consensus with One Faulty Process Section 9.5 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

36 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Theorem: There is no (asynchronous) algorithm that solves the consensus problem using atomic read/write registers in the presence of a single faulty process.  Same result holds also for message passing systems. Why? Section 9.5 Impossibility of Consensus with One Faulty Process

37 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Impossibility of Consensus with One Faulty Process Section 9.5 Notions:  run, empty run, run state event

38 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © final decision values 0-valent 1-valent bivalent time step by pstep by q bivalent & univalent

39 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Lemma 1 (simple) Any consensus algorithm that can tolerate one crash failure has a bivalent empty run (i.e., bivalent initial state). bivalent 01

40 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © Lemma 1 Proof Any consensus algorithm that can tolerate one crash failure has a bivalent empty run

41 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem Assume to the contrary that CONS is a consensus algorithm that can tolerate one crash failure... bivalent step by p 1 step by p 2 step by p n step by p 1 step by p 2 step by p n x Since CONS is correct, this run can not be extended forever.  for some process p, for every extension y of x the run yp is univalent. other processes may take steps, but p 1 must take at least one step

42 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem the shortest extension of x which is x single step by p for every extension y of x the run yp is univalent no steps by p

43 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem x no steps by p option 1: single step by p single step by p

44 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem x no steps by p option 2: not a step by p single step by p

45 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem x no steps by p single step by p p p q Four cases read read write write read write p q So, in both option 1 and option 2 we get the following picture

46 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem p p q Four cases read read write write read write p q

47 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem p p q Four cases read read write write read write p q q does not take steps decide 0 A contradiction

48 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem p p q Four cases read read write write read write p q p does not take steps decide 0 q A contradiction

49 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem p p q Four cases read read write write read write p q All processes take steps decide 0 q A contradiction different registers same register

50 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem p p q q does not take steps decide 0 A contradiction Four cases read read write write read write p q different registers same register

51 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Proof of the theorem QED

The Relative Power of Synchronization Primitives Section 9.6 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

53 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Definition: The consensus number of an object of type o, denoted CN(o), is the largest n for which it is possible to solve consensus for n processes using any number of objects of type o and any number of atomic registers. If no largest n exists, the consensus number of o is infinite. The Relative Power of Synchronization Primitives Examples: CN(atomic-register) = 1 CN(read-modify-write bit) = 2 CN(queue) = 2 CN(3-valued read-modify-write register) =  CN(compare-and-swap) = 

54 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 The Relative Power of Synchronization Primitives Section 9.6 Consensus object Number 1 atomic-register, atomic-snapshot, safe m, regular m 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, read-modify-write bit, fetch-and-increment m, fetch-and-add m  (  m) swap m 2m-2 m-register assignment, atomic-register m (m>1)  compare-and-swap, LL/SC, sticky-bit, queue 2, 3-valued read-modify-write, augmented-queue Consensus object Number 1 atomic-register, atomic-snapshot, safe m, regular m 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, read-modify-write bit, fetch-and-increment m, fetch-and-add m  (  m) swap m 2m-2 m-register assignment, atomic-register m (m>1)  compare-and-swap, LL/SC, sticky-bit, queue 2, 3-valued read-modify-write, augmented-queue O m is an object in which a process is allowed to atomically access m objects of type O

55 The Relative Power of Synchronization Primitives Section 9.6 Consensus object Number 1 atomic-register, atomic-snapshot, safe m, regular m 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, read-modify-write bit, fetch-and-increment m, fetch-and-add m  (  m) swap m 2m-2 m-register assignment, atomic-register m (m>1)  compare-and-swap, LL/SC, sticky-bit, queue 2, 3-valued read-modify-write, augmented-queue Consensus object Number 1 atomic-register, atomic-snapshot, safe m, regular m 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, read-modify-write bit, fetch-and-increment m, fetch-and-add m  (  m) swap m 2m-2 m-register assignment, atomic-register m (m>1)  compare-and-swap, LL/SC, sticky-bit, queue 2, 3-valued read-modify-write, augmented-queue Theorem: Let o 1 and o 2 be two objects such that CN(o 1 ) < CN(o 2 ). Then, in a system with CN(o 2 ) processes,  There is no wait-free implementation of an object of type o 2 from objects of type o 1 and atomic registers;  There is a wait-free implementation of an object of type o 1 from objects of type o 2 and atomic registers. Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

56 The Relative Power of Synchronization Primitives Section 9.6 Consensus object Number 1 atomic-register, atomic-snapshot, safe m, regular m 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, read-modify-write bit, fetch-and-increment m, fetch-and-add m  (  m) swap m 2m-2 m-register assignment, atomic-register m (m>1)  compare-and-swap, LL/SC, sticky-bit, queue 2, 3-valued read-modify-write, augmented-queue Consensus object Number 1 atomic-register, atomic-snapshot, safe m, regular m 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, read-modify-write bit, fetch-and-increment m, fetch-and-add m  (  m) swap m 2m-2 m-register assignment, atomic-register m (m>1)  compare-and-swap, LL/SC, sticky-bit, queue 2, 3-valued read-modify-write, augmented-queue Theorem: Let o 1 and o 2 be two objects with consensus number n. Then, using any number of additional atomic registers,  The objects o 1 and o 2 can wait-free implement each other when the number of processes is less than or equal to n;  The objects o 1 and o 2 do not necessarily wait-free implement each other when the number of processes is more than n. Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

57 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Open Question Consensus object Number 1 atomic-register 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, read-modify-write bit  (  m) 2m-2  Consensus object Number 1 atomic-register 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, read-modify-write bit  (  m) 2m-2 

The Universality of Consensus Section 9.7 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

59 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Definition: An object o is universal for n processes if  any object that has sequential specification has  wait-free  linearizable implementation  using atomic registers and objects of type o  in a system with n processes. UniversalityUniversality Section 9.7

60 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 Definition: An object o is universal for n processes if  any object that has sequential specification has  wait-free  linearizable implementation  using atomic registers and objects of type o  in a system with n processes. The Universality of Consensus Section 9.7 Theorem: A consensus object for n processes is universal in a system with n processes, for any positive n.

61 Consensus Numbers & Universality Section 9.6 Consensus object Number 1 atomic-register, atomic-snapshot, consensus(1) 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, consensus(2) fetch-and-increment m, fetch-and-add m  (  m) swap m consensus(  m) 2m-2 m-register assignment, consensus(2m-2)  compare-and-swap, LL/SC, sticky-bit, queue 2, 3-valued read-modify-write, consensus(  ) Consensus object Number 1 atomic-register, atomic-snapshot, consensus(1) 2 test-and-set, fetch-and-increment, fetch-and-add, swap, queue, stack, consensus(2) fetch-and-increment m, fetch-and-add m  (  m) swap m consensus(  m) 2m-2 m-register assignment, consensus(2m-2)  compare-and-swap, LL/SC, sticky-bit, queue 2, 3-valued read-modify-write, consensus(  ) Theorem: A consensus object for n processes is universal in a system with n processes, for any positive n. Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014

62 Chapter 9 Synchronization Algorithms and Concurrent Programming Gadi Taubenfeld © 2014 A detailed proof appears in Section 9.7 The Universality of Consensus: Proof Section 9.7