1. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 1 Principles of Reliable Distributed Systems Recitation 2: Broadcast Services Spring 2009 Alex Shraer

2. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 2 Broadcast Service for Replication Primitives: broadcast(m), deliver(m). –For simplicity, assume m is unique. Network Broadcast Algorithm Application deliver broadcast receivesend Broadcast Algorithm Application deliver broadcast receivesend

3. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 3 Reliable Broadcast Specifications Validity: if a correct process broadcasts m then all correct processes eventually deliver m Agreement: if a correct process delivers m then all correct processes eventually deliver m –Uniform Agreement: if any process delivers m then all correct processes eventually deliver m Integrity: m is delivered by a correct process at most once, and only if it was previously broadcast

4. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 4 Reliable Broadcast - Quiz What happens if a process fails during the broadcast of a message? Does a message delivery by a faulty process require the delivery of this message by correct processes?

5. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 5 FIFO Broadcast Why is FIFO important? FIFO Order: If a process broadcasts a message m before it broadcasts a message m’, then no correct process delivers m’ unless it has previously delivered m. FIFO Broadcast: Reliable broadcast + FIFO Order Alternative definition of FIFO Order? –“all messages broadcast by the same process are delivered to all processes in the order they are sent” Quiz: Are these definitions equivalent?

6. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 6 Example Also, this alternative definition forces faulty processes to deliver messages. (impossible)

7. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 7 Causal Broadcast Why is causality important? Event e causally precedes event f (e→f) iff: –a process executes both e and f, in that order, or –e is the broadcast of some message m and f is the delivery of m, or –There is an event h, such that e→h and h→f. Causal Order: If the broadcast of a message m causally precedes the broadcast of a message m’, then no correct process delivers m’ unless it has previously delivered m. Causal Broadcast: Reliable broadcast + Causal order

8. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 8 Atomic Broadcast and Uniformity Why would we want more than Causal Broadcast? Atomic Broadcast: Reliable Broadcast + Total Order Total Order: if correct processes p and q both deliver messages m and m’, then p delivers m before m’ if and only if q delivers m before m’.

9. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 9 Broadcast Primitives

10. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 10 Uniformity Agreement, Integrity and Order place no restrictions on the behavior of faulty processes. Uniform – limit the behavior of faulty processes Example 1: Agreement allows a faulty process to deliver a message that is never delivered by correct processes Uniform Agreement: If a process (whether correct or faulty) delivers a message m, then all correct processes eventually deliver m. Example 2: Integrity allows a faulty process to deliver a message more than once, and to deliver messages ‘out of thin air’ Uniform Integrity: For any message m, every process (whether correct or faulty) delivers m at most once, and only if some process broadcast m. Likewise, we can strengthen the Order properties: Uniform FIFO Order: If a process broadcasts a message m before it broadcasts a message m’, then no process (whether correct of faulty) delivers m’ unless it has previously delivered m. Uniform Causal Order: If the broadcast of a message m causally precedes the broadcast of a message m’, then no process (whether correct or faulty) delivers m’ unless it has previously delivered m. Uniform Total Order: if any processes p and q (whether correct or faulty) both deliver messages m and m’, then p delivers m before m’ iff q delivers m before m’.

11. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 11 Crash Failures Suppose processes are only subject to crash failures –They operate correctly up to the time they crash (by definition). Can we assume that the message deliveries that a process makes before crashing are always ‘correct’ (consistent with those of correct processes)? –No

12. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 12 Crash Failures (cont’d) Coordinator-based algorithm: –When a process intends to broadcast a message m, it first sends m to a coordinator. –The coordinator delivers messages in the order in which it receives them, and periodically informs the other processes of this message delivery order. –Other processes deliver messages according to this order. –If the coordinator crashes, another process takes over as coordinator.

13. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 13 Crash Failures (cont’d) The algorithm satisfies the specification Atomic Broadcast Suppose a coordinator delivers m before m’ and crashes. A new coordinator could think m’ is before m. All correct processes follow the new coordinator Thus, the old coordinator delivered messages out of order before it crashed. –Inconsistency can occur even when there are only crash failures. –Protocols should explicitly prevent inconsistency even when there are only crash failures

14. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 14 LTS Broadcast Algorithm - code for process p i Logical Clock Assignment: TS[j] ← 0,  j=0,…,n pending ← empty broadcast (m) TS[i] ← TS[i] + 1 send (m,  TS[i], i  ) to all upon receive (m,  t, j  ) TS[j] ← t add (m,  t, j  ) to pending TS[i] ← max (TS[i], t) + 1 Delivery Rule let (m,  t, j  ) be the entry in pending with the smallest  t, j  if  t, j    TS[k],k   k=0,…n then deliver (m) remove (m,  t, j  ) from pending

15. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 15 0 0  1,2  2  3,1   3,3  4 5  6,2  7 1 4 3 6 האם הרשת מעבירה את כל ההודעות בסדר שמשמר את יחס ה-causality (כלומר happens-before)? סמנו את ערך logical clock בכל פעם שהוא משתנה. סמנו את ערך ה-LTS המצורף לכל הודעה. ציינו מתי (באיזה t) כל תהליך מבצע delivery להודעות m1 ו m3. אם בריצה המתוארת לא מתבצע delivery ציינו זאת בטבלה. m3m1 p1 p2 p3 Example Exam Question 0 2738 12 77 7 10 Delivery according to LTS

16. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 16 Vector Clocks At process p i, on broadcast(m) –VC[i] := VC[i]+1 –use reliable broadcast to send m with VC to all –deliver m locally Upon receive m –place in message buffer Deliver m from p j from buffer if –VC[j] = m.VC[j] – 1 –forall k≠j : VC[k] ≥ m.VC[k] Upon deliver –VC[j] := VC[j] + 1 VC[j] is the number of messages of p j that causally precede p i ’s subsequent messages FIFO

17. Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring 2008 17 Example Exam Question – Cont. סמנו את ה-Vector Clock שהיה מצורף לכל הודעה אם היינו משתמשים בהם. ציינו מתי (באיזה t) כל תהליך מבצע delivery להודעות m1 ו m3. אם בריצה המתוארת לא מתבצע delivery ציינו זאת בטבלה. [0,0,0] [0,1,0] [1,1,0] [0,1,1] [1,1,1] [1,1,0] [1,2,1] [0,1,0] [0,1,1] [1,1,0] [0,1,1] [1,2,1] m3m1 p1 p2 p3 34 17 2 7 Delivery according to VC