1. CS4231 Parallel and Distributed Algorithms AY 2006/2007 Semester 2 Lecture 6 Instructor: Haifeng YU

2. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 22 Review of Last Lecture  Chapter 9 “Global Snapshot”  What is our goal?  Formalizing consistent global snapshot  Protocol for capturing a consistent global snapshot

3. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 23 Today’s Roadmap  Chapter 12 “Message Ordering”  FIFO ordering  Already discussed last lecture  Causal ordering and its application  Total ordering and its application

4. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 24 A Internet Chat Room Example Alice (process1) Bob (process2) Cary (process3) Introducing Bob to Cary Hi Bob, I just introduced you to Cary Hi Cary, this is Bob ??? Sorry, I don’t know any Bob. The notion of “causal order”.

5. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 25 Formalizing Causal Order  If a “send” event s1 caused a “send” event s2  Causal order requires the corresponding receive event r1 be before r2 (r1 and r2 on the same process)  But how do we know whether s1 caused s2?  Based on “Happened-Before” relation  This is the only ordering observable to external users  If a “send” event s1 happens before a “send” event s2  Then s1 may have caused s2  Let’s be pessimistic and be safe -- Assume s1 indeed caused s2  Causal order: If s1 happens before s2, and r1 and r2 are on the same process, then r1 must be before r2

6. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 26 How to Ensure Causal Ordering – Intuition Alice (process1) Bob (process2) Cary (process3) Message1 A to C Message1 A to B PS: Message1 A to C PS: Message1 A to C Message1 A to B Message1 B to C

7. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 27 How to Ensure Causal Ordering – Protocol  Each process maintains n*n matrix M  M[i,j] denotes that number of messages sent from i to j, as known by the local process  If process i send a message to process j  On process i: M[i,j]++  Piggyback M on the message  Upon process j receiving a message from process i with matrix M’ piggybacked  If M[k,j]  M’[k,j] for all k except k = i, AND M[i,j] == M’[i,j] -1, then deliver message and M = pairwise-max(M,M’)  Else delay message

8. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 28 Example Run for the Protocol Alice (process1) Bob (process2) Cary (process3) Message1 B to C (0,0,1) (0,0,0) (0,1,1) (0,0,0) (0,1,1) (0,0,0) can deliver (0,0,0)  (0,1,1) (0,0,1) (0,0,0) can’t deliver (0,0,0) (0,1,1) (0,0,1) (0,0,0)

9. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 29 Correctness Proof of the Protocol  (The textbook does not have an explicit correctness proof.)  s1, s2, r1, r2, where s1 happened before s2  We want to prove that r1 is never delivered after r2  Consider the case where s1 and s2 are on different processes  W.l.o.g., assume s1 on process 1, s2 on process 2, r1 and r2 on process 3  We focus on the first 3 elements of the 3rd column of the matrices

10. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 210 Correctness Proof of the Protocol Alice (process1) Bob (process2) Cary (process3) x1 y1 0 x2 y2 0 x2  x1 y2  y1 x3 y3 0 x4 y4 0 x4  x3  x2 y4 > y3  y2 x1 y1 0 x4 y4 0

11. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 211 Correctness Proof of the Protocol (continued) Cary (process3) x1 y1 0 x4 y4 0 x4  x1 y4 > y1  Prove by contradiction: Assume Red delivered before Blue  After delivering Red: 3rd column of the matrix on process 3 will be  The values in the matrix never decreases: Blue can never be delivered  x4  y4 0

12. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 212 Need More Correctness Proof  What we proved so far:  If s1 happened before s2, then r1 is never delivered after r2 is delivered  We don’t know if r1 and r2 will be delivered at all!  We will now prove that  At any given time, there must be one message that can be delivered to the process – induction will take care of the other messages

13. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 213 More Correctness Proof  Consider a process and its corresponding column in the matrix  If there is any message (from process i) undelivered to the process, then there must be messages whose matrix has the column x_1 x_2 … x_n ? x_i + 1 ? call such message as successor messages

14. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 214 More Correctness Proof  Consider the non-empty set of all successor messages and their corresponding “send” events  There must be a send event s that does not have any other send events that happened before s  Call the corresponding message first successor message  Claim: Any first successor message can be delivered  W.l.o.g, assume the message is from process 1 x_1 x_2 … x_n x_1 + 1 y_2 … y_n The matrix column on the receiver The matrix column in the message need to show x_i  y_i for i between 2 and n W.l.o.g, assume i = 2

15. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 215 More Correctness Proof  Prove by contradiction: Suppose y_2 > x_2 process1 process2 Receiver process x_1 + 1 y_2 …  x_1 x_2 … The last delivered message from process 2 must be another un-delivered message this is not a first successor message then!

16. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 216 Summary of Correctness Proof  All messages will eventually be delivered  The delivery order satisfies causal order

17. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 217 Causal Ordering of Broadcast Messages  Broadcast: Every message is sent to all people (including the sender itself)  Modeled as n point-to-point messages  Application: Internet chat room  For same reason as before, we may need to ensure causal order among messages  Each process uses the previous protocol

18. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 218 A Simpler Protocol for Causal Ordering Broadcast  Each process maintains a message log for all messages delivered  When sending a message, append the message to log and send the whole log  Upon receiving a log, scan the log sequentially, for any message not seen before: Append the message to local log and deliver it  Can be used for point-to-point as well  But inefficient..

19. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 219 Total Ordering of Broadcast Messages  All messages delivered to all processes in exactly the same order  Also called atomic broadcast  Total ordering does not imply causal ordering  Causal ordering does not imply total ordering  Thus…there is something called “causal total order”

20. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 220 Application: Internet Chat Room  We want to assign numbers to people have said  The number have to be consistent across all users [0] Alice: Welcome! [1] Bob: Hello, Alice [2] Alice: Let’s try to prove P=NP [3] Bob: OK, how do we start? …. [1712] Bob: I am confused, what were we trying to prove? [1713] Alice: Please refer to message [2]

21. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 221 Application: Replicated State Machine  State machine can be an abstract of many things  E.g., A database  Typically deterministic  Replicated state machine  Multiple copies of the same state machine for performance / availability  Each “instruction” is broadcast to all state machines  The state machines must execution the “instruction” in exactly the same order!

22. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 222 Using a Coordinator for Total Order Broadcast  A special process is assigned as the coordinator  To broadcast a message  Send a message to the coordinator  Coordinator assigns a sequence number to the message  Coordinator forward the message to all processes with the sequence number  Messages delivered according to sequence number order  Problem:  Coordinator is performance bottleneck  What if coordinator fails?

23. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 223 Skeen’s Algorithm for Total Order Broadcast  Each process maintains  Logical clock and a message buffer for undelivered messages  A message in the buffer is delivered / removed if  All messages in the buffer have been assigned numbers  This message has the smallest number process 1 process 2 process 3 broadcast message acknowledge notify message number put msg in buffer reply with current logical clock value pick the max clock value received as message number

24. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 224 Correctness Proof for Skeen’s Algorithm  Claim: All messages will be assigned message numbers  Claim: All messages will be delivered  Claim: If message A has a number smaller than B, then B is delivered after A – Prove by contradiction (continue on next slide) process 1 process 2 process 3 broadcast message acknowledge notify message number put msg in buffer reply with current logical clock value pick the max clock value received as message number

25. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 225 Correctness Proof for Skeen’s Algorithm -- Continued  Suppose A is delivered on process 3 after B.  A must be placed in buffer after B is delivered  A must have a number larger than B – Contradiction. process 3 B placed in buffer B’s number received and B delivered A must be placed in buffer after B delivered reply for A key: Process 3’s logical clock now must be larger than B’s number

26. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 226 Summary  Chapter 12 “Message Ordering”  FIFO ordering for point-to-point messages  Already discussed last lecture  Causal ordering for point-to-point messages  Applications  Protocol to ensure causal ordering  Causal ordering for broadcast messages  Protocol  Total ordering for broadcast messages  Application  Skeen’s algorithm

27. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 227 Homework Assignment  Page 206, Problem 12.3. For each pair (x, y) of the three pairs (i.e., C1 & C2, C1 & C3, C2 & C3)  Prove x implies y and y implies x, OR  Prove x implies y, and a counter example that satisfies y but not x, OR  Prove y implies x, and a counter example that satisfies x but not y, OR  A counter example that satisfies y but not x, and a second counter example that satisfies x but not y  Think about why the protocol on slide 18 will deliver all messages in causal order  For the protocol on slide 22, think about whether the resulting total order satisfy causal order? Why?  Homework due a week from today  Read Chapter 13