Ordering and Consistent Cuts Presented by Chi H. Ho

Time, Clocks, and the Ordering of Events in a Distributed System Leslie Lamport

Introduction 2000 PODC Influential Paper Award Outline of the paper: not in presented order –Partial and Total Orderings –Logical and Physical Clocks –Clock and Strong Clock Conditions –Synchronize Physical Clocks Beyond…

“Happened Before” a  b : if –a and b are events in the same process and a comes before b, or –a is the send event of some message, and b is the receive event of the same message. Transitive: (a  b) & (b  c)  (a  c) Concurrent: ( a  b) & (b  a). Partial Ordering

Examples q 5  p 4 q 2  q 3 p 1  r 3 q 2 // p 2 q 2 // p 3 Partial Ordering

Logical Clock Clock Condition:  a,b: a  b  C(a) < C(b) Partial Ordering Implementation

Logical Clock Implementation Rules: –IR1: Each process P i increments C i between any two successive events. –IR2: If event a is the sending of a message m by process P i, then the message contains a timestamp T m = C i (a). Upon receiving a message m, process P j sets C j greater than or equal to its present value and greater than T m. Partial Ordering Implementation

Examples Partial Ordering Implementation P0P0 P1P1 0

Examples Partial Ordering Implementation P0P0 P1P1 0 0

Examples Partial Ordering Implementation P0P0 P1P

Examples Partial Ordering Implementation P0P0 P1P

Examples Partial Ordering Implementation P0P0 P1P

Examples Partial Ordering Implementation P0P0 P1P [3]

Examples Partial Ordering Implementation P0P0 P1P [3] 2 [2]

Examples Partial Ordering Implementation P0P0 P1P [3] 2 [2] 4

Examples Partial Ordering Implementation P0P0 P1P [3] 2 [2] 4 4

Extended “Happened Before” a => b: iff –C i (a) < C j (b), or –(C i (a) = C j (b)) & (P i ≺ P j ) Total Ordering

Example Application Shared resource granting –Fixed number of processes –Single shared resource –Requirements: I. Mutual Exclusive II. Fair III. Exhaustive Total Ordering

Example Application Solution: Distributed algorithm Model: –Channels are FIFO –Each process maintains a process queue Algorithm –Request: broadcast T m :P i request resource –Release: broadcast T m :P i release resource –Receive request: enqueue –Receive release: dequeue –Resource granted (local decision): P i T m :P i request resource w/ T m min P i has received from every process a msg timestamped later than T m Note: –Can be generalized to solve Replicated State Machine! Total Ordering

Anomaly Amazon.com [19]

Anomaly Amazon.com [19]

Anomaly Amazon.com [19] [7]

Anomaly Amazon.com [19] [7]

Anomaly Amazon.com [19] [7] External event

Strong Clock Condition S = {events in the system} S = S ⋃ {relevant external events}  is “happened before” for S ∀ a,b ∈ S : a  b  C(a) < C(b) Avoid Anomaly

Physical Clocks PC1: (drift rate bound) ∃  << 1 such that ∀ i: |dC i (t)/dt – 1| <  PC2: (drift bound)  i,j: |C i (t) – C j (t)| < 

Avoid Anomaly  < shortest msg transmission time ∀ i,j,t: C i (t+  ) – C j (t) > 0 Physical Clocks   /(1-  )   Amazon.com j i C j (t) > C i (t+  ) > 

Implementation Rules IR1’: –For each i, if P i does not receive a message at physical time t, then C i is differentiable at t and dC i (t)/dt > 0. IR2’: –(a) If P i sends a message m at physical time t, then m contains a timestamp T m = C i (t). –(b) Upon receiving a message m at time t’, process P j sets C j (t’) equal to maximum (C j (t’-0), T m +  m ) Physical Clocks

Synchronize Physical Clocks Physical Clocks Problem statement: –IR1’ and IR2’ are followed, –Message delay is bounded, –Clocks satisfied PC1, –Goal: PC2 Algorithm: –Every  seconds, a message is sent over every arc. Guarantees: –Clocks are synchronized after t 0 +  d –   d(2  +  )

Beyond… Shortcomings: –No gap-detection property – C(a) < C(b)  ??? –Bounds are not practical (So is PC!)

Gap Detection Property Problem statement: –Given: a, b, C(a), C(b), C(a) < C(b), –Determine if c exists, where C(a) < C(c) < C(b) ? Beyond…

Another Strong Clock Condition a  b  C(a) < C(b) Beyond…

What clock, then? Causal histories: Beyond… Vector Clocks:

More on Vector Clocks  Strong Clock Condition  Concurrent  Pair-wise Inconsistent  Consistent Cut  Counting  Gap Detection Beyond…

More on Vector Clocks  Strong Clock Condition  Concurrent  Pair-wise Inconsistent  Consistent Cut  Counting  Gap Detection, but… Beyond… X  Weak Gap-Detection Given a, b, can detect existence of c such that (c  a) & (c  b)

Reference O. Babaoglu and K. Marzullo. Consistent global states of distributed systems: Fundamental concepts and mechanisms. In Sape Mullender, editor, Distributed Systems, ch. 4, pages Addison Wesley, 2nd ed., tent.html tent.html Note: some materials in this paper are used to clarify a few concepts in the next paper. Beyond…

Distributed Snapshots: Determining Global States of Distributed Systems K. Mani Chandy Leslie Lamport

Introduction Outline of the paper: –Motivation –Model –Algorithm –Correctness –Other issues Beyond…

Motivation Capture the global state of a system. Really? True global state: Impossible!!! p1p1 e11e11 e12e12 e13e13 e14e14 e15e15 e16e16 p2p2 e21e21 e22e22 e23e23 e24e24 e25e25

p1p1 e11e11 e12e12 e13e13 e14e14 e15e15 e16e16 p2p2 e21e21 e22e22 e23e23 e24e24 e25e25 Motivation Capture the global state of a system. Really? These are what can be done Are they useful?

p1p1 e11e11 e12e12 e13e13 e14e14 e15e15 e16e16 p2p2 e21e21 e22e22 e23e23 e24e24 e25e25 Motivation Capture the global state of a system. Useful? Equivalent! p1p1 e11e11 e12e12 e13e13 e14e14 e15e15 e16e16 p2p2 e21e21 e22e22 e23e23 e24e24 e25e25

p1p1 e11e11 e12e12 e13e13 e14e14 e15e15 e16e16 p2p2 e21e21 e22e22 e23e23 e24e24 e25e25 Motivation Capture the global state of a system. Useful? Consistent, but not happens in reality. p1p1 e11e11 e12e12 e13e13 e14e14 e15e15 e16e16 p2p2 e21e21 e22e22 e23e23 e24e24 e25e25

p1p1 e11e11 e12e12 e13e13 e14e14 e15e15 e16e16 p2p2 e21e21 e22e22 e23e23 e24e24 e25e25 Motivation Capture the global state of a system. Useful? Not even consistent! p1p1 e11e11 e12e12 e13e13 e14e14 e15e15 e16e16 p2p2 e21e21 e22e22 e23e23 e24e24 e25e25

Motivation Capture the global state of a system. Useful? Yes: –To detect stable properties of a system: y(S)  y(S’) for all S’ reachable from S. –E.g.: “computation has terminated,” “the system is deadlocked,” “all tokens in a token ring have disappeared.”

Model A distributed system A distributed system (on the right). A global state = set of processes’ and channels’ states. Event: –atomic –e = Computation: –seq =(e i : 0  i  n) –S i+1 = next(S i, e i ) Channels’ assumptions: –Singly directed –FIFO –Asynchronous –Error free –Infinite buffer

Algorithm Invoker: behave as if receiving a marker from a virtual node. Receiving rule for process q receiving a marker along channel c : if q has not recorded its state then begin q records its state; q records the state c as the empty sequence end else q records the state of c as the sequence of messages received along c after q’s state was recorded and before q received the marker along c. Sending rule for a process p : for each outgoing channel c : p sends one marker along c after p records its state and before p sends further messages along c.

Illustration Next 14 slides, courtesy of Professor Birman.

Chandy/Lamport p q r s t u v w x y z A network

Chandy/Lamport p q r s t u v w x y z A network I want to start a snapshot

Chandy/Lamport p q r s t u v w x y z A network p records local state

Chandy/Lamport p q r s t u v w x y z A network p starts monitoring incoming channels

Chandy/Lamport p q r s t u v w x y z A network “contents of channel p- y”

Chandy/Lamport p q r s t u v w x y z A network p floods message on outgoing channels…

Chandy/Lamport p q r s t u v w x y z A network

Chandy/Lamport p q r s t u v w x y z A network q is done

Chandy/Lamport p q r s t u v w x y z A network q

Chandy/Lamport p q r s t u v w x y z A network q

Chandy/Lamport p q r s t u v w x y z A network q z s

Chandy/Lamport p q r s t u v w x y z A network q v z x u s

Chandy/Lamport p q r s t u v w x y z A network q v w z x u s y r

Chandy/Lamport p q r s t u v w x y z A snapshot of a network q x u s v r t w p y z Done!

Correctness Consistency Termination

Consistency m is recorded iff so is send(m) : –sender’s state recording and marker sending are done atomically. m is not recorded more than once: –if channel is recorded before receiver, it will be empty. –if channel is recorded after receiver, none of the in-channel messages will be recorded as the receiver’s state. Correctness:

Termination Assumptions: –L1: no marker remains forever in a channel. –L2: processes’ states are recorded in finite time. Every process either spontaneously records its state, or there is a path from such a process. Every channel is flushed by a marker after the sender records its state. Correctness:

Remained Issues Property of recorded state: S i --> S * --> S f Stable detection: –Stable property: y(S i )  definite definite  y(S f ) –Algorithm: begin record a global state S * ; definite := y(S * ) end.

Beyond… Channels’ assumptions: –Singly directed –FIFO –Asynchronous –Error free –Infinite buffer

Non-FIFO What is FIFO for? –Separate messages between before-snapshot and after- snapshot. A snapshot counter piggybacked on messages would do just fine! Beyond:

Beyond… Channels’ assumptions: –Singly directed –FIFO –Asynchronous –Error free –Infinite buffer Messages can be corrupted/duplicated Messages can be dropped

Unreliable channels How to deal with corruption? –Checksum/ECC; reduced to drop. How to deal with duplication? –Message ID How to deal with dropping? –Channel states are not needed anymore. –Markers indicate completion. Beyond:

Even More Aggressive… Don’t want to piggyback! Step 1: no piggybacking: –Block all messages sent after recording local state and before receiving marker from all neighbors. Step 2: no blocking, min piggybacking –Blocked messages are sent with piggybacked snapshot info. Beyond:

Conclusion Two influential papers. Much work built upon these results. Can be improved significantly when being adopted to particular systems. Additional comments/suggestions?