Timed Quorum Systems … for large-scale and dynamic environments Vincent Gramoli, Michel Raynal

OPODIS’07Gramoli, Raynal Context Large-scale dynamic distributed systems

OPODIS’07Gramoli, Raynal Context Large-scale dynamic distributed systems Nodes communicate through message-passing

OPODIS’07Gramoli, Raynal Context Large-scale dynamic distributed systems Nodes communicate through message-passing Nodes join/leave the system at any time

OPODIS’07Gramoli, Raynal Goal To emulate a shared-memory in this context read write

OPODIS’07Gramoli, Raynal Goal To emulate a shared-memory in this context Providing atomic (i.e. linearizable) read/write operations read write

OPODIS’07Gramoli, Raynal Roadmap 1.Model and preliminary definitions 2.Related work 3.Timed Quorum System (TQS) 4.An efficient implementation of TQS 5.Conclusion

OPODIS’07Gramoli, Raynal Simple model System of n interconnected nodes with unique IDs Asynchronous communication with neighbors (nodes whose ID is known) Dynamism intensity (i.e. churn) c We consider a single object (local atomicity)

OPODIS’07Gramoli, Raynal Quorum System Quorums are sets (of nodes) that mutually intersect. A Quorum System (QS) is a set of quorums. Q1 Q2 Q3 Q1 ∩ Q2 ≠ Ø Q1 ∩ Q3 ≠ Ø Q2 ∩ Q3 ≠ Ø Ex. 3 quorums of size q=2

OPODIS’07Gramoli, Raynal Operations Atomic quorum-based operations for static settings: [Attiya, Bar-Noy, Dolev, JACM 1996] Each node of the quorums maintains: – A local value v of the object – A unique tag t, the version number of this value

OPODIS’07Gramoli, Raynal 1) Reading a value Q1 Q2 Q3 value? tag? v1,t1 Operations Phase 1: Consult the most up-to-date value v

OPODIS’07Gramoli, Raynal Operations 1) Reading a value Q1 Q2 Q3 v1,t1 Phase 1: Consult the most up-to-date value v Phase 2: Propagate the consulted value

OPODIS’07Gramoli, Raynal Operations 1) Reading a value Q1 Q2 Q3 v1,t1 Phase 1: Consult the most up-to-date value v Phase 2: Propagate the consulted value Theorem of Attiya and Welch 1998: « Read must write » to prevent new/old inversions for unbounded # of readers.

OPODIS’07Gramoli, Raynal Operations 1) Reading a value Q1 Q2 Q3 Output: v1 Phase 1: Consult the most up-to-date value v Phase 2: Propagate the consulted value

OPODIS’07Gramoli, Raynal Operations 2) Writing a value v2 Q1 Q2 Q3 Input: v2

OPODIS’07Gramoli, Raynal Operations 2) Writing a value v2 Q1 Q2 Q3 max tag? t1 Phase 1: Consult the value version and choose a new one strictly larger

OPODIS’07Gramoli, Raynal Operations 2) Writing a value v2 Q1 Q2 Q3 v2,t2 (with t2 > t1) Phase 1: Consult the value version and choose a new one strictly larger Phase 2: Propagate the new value associated with the new version

OPODIS’07Gramoli, Raynal Dynamic Solutions Reconfigurable storage: a failing QS is replaced by a new one. –RAMBO: Shvartsman, Lynch, DISC’02 –RDS: Chockler et al. OPODIS’05 Structured dynamic quorums: failed servers are replaced by new ones. –AM05: Abraham, Malkhi, Dist. Comp –NN05: Nadav, Naor, DISC’05 –SQUARE: Gramoli, Anceaume, Virgillito, SAC’07

OPODIS’07Gramoli, Raynal Dynamic Solutions Reconfigurable storage: a failing QS is replaced by new one. –RAMBO: Shvartsman, Lynch, DISC’02 –RDS: Chockler et al. OPODIS’05 Structured dynamic quorums: failed servers are replaced by new ones. –AM05: Abraham, Malkhi, Dist. Comp –NN05: Nadav, Naor, DISC’05 –SQUARE: Gramoli, Anceaume, Virgillito, SAC’07 All solutions require bounded churn during any finite period

OPODIS’07Gramoli, Raynal Dynamic Solutions Reconfiguration complexity vs. operation latency tradeoff RAMBO RDS reconfiguration complexity operation latency SQUARE Prevents scalability! AM05 NN05

OPODIS’07Gramoli, Raynal Timed Quorum System Dynamic quorum systems should be: –Probabilistic: # of failures not necessarily bounded –Timed: no property can hold forever

OPODIS’07Gramoli, Raynal Timed Quorum System Timed access strategy ω: A mapping from any time t to a probability distribution on the possible quorums. Δ-Timed Quorum System (TQS): For any Q 1 and Q 2 accessed resp. with ω(t 1 ) and ω(t 2 ), if |t 2 – t 1 | ≤ Δ, then Q 1  Q 2 ≠ Ø with high probability.

OPODIS’07Gramoli, Raynal Timed Quorum System Δ-Timed Quorum System (TQS): For any Q(t 1 ) and Q(t 2 ) accessed resp. with ω(t 1 ) and ω(t 2 ): if |t 2 – t 1 | ≤ Δ, then Q(t 1 )  Q(t 2 ) ≠ Ø with high probability. Time Q(t1) Q(t2) Q(t3) Q(t4) Q(t5) Q(t1)  Q(t2)Q(t2)  Q(t3)Q(t3)  Q(t4)Q(t3)  Q(t5) Δ Example of a TQS: {Q(t1),Q(t2),Q(t3),Q(t4),Q(t5)}

OPODIS’07Gramoli, Raynal Consistency Probabilistic Atomicity: –In the real-time sequence of operations: Any operation verifies atomicity w.r.t. all preceding successful operations, and it is said successful Or this operation is said unsuccessful –Any operation is successful with high probability

OPODIS’07Gramoli, Raynal Theorem 1: If at least one quorum is accessed every Δ period of time, then Δ-TQS implements probabilistic atomicity. Consistency Probabilistic Atomicity: –In the real-time sequence of operations: Any operation verifies atomicity w.r.t. all preceding successful operations, and it is said successful Or this operation is said unsuccessful –Any operation is successful with high probability

OPODIS’07Gramoli, Raynal Some observations Replication is necessary for data persistence In large-scale systems, operations are frequent Theorem « read must write » of Attiya and Welch indicates that some information must be replicated in any operation

OPODIS’07Gramoli, Raynal Efficient TQS Implementation Underlying gossip-based shuffle of neighborhood: –Each node has constantly a new random set of neighbors Classical quorum-based operations: –Consulting v and t at some quorum –Choosing v’ and t’ to propagate –Propagating v’ and t’ to some quorum

OPODIS’07Gramoli, Raynal Efficient TQS Implementation Disseminate until q = O(  n) nodes are contacted 1 k k 1 k1 Client 1 k l

OPODIS’07Gramoli, Raynal Efficient TQS Implementation Assumptions: –neighbors are chosen uniformly at random –at least one operation succeeds every Δ time –c = rate of arrival = rate of departure  [0,1) Results: –This algorithm implements a TQS

OPODIS’07Gramoli, Raynal Efficient TQS Implementation Assumptions: –neighbors are chosen uniformly at random –at least one operation succeeds every Δ time –c = rate of arrival = rate of departure  [0,1) Results: –This algorithm implements a TQS –Replication is piggybacked into operations

OPODIS’07Gramoli, Raynal Efficient TQS Implementation Assumptions: –neighbors are chosen uniformly at random –at least one operation succeeds every Δ time –c = rate of arrival = rate of departure  [0,1) Results: –This algorithm implements a TQS –Replication is piggybacked into operations –The quorum size is O(  nD ) where D = (1-c) -Δ

OPODIS’07Gramoli, Raynal Efficient TQS Implementation Assumptions: –neighbors are chosen uniformly at random –at least one operation succeeds every Δ time –c = rate of arrival = rate of departure  [0,1) Results: –This algorithm implements a TQS –Replication is piggybacked into operations –The quorum size is O(  nD ) where D = (1-c) -Δ –The operation latency is O(log k  nD ) message delays, where D = (1-c) -Δ

OPODIS’07Gramoli, Raynal Efficient TQS Implementation Assumptions: –neighbors are chosen uniformly at random –at least one operation succeeds every Δ time –c = rate of arrival = rate of departure  [0,1) Results: –This algorithm implements a TQS –Replication is piggybacked into operations –The quorum size is O(  nD ) where D = (1-c) -Δ –The operation latency is O(log k  nD ) message delays, where D = (1-c) -Δ –Smallest quorum size O(  n) for static systems when D=O(1) cf. [Malkhi, Reiter, Wool, Wright, Inf. and Comp. Journal 2001]

OPODIS’07Gramoli, Raynal Conclusion We defined Timed Quorum System that: Is inherently dynamic: –NO underlying structure –Timely intersection requirement Ensures Probabilistic Atomicity Scales well: –O(  nD) messages by operation –O(log k  nD) time by operation Is optimal: –When D=O(1), translates into best known static result: O(  n)

OPODIS’07Gramoli, Raynal Open Issue TQS in Mobile Sensor Networks: –Consultation phase: Gather motes to consult t and v Scatter motes to make t and v likely visible –Propagation phase: Gather motes to propagate t’ and v’ Scatter motes to make t’ and v’ likely visible