Distributed Programming and Consistency: Principles and Practice Peter Alvaro Neil Conway Joseph M. Hellerstein UC Berkeley

Part I: Principles

Motivation

Why are distributed systems hard? Uncertainty In communication

Why are distributed systems hard? We attack at dawn ? ?

Why are distributed systems hard? Wait for my signal Then attack!

Why are distributed systems hard? Attack! No, WAIT! ?

Distributed systems are easier when messages are

Reorderable

Distributed systems are easier when messages are Reorderable Retryable

Distributed systems are easier when messages are Reorderable Retryable Retraction-free

(notes) Point to make: convergent objects are NOT retraction-free.

Context: replicated distributed systems Distributed: connected (but not always, or well) Replicated: redundant data

Context: replicated distributed systems Running example: a key-value store put(“dinner”, “pizza”) Dinner = pizza get(“dinner”) “pizza”

Context: replicated distributed systems Distributed  replication is desirable Distributed  consistency is expensive

Consistency? Definitions abound: pick one. but try to be consistent…

What isn’t consistent? Replication anomalies: Read anomalies (staleness) Write divergence (concurrent updates)

Anomalies Stale reads put(“dinner”, “pasta”) Dinner = pizza get(“dinner”) “pizza” Dinner = pasta ?

Anomalies Write conflicts put(“dessert”, “cake”) Dessert = cake Dessert = fruit put(“dessert”, “fruit”) ?

Consistency Anomalies witness replication. A consistent replicated datastore rules out (some) replication anomalies.

Consistency models Strong consistency Eventual consistency Weaker models

Strong consistency AKA ``single copy’’ consistency Replication is transparent; no witnesses of replication

Strong consistency Reads that could witness replication block Concurrent writes take turns

Strong consistency Some strategies: Single master with synchronous replication – Writes are totally ordered, reads see latest values Quorum systems – A (majority) ensemble simulates a single master ``State machine replication’’ – Use consensus to establish a total order over reads and writes systemwide

Strong consistency Drawbacks: Latency, availability, partition tolerance

Eventual Consistency Tolerate stale reads and concurrent writes Ensure that eventually* all replicas converge * When activity has ceased and all messages are delivered to all replicas

Eventual Consistency Strategies: Establish a total update order off critical path (eg bayou). – Epidemic (gossip-based) replication – Tentatively apply, then possibly retract, updates as the order is learned

Eventual Consistency Strategies: Deliver updates according to a ``cheap’’ order (e.g. causal). – Break ties with timestamps, merge functions hmmmm

Eventual Consistency Strategies: Constrain the application so that updates are reorderable Won’t always work. When will it work?

Eventual consistency – more definitions Convergence – Conflicting writes are uniformly resolved – Reads eventually return current data – State-centric Confluence – A program has deterministic executions Output is a function of input – Program-centric

Eventual consistency – more definitions Confluence is a strong correctness criterion – Not all programs are meant to be deterministic But it’s a nice property – E.g., for replay-based debugging – E.g., because the meaning of a program is its output (not its ``traces’’) Confluent => convergent

Eventual consistency – more definitions Confluent => convergent Deterministic executions imply replica agreement

Eventual consistency – more definitions But convergent state does not imply deterministic executions

(peter notes) EC systems focus only on controlling write anomalies (stale reads always fair game, though session guarantees may restrict which read anomalies can happen) EC systems are convergent – eventually there are no divergent states Deterministic => convergent (but not tother way) Determinism is compositional: two deterministic systems glued together make a deterministic system Convergence is not compositional: two convergent systems glued together do not necessarily make a convergent system (eg if the glue is NM)

(peter notes) Guarded asynchrony – need to carefully explain the significance of this Essentially, confluent programs cannot allow one-shot queries on changing state (even monotonically changing) One-shot queries must be converted into subscriptions to a stream of updates – That way, we are guaranteed to see the last update to a given lattice

(joe notes) There’s stuff you can do in The storage layer…. Or you can pop up. Layer vs language Sequential emulation at the storage layer Crdts – state-centric attempt to achieve relaxed ordering (object-by-object) Then bloom The programming model should match the computation model

Distributed design patterns for eventual consistency

ACID 2.0 The classic ACID has the goal to make the application perceive that there is exactly one computer and it is doing nothing else while this transaction is being processed. Consider the new ACID (or ACID2.0). The letters stand for: Associative, Commutative, Idempotent, and Distributed. The goal for ACID2.0 is to succeed if the pieces of the work happen:  At least once,  Anywhere in the system,  In any order. - Pat Helland, Building on quicksand

ACID 2.0 Associative -- operations can be ``eagerly’’ processed Commutative – operations can be reordered Idempotent – retry is always an option Distributed – (needed a ``D’’)

ACID 2.0 Instead of low-level reads and writes programmers use an abstract vocabulary of reorderable, retryable actions: Retry – a mechanism to ensure that all messages are delivered Reorderability -- ensures that all replicas converge

Putting ACID 2.0 into practice

1.CRDTs – A state-based approach – Keep distributed state in data structures providing only ACI methods 2.Disorderly programming – A language-based approach – Encourage structuring computation using reorderable statements and data

Formalizing ACID 2.0 ACI are precisely the properties that define the LUB operation in a join semilattice If states form a lattice, we can always merge states using the LUB.

C(v)RDTs Convergent Replicated Datatypes Idea: represent state as a join semilattice. Provide a ACI merge function

CRDTs Data structures: 1.Grow-only set (Gset) – Trivial – merge is union and union is commutative 2.2PSet – Two Gsets – one for adds, the other for tombstones – Idiosyncrasy: you can only add/delete once. 3.Counters 1.Tricky! Vector clock with an entry for each replica replica I => VC[i] += 1 3.Value: sum of all VC values

CRDTs Difficulties: Convergent objects alone are not strong enough to build confluent systems.

Asynchronous messaging You never really know

Asynchronous messaging AB C send AB C

Monotonic Logic The more you know, the more you know.

Monotonic Logic AB C E D A C E select/ filter

Monotonic Logic AB C project / map f(A)f(B) f(C)

Monotonic Logic A B C D E B D B D join / compose F

Monotonic Logic is order-insensitive AB C E D A C E

Monotonic Logic is pipelineable AB C E D A C E

Nonmonotonic Logic When do you know for sure?

Nonmonotonic Logic A B C D E B D set minus A B C D E Retraction! X Y Z

Nonmonotonic logic is order-sensitive A B C D E B D set minus A C E

Nonmonotonic logic is blocking A set minus A ? A?

Nonmonotonic logic is blocking A set minus A ? ``Sealed’’

CALM Analysis Asynchrony => loss of order Nonmonotonicity => order-sensitivity Asynchrony ; Nonmonotonicity => Inconsistency […]

CALM Analysis Asynchrony => loss of order Nonmonotonicity => order-sensitivity Asynchrony ; Nonmonotonicity => Inconsistency ? ``Point of Order’’ […]

Disorderly programming An aside about logic programming: In (classical) logic, theories are Associative and commutative – Consequences are the same regardless of the order in which we make deductions Idempotent – Axioms can be reiterated freely

Disorderly programming An aside about logic programming: In (classical) logic, theories are Associative, Commutative, and Idempotent because Knowledge is monotonic: The more you know, the more you know

Disorderly programming An aside about logic programming: It is challenging to even talk about order in logic programming languages [dedalus]. Yet we can build …

Disorderly programming Idea: embody the ACID 2.0 design patterns in how we structure distributed programs. Disorderly data: unordered relations Disorderly code: specify how data changes over time

Bloom

Bloom Rules do |mes, mem| [mem.address, mes.id, mes.payload] end multicast<~(message * members)

Operational model

Time Set (Union) Integer (Max) Boolean (Or) “Growth”: Larger Sets “Growth”: Larger Numbers “Growth”: false  true

Time Set (merge = Union) Integer (merge = Max) Boolean (merge = Or) size() >= 5 Monotone function from set  max Monotone function from max  boolean

Builtin Lattices NameDescription?a t bSample Monotone Functions lboolThreshold testfalse a ∨ b when_true() ! v lmaxIncreasing number 1max(a,b ) gt(n) ! lbool +(n) ! lmax -(n) ! lmax lminDecreasing number −1−1min(a,b)lt(n) ! lbool lsetSet of values;a [ bintersect(lset) ! lset product(lset) ! lset contains?(v) ! lbool size() ! lmax lpsetNon-negative set;a [ bsum() ! lmax lbagMultiset of values;a [ bmult(v) ! lmax +(lbag) ! lbag lmapMap from keys to lattice values empty map at(v) ! any-lat intersect(lmap) ! lmap 72

Quorum Vote in Bloom L QUORUM_SIZE = 5 RESULT_ADDR = "example.org" class QuorumVote include Bud state do channel :vote_chn, :voter_id] channel :result_chn, lset :votes lmax :vote_cnt lbool :got_quorum end bloom do votes <= vote_chn {|v| v.voter_id} vote_cnt <= votes.size got_quorum <= vote_cnt.gt_eq(QUORUM_SIZE) result_chn <~ got_quorum.when_true { [RESULT_ADDR] } end Map set ! max Map max ! bool Threshold test on bool Lattice state declarations 73 Communication interfaces Accumulate votes into set Annotated Ruby class Program state Program logic Merge function for set lattice

Convergence – a 2PSet

The difficulty with queries ``The work of multiple transactions can interleave as long as they are doing the commutative operations. If any transaction dares to READ the value, that does not commute, is annoying, and stops other concurrent work.’’ -- Pat Helland