Declarative Programming over Eventually Consistent Data Stores KC Sivaramakrishnan Gowtham Kaki Suresh Jagannathan

Http AppServer Cache Stateless Consistency, Integrity, Durability, Availability, etc.

Account balances should be non-negative Usernames should be unique Only bona fide bids are accepted in an auction. Application invariants Strong consistency Linearizability & Serializability

INTERNET ☐ Strongly consistent, but not “always on” ☐ Be “always on”, but no strong consistency Eventual Consistency ∞ (convergence)

INTERNET Basic eventualRead-my-writesCausal Monotonic writes Bounded staleness Read committed Parallel Snapshot Isolation Store Consistency Levels Eventually Consistent Data Stores Read-my-writes //init balance = 0 deposit(100) ?  get_balance() Session 1

INTERNET Basic eventualRead-my-writesCausal Monotonic writes Bounded staleness Read committed Parallel Snapshot Isolation Store Consistency Levels Eventually Consistent Data Stores Read-my-writes //init balance = 0 deposit(100) 0  get_balance() Session 1 bal=0 Replica 2 bal=100 Replica 1 Eventual Consistency //init balance = 0 deposit(100) ???  get_balance() Session 1 bal=100 Replica 1 Read-my-writes consistency bal=0 Replica 2

INTERNET Basic eventualRead-my-writesCausal Monotonic writes Bounded staleness Read committed Parallel Snapshot Isolation Store Consistency Levels Eventually Consistent Data Stores Read-my-writes //init balance = 0 deposit(100) 0  get_balance() Session 1 bal=0 Replica 2 bal=100 Replica 1 Eventual Consistency //init balance = 0 deposit(100) 100  get_balance() Session 1 bal=100 Replica 1 Read-my-writes consistency bal=100 Replica 2

INTERNET Basic eventualRead-my-writesCausal Monotonic writes Bounded staleness Read committed Parallel Snapshot Isolation Store Consistency Levels Eventually Consistent Data Stores deposit()withdraw()bid()tweet() Application invariants

INTERNET Basic eventualRead-my-writesCausal Monotonic writes Bounded staleness Read committed Parallel Snapshot Isolation Store Consistency Levels Eventually Consistent Data Stores deposit()withdraw()bid()tweet() Application invariants Can we automate the process of mapping application requirements to store consistency levels?

Application requirements Store consistency guarantees Unique usernames. Non-negative balance. Bona fide bids. Read-my-writes consistency Causal consistency Read committed isolation level Repeatable read isolation level Map Specification Language Classification Scheme Our solution … Sound. Optimal A common medium to express both. An algorithm to …

Deposit(200) Withdraw(20) Withdraw(10) …… Deposit(200) Withdraw(20) Withdraw(10) …… Replica 1 Prelims - System Model …… Deposit(200) Withdraw(10) …… Deposit(200) Withdraw(10) …… Replica n Session 1 v1 = getBalance(); …… v2 = getBalance(); v1 = getBalance(); …… v2 = getBalance(); Session n …… Replicated Data Store Session Order (SO) getBalance Visibility (Vis) getBalance Vis

Primitive relations Specification Language Happens-before Axiomatically capture set of valid executions Associate with each operation a single abstract effect –Express relationship between effects –Visibility (vis), Session order (so), Same object (sameobj) Per-object session order

Replicated Bank Account (1) balance >= 0 violated Session 2 //init balance = 100 withdraw(70); //init balance = 100 withdraw(70); //init balance = 100 withdraw(70); //init balance = 100 withdraw(70); Session 1 a a b b vis Alice Bob

Bank Account Contracts (2) Session 1 deposit(100) deposit(100) a a Session 2 withdraw(50) withdraw(50) b b vis Session  A.getbalance () -50  A.getbalance () getbalance () getbalance () c c vis 50  getbalance () 50  getbalance () Alice Bob Cheryl

Various consistency levels offered by stores can be specified.

Causal Visibility Causal Visibility (CV) is the strongest known guarantee that can be ensured under eventual consistency with high availability. We therefore consider causal visibility formula as the specification for eventual consistency ( ). c c b b vis a a hbo vis

Causal Consistency Causal consistency (CC) guarantees that writes from a session are made visible to subsequent reads in the same session. CV doesn’t guarantee this. Hence CC is stronger than CV Therefore it cannot be achieved with high availability under EC. Still, it is better than SC. b b a a hbo vis

Sequential Consistency Strong/Sequential Consistency (SC) guarantees that all distinct operations performed on a same object are totally ordered w.r.t the visiblity relation. SC is the strongest consistency level we consider. It is most expensive in terms of availability. b b a a vis

Capturing Store Consistency Levels Strong Consistency Causal Consistency Eventual Consistency

Decidable  Automatically discharged with the help of Z3 SMT solver. Classification Scheme deposit  EC withdraw  SC getBalance  CC Strong Consistency Causal Consistency Eventual Consistency

Our classification scheme is parametric with respect to the lattice of consistency levels.

Classification Scheme (2)

Transactions Full atomicity can be ensured without affecting availability. Real applications often use transactions –To write to multiple objects atomically –To read consistent state of objects in isolation. Our specification language can be extended to specify isolation requirements. –Add single primitive relation - sametxn(a,b) –Derived relation: Various isolation requirements, on the other hand, affect availability in various ways.

Session 1 C.withdraw(100) S.deposit(100) C.withdraw(100) S.deposit(100) Alice a a C  W(100) …… C  W(100) R1R2Rn Replicated Data Store Replicated Bank Account (2) Bob Reads inconsistent state!

Session 1 transaction (“save”) { C.withdraw(100) S.deposit(100) } transaction (“save”) { C.withdraw(100) S.deposit(100) } Alice a a …… C  W(100) S  D(100) R1R2Rn Replicated Data Store Replicated Bank Account (2) Bob Reads consistent state. b b Bob “save” committed.

Atomicity of writes itself is not sufficient. We need transactions to ensure proper isolation for reads.

Session 1 transaction (“save”) { C.withdraw(100) S.deposit(100) } transaction (“save”) { C.withdraw(100) S.deposit(100) } Alice Bob Session 2 x  C.getBalance() y  S.getBalance() v  x + y x  C.getBalance() y  S.getBalance() v  x + y a a b b c c Replicated Bank Account (3) C  W(100) S  D(100) …… C  W(100) S  D(100) R1R2Rn Replicated Data Store d d “save” committed. 0  C.getBalance() y  S.getBalance() v  x + y 0  C.getBalance() y  S.getBalance() v  x + y 0  C.getBalance() 0  S.getBalance() 0   C.getBalance() 0  S.getBalance() 0  0 + 0

Session 1 transaction (“save”) { C.withdraw(100) S.deposit(100) } transaction (“save”) { C.withdraw(100) S.deposit(100) } Alice Bob Session 2 transaction (“totalBal”) { x  C.getBalance() y  S.getBalance() v  x + y } transaction (“totalBal”) { x  C.getBalance() y  S.getBalance() v  x + y } a a b b c c Replicated Bank Account (3) d d transaction (“totalBal”) { 0  C.getBalance() y  S.getBalance() v  x + y } transaction (“totalBal”) { 0  C.getBalance() y  S.getBalance() v  x + y } transaction (“totalBal”) { 0  C.getBalance() 100  S.getBalance() 100  } transaction (“totalBal”) { 0  C.getBalance() 100  S.getBalance() 100  } vis

Various transaction isolation levels provided by stores can be specified.

Read Committed Txn 1 (current) X.oper1(…) a a Txn 2 (committed) X.oper2(…) X.oper2(…) b b vis X.oper2(…) X.oper3(…) X.oper2(…) X.oper3(…) c c vis RC is the isolation level that directly follows the atomicity of committed transactions.

Repeatable Read All operations of an RR transaction witness the same set of (committed) transactions. i.e., Snapshot isolation Txn 1 (current) X.oper1(…) a a Txn 2 (committed) X.oper2(…) X.oper2(…) b b vis X.oper2(…) Y.oper4(…) X.oper2(…) Y.oper4(…) c c vis d d X.oper1(…) Y.oper3(…) X.oper1(…) Y.oper3(…) RR =

Monotonic atomic view Repeatable Read Read Committed Capturing Store Isolation Levels

Classification Scheme for Isolation Levels “Save” transaction  RC “totalBal” transaction  RR Read Committed (RC) Repeatable Read (RR) Monotonic Atomic View (MAV) BankAccount Transactions

Haskell library for Eventually Consistent Data Stores (ECDS) –Definition language  define operations and transactions on replicated data. –Specification language  specify consistency and isolation requirements. + DEFS Quelea Data Store GHC

ApplicationRUBiSTwitter-lite #Tables65 #Operations1720 #Transactions610 Invariants e.g.See all bids placed in current session Unique username Results of classification #EC Ops1413 #CC Ops26 #SC Ops11 #RC Txns46 #MAV Txns23 #RR Txns01 Case Studies

Conclusion Quelea  Haskell-library for programming ECDS –Automatic classification of operation and transaction contracts through SMT solver Leveraging off-the-shelf ECDS –Avoid re-engineering complex systems –Makes it practical!

Thank you!

Evaluation Correctness with classification vs without classification –How do they compare in terms of availability? Experimental Setup: –Amazon EC2; 5 replicas (StrongRep & Quelea); 1 replica (NoRep) –Gradually increased # of concurrent clients from 128 to NoRep: No Replication StrongRep: Strong Replication

Related Work S. Burckhardt et al, Replicated Data Types: Specification, Verification and Optimality, POPL’14. –Proposes the axiomatic execution model we use to build our specification language and classification scheme. P. Alvaro et al, Consistency Analysis in Bloom, a CALM and Collected Approach, CIDR’11. C. Li et al, Automating the Choice of Consistency Levels in Replicated Systems, USENIX ATC’14. V. Balegas et al, Putting the Consistency back into Eventual Consistency, EuroSys’15. –Propose analyses to identify the program points where coordination is required, and serialize them. Do not consider fine-grained consistency/isolation levels.

Future Work – Inferring Specifications (1) Our specification language is based on a low-level axiomatic system model. Crafting a specification requires –Comprehensive knowledge about the system model, & –Exhaustive reasoning about application semantics under this system model to determine possible anomalies. Can we reduce the programmer effort required to use Quelea by inferring low-level specifications? But, infer specifications based on what?

Future Work – Inferring Specifications (2) Infer low-level specifications based on application integrity specifications. –E.g: balance≥0 But, application requirements often cannot be stated as simple integrity specifications. –It’s ok for Alice to not see Bob’s tweet immediately after he posted it. –But, Alice must be able to see her tweets immediately in her timeline. –Also, if Bob tweeted in reply to Cheryl, then Alice cannot see Bob’s tweet without also seeing Cheryl’s.

Future Work – Inferring Specifications (3) Infer low-level specifications based on test cases. –Easier to write. Programmers write them all the time. –Execute/simulate the tests to get passing runs (good samples) and failing runs (bad simples). –Specification is the classifier that separates good samples from the bad samples. Challenge: how to overcome the overfit? –Take programmer assistance? –Ongoing research. Comments are welcome!

Future Work – Inferring Specifications (3) Infer low-level specifications based on test cases. –Easier to write. Programmers write them all the time. –Execute/simulate the tests to get passing runs (good samples) and failing runs (bad simples). –Specification is the classifier that separates good samples from the bad samples. Example: Infer monotonicity property of a monotonically increasing counter.

Example: A monotonically increasing counter

Summarization is essential to check the unbounded growth of the log. How is summarization done? –Ask developer for summarization semantics. –Replace (many) original effects with (few) summary effects. State Summarization