1. FAUST: Fail-Aware Untrusted Storage Christian Cachin IBM Zurich Idit Keidar Technion Alex Shraer, Technion, Israel Joint work with:

2. Agenda Motivation Defining a Fail-Aware Untrusted Service FAUST: Fail-Aware Untrusted STorage Conclusions

3. Once upon a time… DSN paper Alice Bob Jlduiy baios kjshh jcouhc lj ljxc jcjjzxkcj hyah dg dgf gh hnbbv dg dgf gh hnbbv hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d ji khcgtu jnn mckpkk,c jjkpackm kj ihgdiash hasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg hays v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcg tujnn mckpkk,c jjkpackm kj ihgdiashhasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg dsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkp ackm kj ihgdiashhasdg hjy v fg djjiaj djsjd,lm cgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c packm kj ihgdiashhasdg gdsihasd jddfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb DSN paper Alice Bob Jlduiy baios kjshh jcouhc lj ljxc jcjjzxkcj hyah dg dgf gh hnbbv dg dgf gh hnbbv hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d ji khcgtu jnn mckpkk,c jjkpackm kj ihgdiash hasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg hays v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcg tujnn mckpkk,c jjkpackm kj ihgdiashhasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg dsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkp ackm kj ihgdiashhasdg hjy v fg djjiaj djsjd,lm cgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c packm kj ihgdiashhasdg gdsihasd jddfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb strong consistency (linearizability/atomicity) strong liveness (wait-freedom) Alice Bob Don't be Evil (DobeE) Inc. DobeE OnlineDocs sometimes (email)

4. But what if… Back in High school… Alice Bob DSN paper Alice Bob Jlduiy baios kjshh jcouhc lj ljxc jcjjzxkcj hyah dg dgf gh hnbbv dg dgf gh hnbbv hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d ji khcgtu jnn mckpkk,c jjkpackm kj ihgdiash hasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg hays v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcg tujnn mckpkk,c jjkpackm kj ihgdiashhasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg dsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkp ackm kj ihgdiashhasdg hjy v fg djjiaj djsjd,lm cgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c packm kj ihgdiashhasdg gdsihasd jddfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb DSN paper Alice Bob Jlduiy baios kjshh jcouhc lj ljxc jcjjzxkcj hyah dg dgf gh hnbbv dg dgf gh hnbbv hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d ji khcgtu jnn mckpkk,c jjkpackm kj ihgdiash hasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg hays v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcg tujnn mckpkk,c jjkpackm kj ihgdiashhasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg dsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkp ackm kj ihgdiashhasdg hjy v fg djjiaj djsjd,lm cgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c packm kj ihgdiashhasdg gdsihasd jddfg dgtrsd hjy v fg djjiaj djsjd,lmcgg nhycgygb

5. What can go wrong? Clients sign data with digital signatures Server cannot forge the data But answer with outdated value (“replay attack”) But send different values to different clients Previously captured by forking conditions [Mazieres&Shasha, Li et al., Oprea & Reiter, Cachin et al.]

6. What would we like to achieve? No bogus data –Still, inconsistencies are possible Alice and Bob to detect inconsistencies –Rollback to consistent state When server is correct (the common case) clients should have strong consistency and liveness for their operations –Linearizability and wait-freedom

7. Agenda Motivation Defining a Fail-Aware Untrusted Service FAUST: Fail-Aware Untrusted STorage Conclusions

8. Eventual Consistency Client’s operations can complete even though they are only “weakly” consistent Client is notified when its operations become “strongly” consistent –May invoke other operations without waiting for notifications Eventual consistency is widely accepted and used –Starting with Bayou (SOSP 95) –Today in commercial systems, e.g., Amazon’s Dynamo (SOSP 07)

9. Fail-Awareness [Fetzer & Cristian 96] Provide notifications when the service does not guarantee its specified semantics –means that the service shouldn’t be used Allows clients to take appropriate recovery actions

10. Fail-Aware Untrusted Service When the server is correct – strong consistency and strong liveness Execution is always “weakly” consistent –at least causal consistency Stability notifications (next slide) Accurate failure notifications Detection completeness: All operations eventually become stable unless failure is detected

11. Example with Correct Server Alice & Bob are from Europe, Carlos is from America Its daytime in Europe, and Carlos is asleep Alice’s “stability cut”: Stability w.r.t all clients = perfect consistency

12. Agenda Motivation Defining a Fail-Aware Untrusted Service FAUST: Fail-Aware Untrusted STorage Conclusions

13. Fork consistency [Mazières&Shasha, PODC 02] Server may present different views to clients  “Fork” their views of the history In such case, their views are forked ever after (never again see each others new updates)  or else, server is exposed as faulty

14. Fork-Consistency – an Example read(X 1 ) → null write(X 1,u) C1 read(X 3 ) → null C1C1 C2C2 C3C3 write(X 3,w) start (X 1 = X 2 = X 3 = null) read(X 1 ) → u read(X 3 ) → null read(X 1 ) → u write(X 3, w) read(X 3 ) → null C1C1 C3C3 C 2 is forked from C 1 C 1 is forked from C3 read(X 3 ) → null C 3 is forked from C 2 C2C2

15. What about the common case? C 2 must receive signed context of the write Must wait until the write commits What if the write crashes? Formal proof in [Cachin, Shelat, Shraer 07] read(X 1 ) → null write(X 1,u) C1 C1C1 C2C2 read(X 1 ) → u C 2 is forked from C 1 A Join – not allowed with fork consistency My context is: start (I am the first operation) My context is: start (I am the first operation) Something is wrong! REPLY: I scheduled op1, op2, … before you value, signed context of corresponding write COMMIT op (signed context) SUBMIT write operation ClientServer SUBMIT read operation REPLY: I scheduled op1, op2, … before you

16. We need a weaker consistency notion C2C2 Two weaker conditions have been defined recently: Fork-* [Li & Mazieres, NSDI 2007] Fork-sequential-consistency [Oprea & Reiter, DISC 2006] We show: both require blocking, in executions with 1 correct server

17. Weak fork At-most one join: If server forks the views of two clients, they may see at most one more common operation Same as in Fork-* Trust server about last operations by other clients in context No checks needed Weaker than Fork-* Causally consistent views Stronger than Fork-* No blocking needed - allows wait-free protocols. In fact, suffices to achieve all properties of FAUST C2C2

18. read(X 1 ) → null write(X 1,u) C1 C1C1 C2C2 read(X 1 ) → u C2C2 read(X 1 ) → null write(X 1,u) C1 C1C1 C2C2 read(X 1 ) → u write(X 1,v) read(X 1 ) → v REPLY: I scheduled op1, op2, … before you value, signed context of last committed op of writer COMMIT op (signed context) ClientServer SUBMIT read operation Server claims that C 1 committed no operations Server can hide context of second write, but must provide context of first write C 2 detects the error USTOR: The untrusted storage protocol

19. Some properties of USTOR Linearizable executions with correct server –A correct server schedules & processes the ops in order of receipt Wait-free with correct server –A read does not need to wait till some write COMMITs –COMMIT info can arrive with SUBMIT of next operation All executions are causally consistent –From properties of Weak-fork Detects server failure, if server returns inconsistent context information What’s missing for FAUST? –Accurate notifications to clients: operation stability, server failure –Completeness: for every op, eventually one of the following happens: op indicated as stable with respect to other clients server failure detected

20. From USTOR to FAUST Application FAUST Protocol USTOR Protocol (client side) Client-Server ChannelClient-to-Client Comm. PROBE CONTEXT FAILURE stability notifications server failure notifications read / write SUBMIT COMMIT REPLY

21. FAUST Layer at client C i Maintain the latest known context: context i Periodically (when no user ops) read data stored by others –Find out their latest known context, update context i –C i always makes operations – others will see that context i changes If no new info from C k for a long time: –send a direct PROBE message (e.g., email) to C k –C k responds with CONTEXT message containing context k If context k is inconsistent with context i –send a FAILURE message to all –report failure to application Otherwise, if context k includes j-th operation of C i –j-th operation reported stable at C i w.r.t. C k

22. Conclusion Need for semantics and client-side tools for services provided by “clouds” –Clients should be able to detect if semantics are violated –ACM SIGACT News Column 34, Distributed Computing in the Clouds; http://www.ee.technion.ac.il/~idish/sigactNews Our suggestion: Fail-Aware Untrusted Service –FAUST provides such semantics for simple storage When a read/write operation completes causal consistency (at least) is guaranteed Eventually, more information is obtained. Then: –Indicate to client that his operation preserves stronger semantics OR: report server failure Common case: when server is correct, each operation completes independently of others + perfect consistency