1. Blazes: coordination analysis for distributed program Peter Alvaro, Neil Conway, Joseph M. Hellerstein David Maier UC Berkeley Portland State

2. Distributed systems are hard AsynchronyPartial Failure

3. Asynchrony isn’t that hard Logical timestamps Deterministic interleaving Ameloriation:

4. Partial failure isn’t that hard Replication Replay Ameloriation:

5. Asynchrony * partial failure is hard 2 Logical timestamps Deterministic interleaving Replication Replay

6. asynchrony * partial failure is hard 2 Replication Replay Today: Consistency criteria for fault-tolerant distributed systems Blazes: analysis and enforcement

7. This talk is all setup Frame of mind: 1.Dataflow: a model of distributed computation 2.Anomalies: what can go wrong? 3.Remediation strategies 1.Component properties 2.Delivery mechanisms Framework: Blazes – coordination analysis and synthesis

8. Little boxes: the dataflow model Generalization of distributed services Components interact via asynchronous calls (streams)

9. Components Input interfacesOutput interface

10. Streams Nondeterministic order

11. Example: a join operator R S T

12. Example: a key/value store put get response

13. Example: a pub/sub service publish subscribe deliver

14. Logical dataflow “Software architecture” Data source client Service X filter cache c a b

15. Dataflow is compositional Components are recursively defined Data source client Service X filter aggregator

16. Dataflow exhibits self-similarity

17. DBHDFS Hadoop Index Combine Static HTTP App1App2Buy Content User requests App1 answers App2 answers

18. Physical dataflow

19. Data source client Service X filter aggregator c a b

20. Physical dataflow Data source Service Xfilter aggregator client “System architecture”

21. What could go wrong?

22. Cross-run nondeterminism Data source client Service X filter aggregator c a b Run 1 Nondeterministic replays

23. Cross-run nondeterminism Data source client Service X filter aggregator c a b Nondeterministic replays Run 2

24. Cross-instance nondeterminism Data source Service X client Transient replica disagreement

25. Divergence Data source Service X client Permanent replica disagreement

26. Hazards Data source client Service X filter aggregator c a b Order  Contents?

27. Preventing the anomalies 1.Understand component semantics (And disallow certain compositions)

28. Component properties Convergence –Component replicas receiving the same messages reach the same state –Rules out divergence

29. InsertRead Convergent data structure (e.g., Set CRDT) Convergence Insert Read Commutativity Associativity Idempotence Reordering Batching Retry/duplication Tolerant to

30. Convergence isn’t compositional Data source client Convergent (identical input contents  identical state)

31. Component properties Convergence –Component replicas receiving the same messages reach the same state –Rules out divergence Confluence –Output streams have deterministic contents –Rules out all stream anomalies Confluent  convergent

32. Confluence output set = f(input set) { } =

33. Confluence is compositional output set = f  g(input set)

34. Preventing the anomalies 1.Understand component semantics (And disallow certain compositions) 2.Constrain message delivery orders 1.Ordering

35. Ordering – global coordination Deterministic outputs Order-sensitive

36. Ordering – global coordination Data source client The first principle of successful scalability is to batter the consistency mechanisms down to a minimum. – James Hamilton

37. Preventing the anomalies 1.Understand component semantics (And disallow certain compositions) 2.Constrain message delivery orders 1.Ordering 2.Barriers and sealing

38. Barriers – local coordination Deterministic outputs Data source client Order-sensitive

39. Barriers – local coordination Data source client

40. Sealing – continuous barriers Do partitions of (infinite) input streams “end”? Can components produce deterministic results given “complete” input partitions? Sealing: partition barriers for infinite streams

41. Sealing – continuous barriers Finite partitions of infinite inputs are common …in distributed systems –Sessions –Transactions –Epochs / views …and applications –Auctions –Chats –Shopping carts

42. Blazes: consistency analysis + coordination selection

43. Blazes: Mode 1: Grey boxes

44. Grey boxes Example: pub/sub x = publish y = subscribe z = deliver x y z Deterministic but unordered SeverityLabelConfluentStateless 1CRXX 2CWX 3OR gate X 4OW gate x->z : CW y->z : CWT

45. Grey boxes Example: key/value store x = put; y = get; z = response x y z Deterministic but unordered SeverityLabelConfluentStateless 1CRXX 2CWX 3OR gate X 4OW gate x->z : OW key y->z : ORT

46. Label propagation – confluent composition CW CR Deterministic outputs CW

47. Label propagation – unsafe composition OW CR Tainted outputs Interposition point

48. Label propagation – sealing OW key CR Deterministic outputs OW key Seal(key=x)

49. Blazes: Mode 1: White boxes

50. white boxes module KVS state do interface input, :put, [:key, :val] interface input, :get, [:ident, :key] interface output, :response, [:response_id, :key, :val] table :log, [:key, :val] end bloom do log <+ put log :key) response :key) do |s,l| [l.ident, s.key, s.val] end put  response: OW key get  response: OR key Negation (  order sensitive) Partitioned by :key

51. white boxes module PubSub state do interface input, :publish, [:key, :val] interface input, :subscribe, [:ident, :key] interface output, :response, [:response_id, :key, :val] table :log, [:key, :val] table :sub_log, [:ident, :key] end bloom do log <= publish sub_log <= subscribe response :key) do |s,l| [l.ident, s.key, s.val] end publish  response: CW subscribe  response: CR

52. The Blazes frame of mind: Asynchronous dataflow model Focus on consistency of data in motion –Component semantics –Delivery mechanisms and costs Automatic, minimal coordination

53. Queries?