1. CS 347Lecture 9B1 CS 347: Parallel and Distributed Data Management Notes X: S4 Hector Garcia-Molina

2. Material based on: CS 347Lecture 9B2

3. S4 Platform for processing unbounded data streams –general purpose –distributed –scalable –partially fault tolerant (whatever this means!) CS 347Lecture 9B3

4. Data Stream CS 347Lecture 9B4 Terminology: event (data record), key, attribute Question: Can an event have duplicate attributes for same key? (I think so...) Stream unbounded, generated by say user queries, purchase transactions, phone calls, sensor readings,...

5. S4 Processing Workflow CS 347Lecture 9B5 user specified “processing unit”

6. Inside a Processing Unit CS 347Lecture 9B6 PE1 PE2 PEn... key=z key=b key=a processing element

7. Example: Stream of English quotes Produce a sorted list of the top K most frequent words CS 347Lecture 9B7

8. CS 347Lecture 9B8

9. Processing Nodes CS 347Lecture 9B9... key=b key=a processing node... hash(key) =1 hash(key) =m

10. Dynamic Creation of PEs As a processing node sees new key attributes, it dynamically creates new PEs to handle them Think of PEs as threads CS 347Lecture 9B10

11. Another View of Processing Node CS 347Lecture 9B11

12. Failures Communication layer detects node failures and provides failover to standby nodes What happens events in transit during failure? (My guess: events are lost!) CS 347Lecture 9B12

13. How do we do DB operations on top of S4? Selects & projects easy! CS 347Lecture 9B13   What about joins?

14. What is a Stream Join? For true join, would need to store all inputs forever! Not practical... Instead, define window join: –at time t new R tuple arrives –it only joins with previous w S tuples CS 347Lecture 9B14 R S

15. One Idea for Window Join CS 347Lecture 9B15 R S code for PE: for each event e: if e.rel=R [store in Rset(last w) for s in Sset: output join(e,s) ] else... “key” is join attribute

16. One Idea for Window Join CS 347Lecture 9B16 R S code for PE: for each event e: if e.rel=R [store in Rset(last w) for s in Sset: output join(e,s) ] else... “key” is join attribute Is this right??? (enforcing window on a per-key value basis) Maybe add sequence numbers to events to enforce correct window?

17. Another Idea for Window Join CS 347Lecture 9B17 R S code for PE: for each event e: if e.rel=R [store in Rset(last w) for s in Sset: if e.C=s.C then output join(e,s) ] else if e.rel=S... All R & S events have “key=fake”; Say join key is C.

18. Another Idea for Window Join CS 347Lecture 9B18 R S code for PE: for each event e: if e.rel=R [store in Rset(last w) for s in Sset: if e.C=s.C then output join(e,s) ] else if e.rel=S... All R & S events have “key=fake”; Say join key is C. Entire join done in one PE; no parallelism

19. Do You Have a Better Idea for Window Join? CS 347Lecture 9B19 R S

20. Final Comment: Managing state of PE CS 347Lecture 9B20 Who manages state? S4: user does Mupet: System does Is state persistent? state

21. CS 347Lecture 9B21 CS 347: Parallel and Distributed Data Management Notes X: Hyracks Hector Garcia-Molina

22. Hyracks Generalization of map-reduce Infrastructure for “big data” processing Material here based on: CS 347Lecture 9B22 Appeared in ICDE 2011

23. A Hyracks data object: Records partitioned across N sites Simple record schema is available (more than just key-value) CS 347Lecture 9B23 site 1 site 2 site 3

24. Operations CS 347Lecture 9B24 operator distribution rule

25. Operations (parallel execution) CS 347Lecture 9B25 op 1 op 2 op 3

26. Example: Hyracks Specification CS 347Lecture 9B26

27. Example: Hyracks Specification CS 347Lecture 9B27 initial partitions input & output is a set of partitions mapping to new partitions 2 replicated partitions

28. Notes Job specification can be done manually or automatically CS 347Lecture 9B28

29. Example: Activity Node Graph CS 347Lecture 9B29

30. Example: Activity Node Graph CS 347Lecture 9B30 sequencing constraint activities

31. Example: Parallel Instantiation CS 347Lecture 9B31

32. Example: Parallel Instantiation CS 347Lecture 9B32 stage (start after input stages finish)

33. System Architecture CS 347Lecture 9B33

34. Map-Reduce on Hyracks CS 347Lecture 9B34

35. Library of Operators: File reader/writers Mappers Sorters Joiners (various types0 Aggregators Can add more CS 347Lecture 9B35

36. Library of Connectors: N:M hash partitioner N:M hash-partitioning merger (input sorted) N:M rage partitioner (with partition vector) N:M replicator 1:1 Can add more! CS 347Lecture 9B36

37. Hyracks Fault Tolerance: Work in progress? CS 347Lecture 9B37

38. Hyracks Fault Tolerance: Work in progress? CS 347Lecture 9B38 save output to files The Hadoop/MR approach: save partial results to persistent storage; after failure, redo all work to reconstruct missing data

39. Hyracks Fault Tolerance: Work in progress? CS 347Lecture 9B39 Can we do better? Maybe: each process retains previous results until no longer needed? Pi output: r1, r2, r3, r4, r5, r6, r7, r8 have "made way" to final result current output

40. CS 347Lecture 9B40 CS 347: Parallel and Distributed Data Management Notes X: Pregel Hector Garcia-Molina

41. Material based on: In SIGMOD 2010 Note there is an open-source version of Pregel called GIRAPH CS 347Lecture 9B41

42. Pregel A computational model/infrastructure for processing large graphs Prototypical example: Page Rank CS 347Lecture 9B42 x e d b a f PR[i+1,x] = f(PR[i,a]/na, PR[i,b]/nb)

43. Pregel Synchronous computation in iterations In one iteration, each node: –gets messages from neighbors –computes –sends data to neighbors CS 347Lecture 9B43 x e d b a f PR[i+1,x] = f(PR[i,a]/na, PR[i,b]/nb)

44. Pregel vs Map-Reduce/S4/Hyracks/... In Map-Reduce, S4, Hyracks,... workflow separate from data CS 347Lecture 9B44 In Pregel, data (graph) drives data flow

45. Pregel Motivation Many applications require graph processing Map-Reduce and other workflow systems not a good fit for graph processing Need to run graph algorithms on many procesors CS 347Lecture 9B45

46. Example of Graph Computation CS 347Lecture 9B46

47. Termination After each iteration, each vertex votes to halt or not If all vertexes vote to halt, computation terminates CS 347Lecture 9B47

48. Vertex Compute (simplified) Available data for iteration i: –InputMessages: { [from, value] } –OutputMessages: { [to, value] } –OutEdges: { [to, value] } –MyState: value OutEdges and MyState are remembered for next iteration CS 347Lecture 9B48

49. Max Computation change := false for [f,w] in InputMessages do if w > MyState.value then [ MyState.value := w change := true ] if (superstep = 1) OR change then for [t, w] in OutEdges do add [t, MyState.value] to OutputMessages else vote to halt CS 347Lecture 9B49

50. Page Rank Example CS 347Lecture 9B50

51. Page Rank Example CS 347Lecture 9B51 iteration count iterate thru InputMessages MyState.value shorthand: send same msg to all

52. Single-Source Shortest Paths CS 347Lecture 9B52

53. Architecture CS 347Lecture 9B53 master worker A input data 1 input data 2 worker B worker C graph has nodes a,b,c,d... sample record: [a, value]

54. Architecture CS 347Lecture 9B54 master worker A vertexes a, b, c input data 1 input data 2 worker B vertexes d, e worker C vertexes f, g, h partition graph and assign to workers

55. Architecture CS 347Lecture 9B55 master worker A vertexes a, b, c input data 1 input data 2 worker B vertexes d, e worker C vertexes f, g, h read input dataworker A forwards input values to appropriate workers

56. Architecture CS 347Lecture 9B56 master worker A vertexes a, b, c input data 1 input data 2 worker B vertexes d, e worker C vertexes f, g, h run superstep 1

57. Architecture CS 347Lecture 9B57 master worker A vertexes a, b, c input data 1 input data 2 worker B vertexes d, e worker C vertexes f, g, h at end superstep 1, send messages halt?

58. Architecture CS 347Lecture 9B58 master worker A vertexes a, b, c input data 1 input data 2 worker B vertexes d, e worker C vertexes f, g, h run superstep 2

59. Architecture CS 347Lecture 9B59 master worker A vertexes a, b, c input data 1 input data 2 worker B vertexes d, e worker C vertexes f, g, h checkpoint

60. Architecture CS 347Lecture 9B60 master worker A vertexes a, b, c worker B vertexes d, e worker C vertexes f, g, h checkpoint write to stable store: MyState, OutEdges, InputMessages (or OutputMessages)

61. Architecture CS 347Lecture 9B61 master worker A vertexes a, b, c worker B vertexes d, e if worker dies, find replacement & restart from latest checkpoint

62. Architecture CS 347Lecture 9B62 master worker A vertexes a, b, c input data 1 input data 2 worker B vertexes d, e worker C vertexes f, g, h

63. Interesting Challenge How best to partition graph for efficiency? CS 347Lecture 9B63 x e d b a f g

64. CS 347Lecture 9B64 CS 347: Parallel and Distributed Data Management Notes X: BigTable, HBASE, Cassandra Hector Garcia-Molina

65. Sources HBASE: The Definitive Guide, Lars George, O’Reilly Publishers, 2011. Cassandra: The Definitive Guide, Eben Hewitt, O’Reilly Publishers, 2011. BigTable: A Distributed Storage System for Structured Data, F. Chang et al, ACM Transactions on Computer Systems, Vol. 26, No. 2, June 2008. CS 347Lecture 9B65

66. Lots of Buzz Words! “Apache Cassandra is an open-source, distributed, decentralized, elastically scalable, highly available, fault-tolerant, tunably consistent, column-oriented database that bases its distribution design on Amazon’s dynamo and its data model on Google’s Big Table.” Clearly, it is buzz-word compliant!! CS 347Lecture 9B66

67. Basic Idea: Key-Value Store CS 347Lecture 9B67 Table T:

68. Basic Idea: Key-Value Store CS 347Lecture 9B68 Table T: keys are sorted API: –lookup(key)  value –lookup(key range)  values –getNext  value –insert(key, value) –delete(key) Each row has timestemp Single row actions atomic (but not persistent in some systems?) No multi-key transactions No query language!

69. Fragmentation (Sharding) CS 347Lecture 9B69 server 1server 2server 3 use a partition vector “auto-sharding”: vector selected automatically tablet

70. Tablet Replication CS 347Lecture 9B70 server 3server 4server 5 primary backup Cassandra: Replication Factor (# copies) R/W Rule: One, Quorum, All Policy (e.g., Rack Unaware, Rack Aware,...) Read all copies (return fastest reply, do repairs if necessary) HBase: Does not manage replication, relies on HDFS

71. Need a “directory” Table Name: Key  Server that stores key  Backup servers Can be implemented as a special table. CS 347Lecture 9B71

72. Tablet Internals CS 347Lecture 9B72 memory disk Design Philosophy (?): Primary scenario is where all data is in memory. Disk storage added as an afterthought

73. Tablet Internals CS 347Lecture 9B73 memory disk flush periodically tablet is merge of all segments (files) disk segments imutable writes efficient; reads only efficient when all data in memory periodically reorganize into single segment tombstone

74. Column Family CS 347Lecture 9B74

75. Column Family CS 347Lecture 9B75 for storage, treat each row as a single “super value” API provides access to sub-values (use family:qualifier to refer to sub-values e.g., price:euros, price:dollars ) Cassandra allows “super-column”: two level nesting of columns (e.g., Column A can have sub-columns X & Y )

76. Vertical Partitions CS 347Lecture 9B76 can be manually implemented as

77. Vertical Partitions CS 347Lecture 9B77 column family good for sparse data; good for column scans not so good for tuple reads are atomic updates to row still supported? API supports actions on full table; mapped to actions on column tables API supports column “project” To decide on vertical partition, need to know access patterns

78. Failure Recovery (BigTable, HBase) CS 347Lecture 9B78 tablet server memory log GFS or HFS master node spare tablet server write ahead logging ping

79. Failure recovery (Cassandra) No master node, all nodes in “cluster” equal CS 347Lecture 9B79 server 1 server 2 server 3

80. Failure recovery (Cassandra) No master node, all nodes in “cluster” equal CS 347Lecture 9B80 server 1 server 2 server 3 access any table in cluster at any server that server sends requests to other servers

81. CS 347Lecture 9B81 CS 347: Parallel and Distributed Data Management Notes X: MemCacheD Hector Garcia-Molina

82. MemCacheD General-purpose distributed memory caching system Open source CS 347Lecture 9B82

83. What MemCacheD Should Be (but ain't) CS 347Lecture 9B83 data source 1 cache 1 data source 2 cache 2 data source 3 cache 3 distributed cache get_object(X)

84. What MemCacheD Should Be (but ain't) CS 347Lecture 9B84 data source 1 cache 1 data source 2 cache 2 data source 3 cache 3 distributed cache get_object(X) x x

85. What MemCacheD Is (as far as I can tell) CS 347Lecture 9B85 data source 1 cache 1 data source 2 cache 2 data source 3 cache 3 get_object(cache 1, MyName) x put(cache 1, myName, X)

86. What MemCacheD Is CS 347Lecture 9B86 data source 1 cache 1 data source 2 cache 2 data source 3 cache 3 get_object(cache 1, MyName) x put(cache 1, myName, X) Can purge MyName whenever each cache is hash table of (name, value) pairs no connection

87. CS 347Lecture 9B87 CS 347: Parallel and Distributed Data Management Notes X: ZooKeeper Hector Garcia-Molina

88. ZooKeeper Coordination service for distributed processes Provides clients with high throughput, high availability, memory only file system CS 347Lecture 9B88 client ZooKeeper / ab c d e znode /a/d/e (has state)

89. ZooKeeper Servers CS 347Lecture 9B89 client server state replica

90. ZooKeeper Servers CS 347Lecture 9B90 client server state replica read

91. ZooKeeper Servers CS 347Lecture 9B91 client server state replica write propagate & sych writes totally ordered: used Zab algorithm

92. Failures CS 347Lecture 9B92 client server state replica if your server dies, just connect to a different one!

93. ZooKeeper Notes Differences with file system: –all nodes can store data –storage size limited API: insert node, read node, read children, delete node,... Can set triggers on nodes Clients and servers must know all servers ZooKeeper works as long as a majority of servers are available Writes totally ordered; read ordered w.r.t. writes CS 347Lecture 9B93

94. CS 347Lecture 9B94 CS 347: Parallel and Distributed Data Management Notes X: Kestrel Hector Garcia-Molina

95. Kestrel Kestrel server handles a set of reliable, ordered message queues A server does not communicate with other servers (advertised as good for scalability!) CS 347Lecture 9B95 client server queues

96. Kestrel Kestrel server handles a set of reliable, ordered message queues A server does not communicate with other servers (advertised as good for scalability!) CS 347Lecture 9B96 client server queues get q put q log