1. Piccolo: Building fast distributed programs with partitioned tables Russell Power Jinyang Li New York University

2. Motivating Example: PageRank for each node X in graph: for each edge X  Z: next[Z] += curr[X] Repeat until convergence A  B,C,D BEBE CDCD … A: 0.12 B: 0.15 C: 0.2 … A: 0 B: 0 C: 0 … CurrNextInput Graph A: 0.2 B: 0.16 C: 0.21 … A: 0.2 B: 0.16 C: 0.21 … A: 0.25 B: 0.17 C: 0.22 … A: 0.25 B: 0.17 C: 0.22 … Fits in memory!

3. PageRank in MapReduce 1 1 2 2 3 3 Distributed Storage Graph streamRank stream A->B,C, B->D A:0.1, B:0.2 Rank stream A:0.1, B:0.2  Data flow models do not expose global state.

4. PageRank in MapReduce 1 1 2 2 3 3 Distributed Storage Graph streamRank stream A->B,C, B->D A:0.1, B:0.2 Rank stream A:0.1, B:0.2  Data flow models do not expose global state.

5. PageRank With MPI/RPC 1 1 2 2 3 3 Distributed Storage Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … User explicitly programs communication

6. Piccolo’s Goal: Distributed Shared State 1 1 2 2 3 3 Distributed Storage Graph A->B,C B->D … Ranks A: 0 B: 0 … Ranks A: 0 B: 0 … read/write Distributed in- memory state

7. Piccolo’s Goal: Distributed Shared State 1 1 2 2 3 3 Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … Piccolo runtime handles communication

8. Ease of usePerformance

10. Talk outline  Motivation  Piccolo's Programming Model  Runtime Scheduling  Evaluation

11. Programming Model 1 1 2 2 3 3 Graph A  B,C B  D … Ranks A: 0 B: 0 … Ranks A: 0 B: 0 … read/write x get/put update/iterate Implemented as library for C++ and Python

12. def main(): for i in range(50): launch_jobs(NUM_MACHINES, pr_kernel, graph, curr, next) swap(curr, next) next.clear() def pr_kernel(graph, curr, next): i = my_instance n = len(graph)/NUM_MACHINES for s in graph[(i-1)*n:i*n] for t in s.out: next[t] += curr[s.id] / len(s.out) Naïve PageRank with Piccolo Run by a single controller Run by a single controller Jobs run by many machines curr = Table(key=PageID, value=double) next = Table(key=PageID, value=double) Controller launches jobs in parallel

13. Naïve PageRank is Slow 1 1 2 2 3 3 Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … get put get

14. PageRank: Exploiting Locality Control table partitioning Co-locate tables Co-locate execution with table curr = Table(…,partitions=100,partition_by=site) next = Table(…,partitions=100,partition_by=site) group_tables(curr,next,graph) def pr_kernel(graph, curr, next): for s in graph.get_iterator(my_instance) for t in s.out: next[t] += curr[s.id] / len(s.out) def main(): for i in range(50): launch_jobs(curr.num_partitions, pr_kernel, graph, curr, next, locality=curr) swap(curr, next) next.clear()

15. Exploiting Locality 1 1 2 2 3 3 Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … get put get

16. Exploiting Locality 1 1 2 2 3 3 Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … put get put get

17. Synchronization 1 1 2 2 3 3 Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … put (a=0.3) put (a=0.2) How to handle synchronization?

18. Synchronization Primitives  Avoid write conflicts with accumulation functions  NewValue = Accum(OldValue, Update) sum, product, min, max  Global barriers are sufficient  Tables provide release consistency

19. PageRank: Efficient Synchronization curr = Table(…,partition_by=site,accumulate=sum) next = Table(…,partition_by=site,accumulate=sum) group_tables(curr,next,graph) def pr_kernel(graph, curr, next): for s in graph.get_iterator(my_instance) for t in s.out: next.update(t, curr.get(s.id)/len(s.out)) def main(): for i in range(50): handle = launch_jobs(curr.num_partitions, pr_kernel, graph, curr, next, locality=curr) barrier(handle) swap(curr, next) next.clear() Accumulation via sum Update invokes accumulation function Explicitly wait between iterations

20. Efficient Synchronization 1 1 2 2 3 3 Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … put (a=0.3) put (a=0.2) update (a, 0.2) update (a, 0.3) Runtime computes sum Workers buffer updates locally  Release consistency Workers buffer updates locally  Release consistency

21. Table Consistency 1 1 2 2 3 3 Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … put (a=0.3) put (a=0.2) update (a, 0.2) update (a, 0.3)

22. PageRank with Checkpointing curr = Table(…,partition_by=site,accumulate=sum) next = Table(…,partition_by=site,accumulate=sum) group_tables(curr,next) def pr_kernel(graph, curr, next): for node in graph.get_iterator(my_instance) for t in s.out: next.update(t,curr.get(s.id)/len(s.out)) def main(): curr, userdata = restore() last = userdata.get(‘iter’, 0) for i in range(last,50): handle = launch_jobs(curr.num_partitions, pr_kernel, graph, curr, next, locality=curr) cp_barrier(handle, tables=(next), userdata={‘iter’:i}) swap(curr, next) next.clear() Restore previous computation User decides which tables to checkpoint and when

23. Distributed Storage Recovery via Checkpointing 1 1 2 2 3 3 Graph A->B,C … Graph A->B,C … Ranks A: 0 … Ranks A: 0 … Graph B->D … Graph B->D … Ranks B: 0 … Ranks B: 0 … Graph C->E,F … Graph C->E,F … Ranks C: 0 … Ranks C: 0 … Runtime uses Chandy-Lamport protocol

24. Talk Outline  Motivation  Piccolo's Programming Model  Runtime Scheduling  Evaluation

25. Load Balancing 1 1 3 3 2 2 master J5 J3 Other workers are updating P6! Pause updates! P1 P1, P2 P3 P3, P4 P5 P5, P6 J1 J1, J2 J3 J3, J4 J6 J5, J6 P6 Coordinates work- stealing

26. Talk Outline  Motivation  Piccolo's Programming Model  System Design  Evaluation

27. Piccolo is Fast NYU cluster, 12 nodes, 64 cores 100M-page graph Main Hadoop Overheads:  Sorting  HDFS  Serialization Main Hadoop Overheads:  Sorting  HDFS  Serialization PageRank iteration time (seconds) Hadoop Piccolo

28. Piccolo Scales Well EC2 Cluster - linearly scaled input graph ideal 1 billion page graph PageRank iteration time (seconds)

29.  Iterative Applications  N-Body Simulation  Matrix Multiply  Asynchronous Applications  Distributed web crawler ‏ Other applications No straightforward Hadoop implementation

30. Related Work  Data flow  MapReduce, Dryad  Tuple Spaces  Linda, JavaSpaces  Distributed Shared Memory  CRL, TreadMarks, Munin, Ivy  UPC, Titanium

31. Conclusion  Distributed shared table model  User-specified policies provide for  Effective use of locality  Efficient synchronization  Robust failure recovery

32. Gratuitous Cat Picture I can haz kwestions? Try it out: piccolo.news.cs.nyu.edu

33. Naïve PageRank – Problems! curr = Table(key=PageID, value=double) next = Table(key=PageID, value=double) def pr_kernel(graph, curr, next): i = my_instance n = len(graph)/NUM_MACHINES for s in graph[(i-1)*n:i*n]: for t in s.out: next[t] += curr[s.id] / len(s.out) def main(): for i in range(50): launch_kernel(NUM_MACHINES, pr_kernel, graph, curr, next) swap(curr, next) next.clear() Lots of remote reads! No synchronization!

34. Failure Handling  Users decide what and when to checkpoint  When to checkpoint?  At global barriers  Periodically during execution  What to checkpoint?  Table data (saved automatically by Piccolo) ‏  User state (saved/restored via callback)

35. System Overview Store table partitions Process table operations Run kernel threads Store table partitions Process table operations Run kernel threads Assign partitions/kernels Run control thread Assign partitions/kernels Run control thread worker master

36. P1 Partition and Kernel Assignment P1, P2 P3 worker P3, P4 P5 P5, P6 master K1 K1, K2 K3 K3, K4 K6 K5, K6 Initially assign kernels by locality

37. Kernel Execution A A C C B B master Remote reads block execution Remote writes are accumulated locally K5 P1 P1, P2 P3 P3, P4 P5 P5, P6 K1 K1, K2 K3 K3, K4 K6 K5, K6

38. Asynchronous Web Crawler  Simple to implement  Crawl pages quickly, as they are discovered  Saturate 100mb link using 32 workers  Simple to implement  Crawl pages quickly, as they are discovered  Saturate 100mb link using 32 workers

39. Exploiting Locality  Explicitly divide tables into partitions  Expose partitioning to users  Users provide key to partition mapping function  Co-locate kernels with table partitions  Co-locate partitions of different tables