1. Cluster Computing with DryadLINQ Mihai Budiu, MSR-SVC PARC, May 8 2008

2. Aknowledgments 2 MSR SVC and ISRC SVC Michael Isard, Yuan Yu, Andrew Birrell, Dennis Fetterly Ulfar Erlingsson, Pradeep Kumar Gunda, Jon Currey

3. Computer Evolution 3 196120082040 ?

4. Computer Evolution 4 ENIAC 1943 30 tons 200kW Datacenter 2008 500,000 ft 2 40MW ? 2040

5. 5

6. Layers 6 Networking Storage Distributed Execution Scheduling Resource Management Applications Identity & Security Caching and Synchronization Programming Languages and APIs Operating System

7. Pieces of the Global Computer 7

8. This Work 8

9. The Rest of This Talk 9 Windows Server Cluster Services Distributed Filesystem Dryad DryadLINQ Windows Server CIFS/NTFS Large Vectors Machine Learning

10. How fast can you sort 10 10 100-byte records (1Tb)? Sequential scan/disk = 4.6 hours Current record: 435 seconds (7.2 min) cluster of 40 Itanium2, 2520 SAN disks Code: 3300 lines of C Our result: 349 seconds (5.8 min) cluster of 240 AMD64 (quad) machines, 920 disks Code: 17 lines of LINQ 10

11. Introduction Dryad DryadLINQ Building on DryadLINQ 11

12. Introduction Dryad – deployed since 2006 – many thousands of machines – analyzes many petabytes of data/day DryadLINQ Building on DryadLINQ 12

13. Goal 13

14. Design Space 14 ThroughputLatency Internet Private data center Data- parallel Shared memory

15. Data Partitioning 15 RAM DATA

16. 2-D Piping Unix Pipes: 1-D grep | sed | sort | awk | perl Dryad: 2-D grep 1000 | sed 500 | sort 1000 | awk 500 | perl 50 16

17. Dryad = Execution Layer 17 Job (application) Dryad Cluster Pipeline Shell Machine ≈

18. Virtualized 2-D Pipelines 18

19. Virtualized 2-D Pipelines 19

20. Virtualized 2-D Pipelines 20

21. Virtualized 2-D Pipelines 21

22. Virtualized 2-D Pipelines 22 2D DAG multi-machine virtualized

23. Dryad Job Structure 23 grep sed sort awk perl grep sed sort awk Input files Vertices (processes) Output files Channels Stage

24. Channels 24 X M Items Finite Streams of items distributed filesystem files (persistent) SMB/NTFS files (temporary) TCP pipes (inter-machine) memory FIFOs (intra-machine)

25. Architecture 25 Files, TCP, FIFO, Network job schedule data plane control plane NSPD V VV Job managercluster

26. Fault Tolerance

27. X[0]X[1]X[3]X[2]X’[2] Completed vertices Slow vertex Duplicate vertex Dynamic Graph Rewriting Duplication Policy = f(running times, data volumes)

28. SSSS AAA SS T SSSSSS T # 1# 2# 1# 3 # 2 # 3# 2# 1 static dynamic rack # Dynamic Aggregation 28

29. Data-Parallel Computation 29 Storage Execution Application Parallel Databases Map- Reduce GFS BigTable Dryad

30. Introduction Dryad DryadLINQ Building on Dryad 30

31. DryadLINQ 31 Dryad

32. 32 LINQ Collection collection; bool IsLegal(Key); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

33. Collection collection; bool IsLegal(Key k); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value}; 33 DryadLINQ = LINQ + Dryad C# collection results C# Vertex code Query plan (Dryad job) Data

34. Data Model 34 Partition Collection C# objects

35. Query Providers 35 DryadLINQ Client machine (11) Distributed query plan C# Query Expr Data center Output Tables Results Input Tables Invoke Query Output DryadTable Dryad Execution C# Objects JM ToDryadTable foreach

36. Demo 36

37. Example: Histogram 37 public static IQueryable Histogram( IQueryable input, int k) { var words = input.SelectMany(x => x.line.Split(' ')); var groups = words.GroupBy(x => x); var counts = groups.Select(x => new Pair(x.Key, x.Count())); var ordered = counts.OrderByDescending(x => x.count); var top = ordered.Take(k); return top; } “A line of words of wisdom” [“A”, “line”, “of”, “words”, “of”, “wisdom”] [[“A”], [“line”], [“of”, “of”], [“words”], [“wisdom”]] [ {“A”, 1}, {“line”, 1}, {“of”, 2}, {“words”, 1}, {“wisdom”, 1}] [{“of”, 2}, {“A”, 1}, {“line”, 1}, {“words”, 1}, {“wisdom”, 1}] [{“of”, 2}, {“A”, 1}, {“line”, 1}]

38. Histogram Plan 38 SelectMany HashDistribute Merge GroupBy Select OrderByDescending Take MergeSort Take

39. Map-Reduce in DryadLINQ 39 public static IQueryable MapReduce ( this IQueryable input, Expression >> mapper, Expression > keySelector, Expression,S>> reducer) { var map = input.SelectMany(mapper); var group = map.GroupBy(keySelector); var result = group.Select(reducer); return result; }

40. Map-Reduce Plan 40 M D R G M Q G1G1 R D MS G2G2 R (1)(2)(3) X X M Q G1G1 R D MS G2G2 R X M Q G1G1 R D G2G2 R X M Q G1G1 R D M Q G1G1 R D G2G2 R X M Q G1G1 R D G2G2 R X M Q G1G1 R D G2G2 R G2G2 R map sort groupby reduce distribute mergesort groupby reduce mergesort groupby reduce consumer map partial aggregation reduce SSSS AAA SS T

41. Distributed Sorting in DryadLINQ 41 public static IQueryable DSort (this IQueryable source, Expression > keySelector, int pcount) { var samples = source.Apply(x => Sampling(x)); var keys = samples.Apply(x => ComputeKeys(x, pcount)); var parts = source.RangePartition(keySelector, keys); return parts.OrderBy(keySelector); }

42. Distributed Sorting Plan 42 O DS H D M S H D M S D H D M S D M S M S (1)(2)(3)

43. Introduction Dryad DryadLINQ Building on DryadLINQ 43

44. Machine Learning in DryadLINQ 44 Dryad DryadLINQ Large Vector Machine learning Data analysis

45. Operations on Large Vectors: Map 1 45 U T T U f f f preserves partitioning

46. V Map 2 (Pairwise) 46 T U f V U T f

47. Map 3 (Vector-Scalar) 47 T U f V V U T f

48. Reduce (Fold) 48 UUU U f fff f UUU U

49. Linear Algebra 49 T U V =,, T

50. Linear Regression Data Find S.t. 50

51. Analytic Solution 51 X×X T Y×X T Σ X[0]X[1]X[2]Y[0]Y[1]Y[2] Σ [ ] -1 * A Map Reduce

52. Linear Regression Code Vectors x = input(0), y = input(1); Matrices xx = x.Map(x, (a,b) => a.OuterProd(b)); OneMatrix xxs = xx.Sum(); Matrices yx = y.Map(x, (a,b) => a.OuterProd(b)); OneMatrix yxs = yx.Sum(); OneMatrix xxinv = xxs.Map(a => a.Inverse()); OneMatrix A = yxs.Map(xxinv, (a, b) => a.Mult(b)); 52

53. Expectation Maximization (Gaussians) 53 160 lines 3 iterations shown

54. Conclusions Dryad = distributed execution environment Application-independent (semantics oblivious) Supports rich software ecosystem – Relational algebra, Map-reduce, LINQ DryadLINQ = Compiles LINQ to Dryad C# objects and declarative programming.Net and Visual Studio for parallel programming 54

55. Backup Slides 55

56. Software Stack 56 Windows Server Cluster Services Distributed Filesystem Dryad Distributed Shell PSQL DryadLINQ Perl SQL server C++ Windows Server C++ CIFS/NTFS legacy code sed, awk, grep, etc. SSIS Scope C# Vectors Machine Learning C# Job queueing, monitoring

57. Very Large Vector Library PartitionedVector 57 T Scalar TT T

58. DryadLINQ 58 Declarative programming Integration with Visual Studio Integration with.Net Type safety Automatic serialization Job graph optimizations  static  dynamic Conciseness

59. Sort & Map-Reduce in DryadLINQ 59

60. Many similarities Exe + app. model Map+sort+reduce Few policies Program=map+reduce Simple Mature (> 4 years) Widely deployed Hadoop Dryad Map-Reduce Execution layer Job = arbitrary DAG Plug-in policies Program=graph gen. Complex ( features) New (< 2 years) Still growing Internal 60

61. PLINQ 61 public static IEnumerable DryadSort (IEnumerable source, Func keySelector, IComparer comparer, bool isDescending) { return source.AsParallel().OrderBy(keySelector, comparer); }

62. Query histogram computation Input: log file (n partitions) Extract queries from log partitions Re-partition by hash of query (k buckets) Compute histogram within each bucket

63. Naïve histogram topology Pparse lines D hash distribute S quicksort C count occurrences MSmerge sort

64. Efficient histogram topology Pparse lines D hash distribute S quicksort C count occurrences MSmerge sort M non-deterministic merge Q' is:Each R is: Each MS C M P C S Q' RR k T k n T is: Each MS D C

65. Final histogram refinement 1,800 computers 43,171 vertices 11,072 processes 11.5 minutes

66. Data Distribution (Group By) 66 Dest Source Dest Source Dest Source m n m x n

67. TT [0-?)[?-100) Range-Distribution Manager S DDD SS SSS T static dynamic 67 Hist [0-30),[30-100) [30-100)[0-30) [0-100)

68. Goal: Declarative Programming 68 X T S XX SS TTT X staticdynamic

69. JM code vertex code Staging 1. Build 2. Send.exe 3. Start JM 5. Generate graph 7. Serialize vertices 8. Monitor Vertex execution 4. Query cluster resources Cluster services 6. Initialize vertices

70. SkyServer Query 18 70 select distinct P.ObjID into results from photoPrimary U, neighbors N, photoPrimary L where U.ObjID = N.ObjID and L.ObjID = N.NeighborObjID and P.ObjID < L.ObjID and abs((U.u-U.g)-(L.u-L.g))<0.05 and abs((U.g-U.r)-(L.g-L.r))<0.05 and abs((U.r-U.i)-(L.r-L.i))<0.05 and abs((U.i-U.z)-(L.i-L.z))<0.05

71. 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 0246810 Number of Computers Speed-up (times) Dryad In-Memory Dryad Two-pass SQLServer 2005 SkyServer Q18 Performance 71