Cluster Computing with DryadLINQ Mihai Budiu, MSR-SVC PARC, May

Aknowledgments 2 MSR SVC and ISRC SVC Michael Isard, Yuan Yu, Andrew Birrell, Dennis Fetterly Ulfar Erlingsson, Pradeep Kumar Gunda, Jon Currey

Computer Evolution ?

Computer Evolution 4 ENIAC tons 200kW Datacenter ,000 ft 2 40MW ? 2040

5

Layers 6 Networking Storage Distributed Execution Scheduling Resource Management Applications Identity & Security Caching and Synchronization Programming Languages and APIs Operating System

Pieces of the Global Computer 7

This Work 8

The Rest of This Talk 9 Windows Server Cluster Services Distributed Filesystem Dryad DryadLINQ Windows Server CIFS/NTFS Large Vectors Machine Learning

How fast can you sort byte records (1Tb)? Sequential scan/disk = 4.6 hours Current record: 435 seconds (7.2 min) cluster of 40 Itanium2, 2520 SAN disks Code: 3300 lines of C Our result: 349 seconds (5.8 min) cluster of 240 AMD64 (quad) machines, 920 disks Code: 17 lines of LINQ 10

Introduction Dryad DryadLINQ Building on DryadLINQ 11

Introduction Dryad – deployed since 2006 – many thousands of machines – analyzes many petabytes of data/day DryadLINQ Building on DryadLINQ 12

Goal 13

Design Space 14 ThroughputLatency Internet Private data center Data- parallel Shared memory

Data Partitioning 15 RAM DATA

2-D Piping Unix Pipes: 1-D grep | sed | sort | awk | perl Dryad: 2-D grep 1000 | sed 500 | sort 1000 | awk 500 | perl 50 16

Dryad = Execution Layer 17 Job (application) Dryad Cluster Pipeline Shell Machine ≈

Virtualized 2-D Pipelines 18

Virtualized 2-D Pipelines 19

Virtualized 2-D Pipelines 20

Virtualized 2-D Pipelines 21

Virtualized 2-D Pipelines 22 2D DAG multi-machine virtualized

Dryad Job Structure 23 grep sed sort awk perl grep sed sort awk Input files Vertices (processes) Output files Channels Stage

Channels 24 X M Items Finite Streams of items distributed filesystem files (persistent) SMB/NTFS files (temporary) TCP pipes (inter-machine) memory FIFOs (intra-machine)

Architecture 25 Files, TCP, FIFO, Network job schedule data plane control plane NSPD V VV Job managercluster

Fault Tolerance

X[0]X[1]X[3]X[2]X’[2] Completed vertices Slow vertex Duplicate vertex Dynamic Graph Rewriting Duplication Policy = f(running times, data volumes)

SSSS AAA SS T SSSSSS T # 1# 2# 1# 3 # 2 # 3# 2# 1 static dynamic rack # Dynamic Aggregation 28

Data-Parallel Computation 29 Storage Execution Application Parallel Databases Map- Reduce GFS BigTable Dryad

Introduction Dryad DryadLINQ Building on Dryad 30

DryadLINQ 31 Dryad

32 LINQ Collection collection; bool IsLegal(Key); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

Collection collection; bool IsLegal(Key k); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value}; 33 DryadLINQ = LINQ + Dryad C# collection results C# Vertex code Query plan (Dryad job) Data

Data Model 34 Partition Collection C# objects

Query Providers 35 DryadLINQ Client machine (11) Distributed query plan C# Query Expr Data center Output Tables Results Input Tables Invoke Query Output DryadTable Dryad Execution C# Objects JM ToDryadTable foreach

Demo 36

Example: Histogram 37 public static IQueryable Histogram( IQueryable input, int k) { var words = input.SelectMany(x => x.line.Split(' ')); var groups = words.GroupBy(x => x); var counts = groups.Select(x => new Pair(x.Key, x.Count())); var ordered = counts.OrderByDescending(x => x.count); var top = ordered.Take(k); return top; } “A line of words of wisdom” [“A”, “line”, “of”, “words”, “of”, “wisdom”] [[“A”], [“line”], [“of”, “of”], [“words”], [“wisdom”]] [ {“A”, 1}, {“line”, 1}, {“of”, 2}, {“words”, 1}, {“wisdom”, 1}] [{“of”, 2}, {“A”, 1}, {“line”, 1}, {“words”, 1}, {“wisdom”, 1}] [{“of”, 2}, {“A”, 1}, {“line”, 1}]

Histogram Plan 38 SelectMany HashDistribute Merge GroupBy Select OrderByDescending Take MergeSort Take

Map-Reduce in DryadLINQ 39 public static IQueryable MapReduce ( this IQueryable input, Expression >> mapper, Expression > keySelector, Expression,S>> reducer) { var map = input.SelectMany(mapper); var group = map.GroupBy(keySelector); var result = group.Select(reducer); return result; }

Map-Reduce Plan 40 M D R G M Q G1G1 R D MS G2G2 R (1)(2)(3) X X M Q G1G1 R D MS G2G2 R X M Q G1G1 R D G2G2 R X M Q G1G1 R D M Q G1G1 R D G2G2 R X M Q G1G1 R D G2G2 R X M Q G1G1 R D G2G2 R G2G2 R map sort groupby reduce distribute mergesort groupby reduce mergesort groupby reduce consumer map partial aggregation reduce SSSS AAA SS T

Distributed Sorting in DryadLINQ 41 public static IQueryable DSort (this IQueryable source, Expression > keySelector, int pcount) { var samples = source.Apply(x => Sampling(x)); var keys = samples.Apply(x => ComputeKeys(x, pcount)); var parts = source.RangePartition(keySelector, keys); return parts.OrderBy(keySelector); }

Distributed Sorting Plan 42 O DS H D M S H D M S D H D M S D M S M S (1)(2)(3)

Introduction Dryad DryadLINQ Building on DryadLINQ 43

Machine Learning in DryadLINQ 44 Dryad DryadLINQ Large Vector Machine learning Data analysis

Operations on Large Vectors: Map 1 45 U T T U f f f preserves partitioning

V Map 2 (Pairwise) 46 T U f V U T f

Map 3 (Vector-Scalar) 47 T U f V V U T f

Reduce (Fold) 48 UUU U f fff f UUU U

Linear Algebra 49 T U V =,, T

Linear Regression Data Find S.t. 50

Analytic Solution 51 X×X T Y×X T Σ X[0]X[1]X[2]Y[0]Y[1]Y[2] Σ [ ] -1 * A Map Reduce

Linear Regression Code Vectors x = input(0), y = input(1); Matrices xx = x.Map(x, (a,b) => a.OuterProd(b)); OneMatrix xxs = xx.Sum(); Matrices yx = y.Map(x, (a,b) => a.OuterProd(b)); OneMatrix yxs = yx.Sum(); OneMatrix xxinv = xxs.Map(a => a.Inverse()); OneMatrix A = yxs.Map(xxinv, (a, b) => a.Mult(b)); 52

Expectation Maximization (Gaussians) lines 3 iterations shown

Conclusions Dryad = distributed execution environment Application-independent (semantics oblivious) Supports rich software ecosystem – Relational algebra, Map-reduce, LINQ DryadLINQ = Compiles LINQ to Dryad C# objects and declarative programming.Net and Visual Studio for parallel programming 54

Backup Slides 55

Software Stack 56 Windows Server Cluster Services Distributed Filesystem Dryad Distributed Shell PSQL DryadLINQ Perl SQL server C++ Windows Server C++ CIFS/NTFS legacy code sed, awk, grep, etc. SSIS Scope C# Vectors Machine Learning C# Job queueing, monitoring

Very Large Vector Library PartitionedVector 57 T Scalar TT T

DryadLINQ 58 Declarative programming Integration with Visual Studio Integration with.Net Type safety Automatic serialization Job graph optimizations  static  dynamic Conciseness

Sort & Map-Reduce in DryadLINQ 59

Many similarities Exe + app. model Map+sort+reduce Few policies Program=map+reduce Simple Mature (> 4 years) Widely deployed Hadoop Dryad Map-Reduce Execution layer Job = arbitrary DAG Plug-in policies Program=graph gen. Complex ( features) New (< 2 years) Still growing Internal 60

PLINQ 61 public static IEnumerable DryadSort (IEnumerable source, Func keySelector, IComparer comparer, bool isDescending) { return source.AsParallel().OrderBy(keySelector, comparer); }

Query histogram computation Input: log file (n partitions) Extract queries from log partitions Re-partition by hash of query (k buckets) Compute histogram within each bucket

Naïve histogram topology Pparse lines D hash distribute S quicksort C count occurrences MSmerge sort

Efficient histogram topology Pparse lines D hash distribute S quicksort C count occurrences MSmerge sort M non-deterministic merge Q' is:Each R is: Each MS C M P C S Q' RR k T k n T is: Each MS D C

Final histogram refinement 1,800 computers 43,171 vertices 11,072 processes 11.5 minutes

Data Distribution (Group By) 66 Dest Source Dest Source Dest Source m n m x n

TT [0-?)[?-100) Range-Distribution Manager S DDD SS SSS T static dynamic 67 Hist [0-30),[30-100) [30-100)[0-30) [0-100)

Goal: Declarative Programming 68 X T S XX SS TTT X staticdynamic

JM code vertex code Staging 1. Build 2. Send.exe 3. Start JM 5. Generate graph 7. Serialize vertices 8. Monitor Vertex execution 4. Query cluster resources Cluster services 6. Initialize vertices

SkyServer Query select distinct P.ObjID into results from photoPrimary U, neighbors N, photoPrimary L where U.ObjID = N.ObjID and L.ObjID = N.NeighborObjID and P.ObjID < L.ObjID and abs((U.u-U.g)-(L.u-L.g))<0.05 and abs((U.g-U.r)-(L.g-L.r))<0.05 and abs((U.r-U.i)-(L.r-L.i))<0.05 and abs((U.i-U.z)-(L.i-L.z))<0.05

Number of Computers Speed-up (times) Dryad In-Memory Dryad Two-pass SQLServer 2005 SkyServer Q18 Performance 71