From LINQ to DryadLINQ Michael Isard Workshop on Data-Intensive Scientific Computing Using DryadLINQ

Overview From sequential code to parallel execution Dryad fundamentals Simple program example, plan for practicals

Distributed computation Single computer, shared memory – All objects always available for read and write Cluster of workstations – Each computer sees a subset of objects – Writes on one computer must be explicitly shared System automatically handles complexity – Needs some help

Data-parallel computation LINQ is high-level declarative specification Same action on entire collection of objects set.Select(x => f(x)) – Compute f(x) on each x in set, independently set.GroupBy(x => key(x)) – Group by unique keys, independently set.OrderBy(x => key(x)) – Sort whole set (system chooses how)

Distributed cluster computing Dataset is stored on local disks of cluster set set.0 set.7 set.1 set.6set.4 set.3set.2 set.5

Distributed cluster computing Dataset is stored on local disks of cluster set.0 set.7 set.1 set.6set.4 set.3set.2 set.5

Simple distributed computation var set2 = set.Select(x => f(x)) set set2

Simple distributed computation var set2 = set.Select(x => f(x)) set.0 set.7 set.1 set.6 set.4 set.3 set.2 set.5 set2.0 set2.1 set2.2 set2.3 set2.4 set2.5 set2.6 set2.7

Simple distributed computation var set2 = set.Select(x => f(x)) set.0 set.1 set.2set.3 set.4set.5set.6 set.7 set2.0set2.1set2.2set2.3set2.4set2.5set2.6set2.7 ffffffff

Simple distributed computation var set2 = set.Select(x => f(x)) set.0 set.1 set.2set.3 set.4set.5set.6 set.7 set2.0set2.1set2.2set2.3set2.4set2.5set2.6set2.7 ffffffff

Distributed acyclic graph Computation reads and writes along edges Graph shows parallelism via independence Goals of DryadLINQ optimizer – Extract parallelism (find independent work) – Control data skew (balance work across nodes) – Limit cross-computer data transfer

Distributed grouping var groups = set.GroupBy(x => x.key) set is a collection of records each with a key Don’t know what keys are present – Or in which partitions First, reorganize data – All records with same key on same computer Then can do final grouping in parallel

Distributed grouping var groups = set.GroupBy(x => x.key) set hash partition by key group locally groups acac adad dbdb baba acac a c a a adad d d b b dbdb baba

Distributed grouping var groups = set.GroupBy(x => x.key) set hash partition by key group locally groups acac adad dbdb baba acac a c a adad d d b b dbdb baba a a a c b d a a a c b d

Distributed sorting var sorted = set.OrderBy(x => x.key) range partition by key sort locally sorted set sample compute histogram

Distributed sorting var sorted = set.OrderBy(x => x.key) range partition by key sort locally sorted set sample compute histogram [1,1] [2,100]

Distributed sorting var sorted = set.OrderBy(x => x.key) range partition by key sort locally sorted set sample compute histogram [1,1] [2,100]

Additional optimizations var histogram = set.GroupBy(x => x.key).Select(x => {x.Key, x.Count()}) set hash partition by key group locally histogram a b b a a d b d a b b a a b b a a d b d b d b b d a b b a count

Additional optimizations var histogram = set.GroupBy(x => x.key).Select(x => {x.Key, x.Count()}) set hash partition by key group locally histogram a a a count a b b a a d b d a b b a a b b a a d b d a b b a a b d b d b b b b d d b b b d d

Additional optimizations var histogram = set.GroupBy(x => x.key).Select(x => {x.Key, x.Count()}) set hash partition by key group locally histogram a,6 b,6 d,4 count a b b a a d b d a b b a a b b a a d b d a b b a a a a b b b d d a a a a,6 b,6 d,4 b b b d d

var histogram = set.GroupBy(x => x.key).Select(x => {x.Key, x.Count()}) set hash partition by key group locally histogram a b b a a d b d a b b a a,2 b,2 a,2 d,2 b,2 d,2 b,2 d,2 b,2 d,2 a,2 b,2 combine counts group locally a,2 b,2 a,2 d,2 b,2 d,2 a,2 b,2

var histogram = set.GroupBy(x => x.key).Select(x => {x.Key, x.Count()}) set hash partition by key group locally histogram a b b a a d b d a b b a a,2 b,2 a,2 d,2 b,2 d,2 b,2 d,2 b,2 d,2 a,2 b,2 combine counts group locally a,2 b,2 a,2 d,2 b,2 d,2 a,2 b,2 a,2 a,2 a,2 b,2 b,2 b,2 d,2 a,2 a,2 a,2 b,2 b,2 b,2 d,2

var histogram = set.GroupBy(x => x.key).Select(x => {x.Key, x.Count()}) set hash partition by key group locally histogram a b b a a d b d a b b a a,2 b,2 a,2 d,2 b,2 d,2 a,2 b,2 combine counts group locally a,2 b,2 a,2 d,2 b,2 d,2 a,2 b,2 a,2 a,2 a,2 b,2 b,2 b,2 d,2 a,2 a,2 a,2 b,2 b,2 b,2 d,2 a,6 b,6 d,4 b,6 d,4

What Dryad does Abstracts cluster resources – Set of computers, network topology, etc. Schedule DAG: choose cluster computers – Fairly among competing jobs – So computation is close to data Recovers from transient failures – Rerun computations on machine or network fault – Speculate duplicates for slow computations

Resources are virtualized Each graph node is process – Writes outputs to disk – Reads inputs from upstream nodes’ output files Graph generally larger than cluster – 1TB input, 250MB partition, 4000 parts Cluster is shared – Don’t size program for exact cluster – Use whatever share of resources are available

What controls parallelism Initially based on partitioning of inputs After reorganization, system or user decides

DryadLINQ-specific operators set = PartitionedTable.Get (uri) set.ToPartitionedTable(uri) set.HashPartition(x => f(x), numberOfParts) set.AssumeHashPartition(x => f(x)) [Associative] f(x) { … } RangePartition(…), Apply(…), Fork(…) [Decomposable], [Homomorphic], [Resource] Field mappings, Multiple partitioned tables, …

using System; using System.Collections.Generic; using System.Linq; using System.Text; using LinqToDryad; namespace Count { class Program { public const string inputUri static void Main(string[] args) { PartitionedTable table = PartitionedTable.Get (inputUri); Console.WriteLine("Lines: {0}", table.Count()); Console.ReadKey(); }

Form into groups 9 groups, one MSRI member per group Try to pick common interest for project later

sherwood-246 — sherwood-253,sherwood-255 d:\dryad\data\Workshop\DryadLINQ\samples Count, Points, Robots Cluster job browser d:\dryad\data\Workshop\DryadLINQ\job_browser\DryadAnalysis.exe TidyFS (file system) browser d:\dryad\data\Workshop\DryadLINQ\bin\retail\tidyfsexplorerwpf.exe