DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Yuan Yu, Michael Isard, Dennis Fetterly, Mihai Budiu, Úlfar Erlingsson, Pradeep Kumar Gunda, Jon Currey Microsoft Research Silicon Valley

Distributed Data-Parallel Computing Research problem: How to write distributed data-parallel programs for a compute cluster? The DryadLINQ programming model – Sequential, single machine programming abstraction – Same program runs on single-core, multi-core, or cluster – Familiar programming languages – Familiar development environment

DryadLINQ Overview Automatic query plan generation by DryadLINQ Automatic distributed execution by Dryad

LINQ Microsoft’s Language INtegrated Query – Available in Visual Studio products A set of operators to manipulate datasets in.NET – Support traditional relational operators Select, Join, GroupBy, Aggregate, etc. – Integrated into.NET programming languages Programs can call operators Operators can invoke arbitrary.NET functions Data model – Data elements are strongly typed.NET objects – Much more expressive than SQL tables Highly extensible – Add new custom operators – Add new execution providers

LINQ System Architecture PLINQ Local machine.Net program (C#, VB, F#, etc) Execution engines Query Objects LINQ-to-SQL DryadLINQ LINQ-to-Obj LINQ provider interface Scalability Single-core Multi-core Cluster

Dryad System Architecture 6 Files, TCP, FIFO, Network job schedule data plane control plane NSPD V VV Job managercluster

A Simple LINQ Example: Word Count Count word frequency in a set of documents: var docs = [A collection of documents]; var words = docs.SelectMany(doc => doc.words); var groups = words.GroupBy(word => word); var counts = groups.Select(g => new WordCount(g.Key, g.Count()));

Word Count in DryadLINQ Count word frequency in a set of documents: var docs = DryadLinq.GetTable (“file://docs.txt”); var words = docs.SelectMany(doc => doc.words); var groups = words.GroupBy(word => word); var counts = groups.Select(g => new WordCount(g.Key, g.Count())); counts.ToDryadTable(“counts.txt”);

Distributed Execution of Word Count SM DryadLINQ GB S LINQ expression IN OUT Dryad execution

DryadLINQ System Architecture 10 DryadLINQ Client machine (11) Distributed query plan.NET program Query Expr Data center Output Tables Results Input Tables Invoke Query Output DryadTable Dryad Execution.Net Objects JM ToTable foreach Vertex code

DryadLINQ Internals Distributed execution plan – Static optimizations: pipelining, eager aggregation, etc. – Dynamic optimizations: data-dependent partitioning, dynamic aggregation, etc. Automatic code generation – Vertex code that runs on vertices – Channel serialization code – Callback code for runtime optimizations – Automatically distributed to cluster machines Separate LINQ query from its local context – Distribute referenced objects to cluster machines – Distribute application DLLs to cluster machines

Execution Plan for Word Count 12 (1) SM GB S SM Q GB C D MS GB Sum SelectMany sort groupby count distribute mergesort groupby Sum pipelined

Execution Plan for Word Count 13 (1) SM GB S SM Q GB C D MS GB Sum (2) SM Q GB C D MS GB Sum SM Q GB C D MS GB Sum SM Q GB C D MS GB Sum

MapReduce in DryadLINQ 14 MapReduce(source, // sequence of Ts mapper, // T -> Ms keySelector, // M -> K reducer) // (K, Ms) -> Rs { var map = source.SelectMany(mapper); var group = map.GroupBy(keySelector); var result = group.SelectMany(reducer); return result; // sequence of Rs }

Map-Reduce Plan (When reduce is combiner-enabled) M Q G1G1 C D MS G2G2 R M Q G1G1 C D G2G2 R M Q G1G1 C D G2G2 R G2G2 R map sort groupby combine distribute mergesort groupby reduce mergesort groupby reduce map Dynamic aggregation reduce

An Example: PageRank Ranks web pages by propagating scores along hyperlink structure Each iteration as an SQL query: 1.Join edges with ranks 2.Distribute ranks on edges 3.GroupBy edge destination 4.Aggregate into ranks 5.Repeat

One PageRank Step in DryadLINQ // one step of pagerank: dispersing and re-accumulating rank public static IQueryable PRStep(IQueryable pages, IQueryable ranks) { // join pages with ranks, and disperse updates var updates = from page in pages join rank in ranks on page.name equals rank.name select page.Disperse(rank); // re-accumulate. return from list in updates from rank in list group rank.rank by rank.name into g select new Rank(g.Key, g.Sum()); }

The Complete PageRank Program var pages = DryadLinq.GetTable (“file://pages.txt”); var ranks = pages.Select(page => new Rank(page.name, 1.0)); // repeat the iterative computation several times for (int iter = 0; iter < iterations; iter++) { ranks = PRStep(pages, ranks); } ranks.ToDryadTable (“outputranks.txt”); public struct Page { public UInt64 name; public Int64 degree; public UInt64[] links; public Page(UInt64 n, Int64 d, UInt64[] l) { name = n; degree = d; links = l; } public Rank[] Disperse(Rank rank) { Rank[] ranks = new Rank[links.Length]; double score = rank.rank / this.degree; for (int i = 0; i < ranks.Length; i++) { ranks[i] = new Rank(this.links[i], score); } return ranks; } } public struct Rank { public UInt64 name; public double rank; public Rank(UInt64 n, double r) { name = n; rank = r; } } public static IQueryable PRStep(IQueryable pages, IQueryable ranks) { // join pages with ranks, and disperse updates var updates = from page in pages join rank in ranks on page.name equals rank.name select page.Disperse(rank); // re-accumulate. return from list in updates from rank in list group rank.rank by rank.name into g select new Rank(g.Key, g.Sum()); }

One Iteration PageRank J S G C D M G R J S G C D M G R J S G C D Join pages and ranks Disperse page’s rank Group rank by page Accumulate ranks, partially Hash distribute Merge the data Group rank by page Accumulate ranks M G R … … Dynamic aggregation

Multi-Iteration PageRank pagesranks Iteration 1 Iteration 2 Iteration 3 Memory FIFO

LINQ System Architecture PLINQ Local machine.Net program (C#, VB, F#, etc) Execution engines Query Objects LINQ-to-SQL DryadLINQ LINQ-to-Obj LINQ provider interface Scalability Single-core Multi-core Cluster

Combining with PLINQ 22 Query DryadLINQ PLINQ subquery

Combining with LINQ-to-SQL 23 DryadLINQ Subquery Query LINQ-to-SQL

Combining with LINQ-to-Objects Query DryadLINQ Local machine Cluster LINQ-to-Object debug production

Current Status Works with any LINQ enabled language – C#, VB, F#, IronPython, … Works with multiple storage systems – NTFS, SQL, Windows Azure, Cosmos DFS Released internally within Microsoft – Used on a variety of applications External academic release announced at PDC – DryadLINQ in source, Dryad in binary – UW, UCSD, Indiana, ETH, Cambridge, …

Image Processing Cosmos DFSSQL Servers Software Stack 26 Windows Server Cluster Services Azure Platform Dryad DryadLINQ Windows Server Other Languages CIFS/NTFS Machine Learning Graph Analysis Data Mining Applications … Other Applications

Lessons Deep language integration worked out well – Easy expression of massive parallelism – Elegant, unified data model based on.NET objects – Multiple language support: C#, VB, F#, … – Visual Studio and.NET libraries – Interoperate with PLINQ, LINQ-to-SQL, LINQ-to-Object, … Key enablers – Language side LINQ extensibility: custom operators/providers.NET reflection, dynamic code generation, … – System side Dryad generality: DAG model, runtime callback Clean separation of Dryad and DryadLINQ

Future Directions Goal: Use a cluster as if it is a single computer – Dryad/DryadLINQ represent a modest step On-going research – What can we write with DryadLINQ? Where and how to generalize the programming model? – Performance, usability, etc. How to debug/profile/analyze DryadLINQ apps? – Job scheduling How to schedule/execute N concurrent jobs? – Caching and incremental computation How to reuse previously computed results? – Static program checking A very compelling case for program analysis? Better catch bugs statically than fighting them in the cloud?

Conclusions A powerful, elegant programming environment for large-scale data-parallel computing To request a copy of Dryad/DryadLINQ, contact For academic use only See a demo of the system at the poster session!