Large-scale Machine Learning using DryadLINQ Mihai Budiu Microsoft Research, Silicon Valley Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop Stanford, June 11, 2019

Goal of DryadLINQ 2

Software Stack 3 Windows Server Cluster services Cluster storage Dryad DryadLINQ Windows Server Applications.Net + LINQ

Dryad = Execution Layer 4 Job (application) Dryad Cluster Pipeline Unix Shell Machine ≈

Collection.NET objects of type T LINQ Data Model

LINQ Language Summary 6 Where (filter) Select (map) GroupBy OrderBy (sort) Aggregate (fold) Join Input

LINQ 7 Dryad => DryadLINQ

DryadLINQ Data Model 8 Partition Collection.Net objects

Collection collection; static bool IsLegal(Key c); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value}; 9 DryadLINQ = LINQ + Dryad C# collection results C# Code Dryad job Data

Example: Natal Training 10

Natal Problem 11 Recognize players from depth map At frame rate Low resource usage

Learn from Data 12 Motion Capture (ground truth) Classifier Training examples Machine learning Rasterize

Running on Xbox 13

Cluster-based training 14 Classifier Training examples Dryad DryadLINQ Machine learning

You can have it! Dryad+DryadLINQ available for download – Academic license – Commercial evaluation license Runs on Windows HPC platform Dryad is in binary form, DryadLINQ in source Requires signing a 3-page licensing agreement

Conclusions 17 =