The Cloud Workloads Archive: A Status Report Special thanks to Ion for this opportunity! Alexandru Iosup Rean Griffith, Andrew Konwinski, Matei Zaharia, Ali Ghodsi, Ion Stoica Parallel and Distributed Systems Group, Delft University of Technology, The Netherlands RADLab, University of California, Berkeley, USA April 22, 2017 Berkeley, CA, USA

About the Team Recent Work in Performance Speaker: Alexandru Iosup The Grid Workloads Archive (Nov 2006) The Failure Trace Archive (Nov 2009) Analysis of Facebook, Yahoo, and Google data center workloads (2009-2010) The Peer-to-Peer Trace Archive (Apr 2010) Tools: GrenchMark workload-based grid benchmarking, RAIN Speaker: Alexandru Iosup Systems work: Tribler (P2P file sharing), Koala (grid scheduling), POGGI and CAMEO (massively multiplayer online gaming) Performance evaluation of clouds for sci.comp.: EC2 & three others Team of 15+ active collaborators in NL, AT, RO, US Happy to be in Berkeley until September April 22, 2017

Traces: Sine Qua Non in Comp.Sys.Res. “My system/method/algorithm is better than yours (on my carefully crafted workload)” Unrealistic (trivial): Prove that ‘prioritize jobs from users whose name starts with A’ is a good scheduling policy Realistic? 85% jobs are short, 15% are long Major problem in Computer Systems research Workload Trace = recording of real activity from a (real) system, often as a sequence of jobs / requests submitted by users for execution Main use: compare and cross-validate new job and resource management techniques and algorithms Major problem: obtaining and using real workload traces April 22, 2017

Previous Data Sharing Efforts Critical datasets in computer science Grid Workloads Archive Failure Trace Archive Peer-to-Peer Trace Archive Game Trace Archive (soon) … PWA, ITA, CRAWDAD, … 1,000s of scientists From theory to practice Research Question: Are data center workloads unique? (vs GWA, PWA, …) Dataset Size Year 1GB 10GB 100GB 1TB 1TB/yr P2PTA GamTA ‘09 ‘10 ‘11 ‘06 April 22, 2017

Agenda Introduction & Motivation The Cloud Workloads Archive: What’s in a Name? Format and Tools Contents Analysis & Modeling Applications Take Home Message April 22, 2017

The Cloud Workloads Archive (CWA) What’s in a Name? CWA = Public collection of cloud/data center workload traces and of tools to process these traces; allows us to: Compare and cross-validate new job and resource management techniques and algorithms, across various workload traces Determine which (part of a) trace is most interesting for a specific job and resource management technique or algorithm Design a general model for data center workloads, and validate it with various real workload traces Evaluate the generality of a particular workload trace, to determine if results are biased towards a particular trace Analyze the evolution of workload characteristics across long timescales, both intra- and inter-trace April 22, 2017

One Format Fits Them All Flat format Job and Tasks Summary (20 unique data fields) and Detail (60 fields) Categories of information Shared with GWA, PWA: Time, Disk, Memory, Net Jobs/Tasks that change resource consumption profile MapReduce-specific (two-thirds data fields) CWJ CWJD CWT CWTD A. Iosup, R. Griffith, A. Konwinski, M. Zaharia, A. Ghodsi, I. Stoica, Data Format for the Cloud Workloads Archive, v.3, 13/07/10 April 22, 2017

CWA Contents: Large-Scale Workloads Trace ID System Size J/T/Obs Period Notes CWA-01 Facebook 1.1M/-/- 5m/2009 Time & IO CWA-02 Yahoo M 28K/28M/- 20d/2009 ~Full detail CWA-03 Facebook 2 61K/10M/- 10d/2009 Full detail CWA-04 Facebook 3 ?/?/- 10d/01-2010 CWA-05 Facebook 4 3m/02+2010 CWA-06 Google 2 25 Aug 2010 CWA-07 eBay 23 Sep 2010 CWA-08 Twitter Need help! CWA-09? Google 9K/177K/4M 7h/2009 Coarse,Period Tools Convert to CWA format Analyze and model automatically  Report April 22, 2017

Agenda Introduction & Motivation The Cloud Workloads Archive: What’s in a Name? Format and Tools Contents Analysis & Modeling Applications Take Home Message April 22, 2017

Types of Analysis Analysis Focus Time-related Structure-related Run, Wait, Resp.Time Bounded Slowdown Structure-related Number of tasks IO-related IO sizes and ratios Status-related Sys. Utilization-related Counts/Ratios Analysis Type Basic statistics Evolution over time Correlations Data Break-down Overall By Task Type (M/R) By App. Type (ID) By User (ID) By Duration (Short) April 22, 2017

Types of Analysis Sys.U., Over Time, By RunTime Also 1h, 10mins, … counting intervals Study Short-/Long- Range Dependence (self-similarity) Also Job count, Running/Waiting counts, … Study system utilization behavior April 22, 2017

Modeling Process Well-known prob. distrib. MLE to fit Goodness-of-Fit Normal, Exp, LogNormal, Gamma, Weibull, Gen-Pareto, MLE to fit Fit known distribution to empirical distribution  parameters Goodness-of-Fit Assess how good the fit is; select best-fitting distribution Kolmogorov-Smirnov: sensitive to body of distribution + D stat Anderson-Darling: sensitive to tails of distribution Hybrid method*: works for very large populations *Kondo et al., Failure Trace Archive, CCGrid’10, Best Paper Award. April 22, 2017

Main Results: Basic Stats Trace ID TRunTime [s] #Tasks/Job Pk.Arr.Rate/D # users CWA-01 165J n/a 21KJ/-T CWA-02 512/80med 901/712Map 6KJ/3.2MT CWA-03 433/86med 153/143Map 8KJ/2MT 18 GWA-T1 370 5—20 -/20KT 332 GWA-T3 89,274 -/8KT 387 GWA-T6 14,599 -/22.5KT 206 GWA-T10 31,964 -/1.6KTph 216 GWA-T11 8,971 -/22KTph 412 MapReduce vs Grid workloads [vs Parallel Prod. Env.] Massive short tasks vs Many long tasks vs Few very long tasks Fewer users for MapReduce environments? TODO: Analyse amounts per core April 22, 2017

Agenda Introduction & Motivation The Cloud Workloads Archive: What’s in a Name? Format and Tools Contents Analysis & Modeling Applications Take Home Message April 22, 2017

Applications Mesos running mixtures of workloads Workloads: MPI, MapReduce, grid, … Find bottlenecks Find workloads that are particularly difficult to run Improve the system! Status: in progress, using cluster in Finland (Petri Savolainen) All the apps typical to trace-based work: design, validation, and comparison of algorithms, methods, and systems. April 22, 2017

Agenda Introduction & Motivation The Cloud Workloads Archive: What’s in a Name? Format and Tools Contents Analysis & Modeling Applications Take Home Message April 22, 2017

Take Home Message Cloud Workloads Archive Datasets Tools to convert, analyze, and model the datasets Need your help to collect more traces Converted and analyzed three MapReduce workloads Different from grid and parallel production environment workloads (ask about additional proof and let me show a couple more slides) Invariants? Applications 1: Model of Cloud/MapReduce workloads 2: Test and improve Mesos April 22, 2017

Continuing Our Collaboration Scheduling mixtures of grid/HPC/cloud workloads Scheduling and resource management in practice Modeling aspects of cloud infrastructure and workloads Condor on top of Mesos Massively Social Gaming and Mesos Step 1: Game analytics and social network analysis in Mesos … April 22, 2017

Thank you! Questions? Observations? Alex Iosup, Rean Griffith, Andrew Konwinski, Matei Zaharia, Ali Ghodsi, Ion Stoica email: A.Iosup@tudelft.nl Thanks for all: AliG, Andrew, AndyK, Ari, Beth, Blaine, David, Ion, Justin, Lucian, Matei, Petri, Rean, Tim, … More Information: The Grid Workloads Archive: gwa.ewi.tudelft.nl The Failure Trace Archive: fta.inria.fr The GrenchMark perf. eval. tool: grenchmark.st.ewi.tudelft.nl Cloud research: www.st.ewi.tudelft.nl/~iosup/research_cloud.html see PDS publication database at: www.pds.twi.tudelft.nl/ Big thanks to our collaborators: U. Wisc.-Madison, U Chicago, U Dortmund, U Innsbruck, LRI/INRIA Paris, INRIA Grenoble, U Leiden, Politehnica University of Bucharest, Technion, … April 22, 2017

Additional Slides April 22, 2017

Main Results: Basic Stats Trace ID Total IO [MB] Rd. [MB] Wr [%] HDFS Wr[MB] CWA-01 10,934 6,805 38% 1,538 CWA-02 75,546 47,539 37% 8,563 CWA-03 - GWA12.1 469 174 63% n/a GWA12.2 144 114 21% GWA12.3 161 130 19% GWA12.4 389 33 92% GWA12.5 330 31 91% MapReduce vs Grid workloads IO-intensive vs Compute-intensive Constant Wr[%]~40%IO for MapReduce traces? TODO: More MapReduce traces to validate findings April 22, 2017

Main Results Two-mode trace  do NOT analyze as whole April 22, 2017

April 22, 2017