1. June 25, 2015 1 GrenchMark: Synthetic workloads for Grids First Demo at TU Delft A. Iosup, D.H.J. Epema PDS Group, ST/EWI, TU Delft

2. June 25, 2015 2 Evaluating Grid schedulers performance Grid schedulers performance Qualitative metrics supported application types, advanced fault tolerance, advanced... Quantitative metrics resource consumption, system performance, success rate Other metrics cost, single number system description Needs Applications, workloads, more workloads…

3. June 25, 2015 3 Synthetic workloads for Grid schedulers Good synthetic workloads for Grid schedulers Specific scheduler comparison requirements (metrics, jobs inter-arrival time,...) Many different types of representative Grid applications Traditional software engineering requirements (flexibility, extensibility, usability,...) Can also be used for... Functionality testing and system tuning Application performance testing Systems design and procurement …

4. June 25, 2015 4 Representative Grid applications Unitary applications Just one scheduling unit (otherwise recursive definition) Examples: Sequential, MPI, Java RMI, Ibis, … Composite applications Composed of several unitary or composite applications Examples: Parameter sweeps, chains of tasks, DAGs, workflows, …

5. June 25, 2015 5 Outline Introduction The GrenchMark framework Experience with GrenchMark Extending GrenchMark Conclusions [done] [here]

6. June 25, 2015 6 The GrenchMark framework What’s in a name? grid benchmark → help standardizing the testing procedures, but benchmarks are too early… GrenchMark A systematic approach to testing Grid schedulers A set of metrics for comparing schedulers A set of representative Grid applications Both real and synthetic Easy-to-use tools to create synthetic workloads Flexible, portable, extensible Can also be used for testing other Grid components

7. June 25, 2015 7 The GrenchMark framework

8. June 25, 2015 8 GrenchMark: Preliminary notions Job, workload, workload unit Job = Set of components (support for co-allocation) [Job = one program execution / the basic scheduling unit / … ] Workload = Set of Jobs Workload Unit = Set of jobs with the same property, generated from one description line (definition useful only for workload generator) Other JDF = Job description file Inter arrival time

9. June 25, 2015 9 GrenchMark: Some more notions Job site type / per site description Single – run at one site Fixed – run at several sites, all specified my license is on those machines Un-Fixed – run at several sites, all unspecified I can run anywhere, just give me the resources Semi-fixed – run at several sites, some specified I prefer those machines, but I can work anywhere Inter-arrival time distributions Constant, Uniform, Normal, Exponential(λ), Poisson(Mean), HExp2, HPoisson2, Weibull, LogNormal, Gamma (~)

10. June 25, 2015 10 WL description: an example Describe the workload to be generated in a few lines Very simple language + custom extensions (Native) (ExternalFile field) Support Co-allocation Start time, inter-arrival time Mixes of jobs

11. June 25, 2015 11 The GrenchMark process $./wl-gen.py wl-desc.in $./wl-submit.py out/wl-to-submit.wl 1 2 1 2 Semi-automated Sample run: 4 lines of description 100 jobs / 411 components 100 files / 132 directories 300KB data Sample run: defined inter-arrival rate – submission delay +/- 0.01s 100 JDFs

12. June 25, 2015 12 GrenchMark status Already done in Python [http://www.python.org] Generator + Globus, KOALA generators + RSL printer Submitter Results analyzer (crude) Applications Unitary, 3 types: sequential, MPI, Ibis (Java) +35 different applications Ongoing work Composite applications Automated results analyzer

13. June 25, 2015 13 Demo: Generating mixes of jobs 10 jobs 8 MPI, multi-component jobs (need co-allocation) 2 sequential

14. June 25, 2015 14 Outline Introduction The GrenchMark framework Experience with GrenchMark Extending GrenchMark Conclusions [done] [here] [done]

15. June 25, 2015 15 GrenchMark for testing KOALA… Testing 3 different runners: drunner, grunner, krunner Pre-release status: supposed stable Workloads with different jobs requirements, inter-arrival rates, co-allocated v. single site jobs… Evaluate Jobs success rate, KOALA’s overhead and bottlenecks Results +5,000 jobs successfully run 2 major bugs first day, +10 bugs overall (all fixed) KOALA is officially released (full credit to KOALA developers, 10x for testing with GrenchMark)

16. June 25, 2015 16 GrenchMark for testing KOALA: A full workload example KOALA test workload, run 10 times: Globus MPICH-G2 / MPI jobs Components: 4 and 8 Component Sizes: 4, 8 and 16 Inter-arrival time: Poisson(5s), spikes Co-allocation, 1 site Submit time 1 day Generate: Submit: Total: 3200 jobs, 19200 components / 3k files, 4k dirs Timing: 30s generate / 86,400s submit (1 day) $ wl-gen.py --duration=86400000 wl-desc.in $ wl-submit.py –onefile out/wl-desc.in

17. June 25, 2015 17 … and DAS-2’s functionality Already done Evaluate for KOALA + Globus + DAS-2 jobs success rate, turnaround time, middleware overhead, types and sources of errors Results 5 workloads 500 jobs A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema, Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, 2005 (submitted). Currently examine DAS-2 support for composite applications

18. June 25, 2015 18 Outline Introduction The GrenchMark framework Experience with GrenchMark Extending GrenchMark Conclusions [done] [here] [done]

19. June 25, 2015 19 Extending GrenchMark (1) Motto: Extending GrenchMark is easy! Need: Good knowledge about the application type Good understanding of workflows, or Good understanding of grid middleware Minimal Python knowledge (follow the official 2hrs tutorial: http://docs.python.org/tut/tut.html)

20. June 25, 2015 20 Extending GrenchMark (2) 1.Write your own Job Generators a function with a predefined name in a Python module auto-loaded 2.Write your own Unit Generators a function with a predefined name in a Python module auto-loaded 3.Interface with C/C++, Ruby, Perl, Java, … define your own protocol 4.Write your own printers a function with a predefined name in a Python module auto-loaded

21. June 25, 2015 21 Outline Introduction The GrenchMark framework Experience with GrenchMark Extending GrenchMark Conclusions [done] [here] [done]

22. June 25, 2015 22 Conclusions and future work GrenchMark generates diverse workloads of Grid applications easy-to-use, flexible, portable, extensible, … Experience used GrenchMark to test KOALA’s functionality and performance. used GrenchMark to test some DAS Grid functionality. +5,000 jobs generated and run … and counting. (more) advertisment Have specific Grid applications types you would like to test? Test with GrenchMark!

23. June 25, 2015 23 Thank you! Questions? Remarks? Observations? All welcome! Grenchmark http://grenchmark.st.ewi.tudelft.nl/ [10x Paulo] Alexandru IOSUP http://grenchmark.st.ewi.tudelft.nl/ TU Delft A.Iosup@ewi.tudelft.nl http://www.pds.ewi.tudelft.nl/~iosup/index.html [google: “iosup”]