1. June 3, 2015 Synthetic Grid Workloads with Ibis, K OALA, and GrenchMark CoreGRID Integration Workshop, Pisa A. Iosup, D.H.J. Epema Jason Maassen, Rob van Nieuwpoort PDS Group, EEMCS, TU DelftComputer Science Dept., VU Amsterdam

2. June 3, 2015 2 Collaboration context The Partners WP6 Technische Universiteit Delft (TU Delft) Alexandru Iosup, Dick Epema WP3 Vrije Universiteit Amsterdam (VUA) Jason Maassen, Rob van Nieuwpoort This work is the result of the collaboration of the Partners, within the CoreGRID framework.

3. June 3, 2015 3 Outline Introduction The GrenchMark framework Experience with GrenchMark Extending GrenchMark Conclusions

4. June 3, 2015 4 Introduction Synthetic worklods are the basis of grid schedulers comparison [our work], and of... Grid systems analysis Functionality testing in grid environments Comparing grid settings Grid systems design and procurement Many others … Synthetic grid workloads Many different types of representative Grid applications Need a standard tool for synthetic grid workloads generation and submission

5. June 3, 2015 5 Representative Grid applications (1/3) Unitary applications Just one scheduling unit Examples: Sequential, MPI, Java RMI, Ibis, … Composite applications Composed of several unitary or composite applications Examples: Parameter sweeps, chains of tasks, DAGs, workflows, …

6. June 3, 2015 6 Representative Grid applications (2/3) Unitary: synthetic [I] I1 data elements from stdin [O] O1 data elements to stdout for each N steps (i) # superstep [I] I2 data elements from stdin for each M memory locations (j) [M] get item j of size S [P] compute C computation units per memory item (fmul), and store results into temp memory location [M] put S values from temp to location j [O] O2 data elements to stdout [O] O3 data elements to stdout Also with I/O Also parallel, with communication Synthetic application model with CPU cycles, memory, and I/O requirements

7. June 3, 2015 7 Representative Grid applications (3/3) Unitary: Ibis jobs No modeling, just real parallel applications [ http://www.cs.vu.nl/ibis ]http://www.cs.vu.nl/ibis physical simulation parallel rendering computational mathematics state space search bioinformatics data compression grid methods optimization

8. June 3, 2015 8 The K OALA scheduler Grid scheduler [ http://www.st.ewi.tudelft.nl/koala / ]http://www.st.ewi.tudelft.nl/koala / Native processor and data co-allocation support H.H. Mohamed and D.H.J. Epema, An Evaluation of the Close-to-Files Processor and Data Co-Allocation Policy in Multiclusters, IEEE Int'l Conference Cluster Computing, Sep 2004. By PDS group, EEMCS, PDS / Virtual Laboratory for e-Science Application-specific support Ibis jobs Multi-site MPI jobs …

9. June 3, 2015 9 The GrenchMark framework What’s in a name? grid benchmark → help standardizing the testing procedures, but benchmarks are too early… GrenchMark A systematic approach to testing Grid schedulers A set of metrics for comparing schedulers A set of representative Grid applications Both real and synthetic Easy-to-use tools to create synthetic workloads Flexible, portable, extensible Can also be used for testing other Grid components, …

10. June 3, 2015 10 The GrenchMark framework

11. June 3, 2015 11 GrenchMark status Already done in Python Generator + Globus, KOALA generators + RSL printer Submitter Results analyzer (crude) Applications Unitary, 3 types: sequential, MPI, Ibis (Java) +35 different applications Composite applications: DAG-based Ongoing work Automated results analyzer Better modeling capabilities

12. June 3, 2015 12 GrenchMark for testing K OALA … K OALA Non-fixed, fixed, semi-fixed jobs Test & evaluate Status: pre-release, supposed stable Workloads with different jobs requirements, inter-arrival rates, co- allocated v. single site jobs… Jobs success rate, K OALA ’s overhead and bottlenecks Results +5,000 jobs successfully run 2 major bugs first day, +10 bugs overall (all fixed) K OALA is officially released (full credit to KOALA developers, 10x for testing with GrenchMark)

13. June 3, 2015 13 … and DAS-2’s functionality … Already done Evaluate for KOALA + Globus + DAS-2 jobs success rate, turnaround time, middleware overhead, types and sources of errors Results 5 workloads 500 jobs A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema, Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, CoreGRID IW, Nov 2005 [ here, now].

14. June 3, 2015 14 … and for analyzing, testing, and comparing grids Grid system analysis Performance testing / analysis What-if analysis Functionality testing in grid environments System functionality testing Periodic testing Comparing grid settings Single-site vs. co-allocated jobs Unitary vs. composite jobs A.Iosup, D.H.J.Epema, GrenchMark: A Framework for Analyzing, Testing, and Comparing Grids, 2005 (submitted).

15. June 3, 2015 15 Extending GrenchMark Motto: Extending GrenchMark is easy! 1.Write your own Job Generators a function with a predefined name in a Python module auto-loaded 2.Write your own Unit Generators a function with a predefined name in a Python module auto-loaded 3.Interface with C/C++, Ruby, Perl, Java, … define your own protocol 4.Write your own printers a function with a predefined name in a Python module auto-loaded

16. June 3, 2015 16 Conclusions and future work GrenchMark generates diverse workloads of Grid applications easy-to-use, flexible, portable, extensible, … Experience used GrenchMark to test KOALA’s functionality and performance. used GrenchMark to test some DAS Grid functionality. +5,000 jobs generated and run … and counting. Ongoing collaboration within CoreGRID with U. Dortmund group (R. Yahyapour, B. Song, C. Ernemann, …) … advertisement Have specific Grid applications types you would like to test? Test with GrenchMark!

17. June 3, 2015 17 Thank you! Questions? Remarks? Observations? All welcome! Grenchmark http://grenchmark.st.ewi.tudelft.nl/ [ “grenchmark”] Alexandru IOSUP http://grenchmark.st.ewi.tudelft.nl/ TU Delft A.Iosup@ewi.tudelft.nl [ “iosup”] Also: H. Mohamed and W. Lammers (K OALA ), P. Anita (admin) G. Wrzesinska, N. Drost, and M. den Burger (Ibis)