GridBench: A Tool for Benchmarking Grids George Tsouloupas Marios Dikaiakos Dept. of Computer Science University of Cyprus {georget,mdd}@ucy.ac.cy

Introduction Utility of Grid Benchmarking: The experimental evaluation of Grid Infrastructures. Researcher: Characterization of factors of application and infrastructure performance; provide insight to properties of Grid Architectures; Study the dynamic nature of the Grid. User: Evaluate fitness of a collection of resources for running a specific application. Investigate/troubleshoot application performance (by comparison to benchmark runs)

Introduction (cont’d) Utility of Grid Benchmarking (cont’d) Developer: Evaluate fitness of a collection of resources for running a general class of applications, by looking into different programming modelsadigms or which type of middleware to use. Integrator/Administrator: Evaluate the employment of specific middleware Evaluate scheduling and resource allocation algorithms.

Benchmarking the Grid: A Hierarchical Framework Targets: Individual Resources (cluster nodes, Storage Elements) Sites (clusters, SMPs) Grid Constellations (i.e. multiple sites / VOs) Middleware (e.g. mpich-g2) Benchmarks: Micro-benchmarks –isolate basic performance characteristics Micro-kernel Benchmarks –synthetic codes Application Benchmarks –derived from real applications deployed on grids

A Simple Model VO SITE Storage NODE … Site

GridBench Design: Software Architecture Components: RSL/JDL Compiler (XML benchmark descriptions to job description languages) Orchestrator (benchmark execution, collection of results) Benchmark component (the benchmark executable) Monitoring component (collects monitoring information) Archive Database (benchmark result archive) Information provider (results to information services) Benchmark Definition GUI Benchmark Browser GUI

GridBench Design: Software Architecture

GridBench Design: Archival & Publication of Results Benchmark results are archived in a native-XML database (Apache Xindice) The Benchmark results are stored together with their GBDL definition. Results are meaningless without the specific parameters etc. that were used (Reference to) Monitoring data also archived. Comprehension/Analysis of results is (hopefully considerably) enhanced when combined with infrastructure monitoring during the run. Benchmark results related to a Site are published to the Site’s MDS for easy access by users and schedulers

Micro-benchmarks & Micro-kernels Micro-benchmarks at the Worker-node level EPWhetstone: embarrassingly parallel adaptation of the serial whetstone benchmark. BlasBench: evaluate serial performance of the BLAS routines. Micro-benchmarks at the Site level Bonnie++: Storage I/O performance MPPTest: MPI performance measurements Micro-benchmarks at the VO level MPPTest: MPI performance measurements (spanning sites) gb_ftb: File Transfer Benchmark Micro-kernels at the Site level High-Performance Linpack Selected kernels from the NAS Parallel Benchmarks Micro-kernels at the VO level Computationally Intensive Grid Benchmarks

GridBench Design: Meta-data GBDL Main Parts: Components Metrics Parameters Location Co/pre-requisites Monitors

Metrics representation Benchmark-level metrics Vs component-level metrics (e.g. CIGB completion time Vs EP component completion time) Single-value metrics <metric name="epwhetstone_IPS“ type="value"> <vector unit="MIPS">53</vector> </metric> Multi-value metrics <metric name="xfer-rate" type="list"> <vector unit="bps"> 2039430,2083930,1909830,2184750,...</vector> <vector unit="second" toffset="1055327287"> 0,10,20,30,...</vector>

Monitoring GBDL can contain monitoring definitions. <component name="data-transfer" ID="xfer01">...</component> <monitor type="RGMA" source="ccwp71.in2p3.fr:3306" query="select * from NetworkTCPThroughput where NMIdSource='adc0003.cern.ch' and NMIdDestination='ccwp7.in2p3.fr' <parameter name="begin">comp-begin="xfer01"<ameter> <parameter name="end">comp-end="xfer01"<ameter> </monitor> Implies that a monitoring client must be implemented/adapted (potentially several monitoring clients can be employed)

Compilation to RSL/JDL GBDL XML-based GBDL to Job Description Support for simple jobs can be through the use of simple templates. (executable, parameters and locations are transformed to simple RSL/JDL ‘scripts’) Most benchmarks need special command-line parameter formatting, or parameter files. -> ParameterHandler for each benchmark that needs it. Param handler compiler RSL JDL …

Compilation to RSL (example) GBDL <benchmark date= name="nbody"> <component id="A" name="nbody“ type="mpi"> <location type="single"> <resource cpucount="2” name="ce1.grid.ucy"/> name="ce2.grid.ucy"/> </location> <parameter name="executable" type="attribute"> /bin/nbody.exec </parameter> <parameter name="nparticles" type="value"> 1000 </parameter> </component> </benchmark> RSL +(&(resourceManagerContact= “ce1.grid.ucy") (label="subjob 0") (environment= (GLOBUS_DUROC_SUBJOB_INDEX 0)) (count=2) (arguments="-n 1000") (executable="/bin/nbody.exec" )) (&(resourceManagerContact= "ce2.grid.ucy ") (label="subjob 1") (GLOBUS_DUROC_SUBJOB_INDEX 1)) (arguments="-n 1000")

Experiments User Interface Compiler GridBench UI Monitor CE Benchmark Component Executable RSL GBDL GRAM Orchestrator std out GRAM std out CE Benchmark Component Executable Archiver SE Xindice

Results

WIP and Future Work Complete GBDL specification Revise Metric and Monitoring representations Implementation of more benchmarks.