1. GridBench: A Tool for Benchmarking Grids
George Tsouloupas Marios Dikaiakos
Dept. of Computer Science
University of Cyprus

2. Introduction
Utility of Grid Benchmarking: The experimental evaluation of Grid Infrastructures.
Researcher:
Characterization of factors of application and infrastructure performance;
provide insight to properties of Grid Architectures;
Study the dynamic nature of the Grid.
User:
Evaluate fitness of a collection of resources for running a specific application.
Investigate/troubleshoot application performance (by comparison to benchmark runs)

3. Introduction (cont’d)
Utility of Grid Benchmarking (cont’d)
Developer:
Evaluate fitness of a collection of resources for running a general class of applications, by looking into different programming modelsadigms or which type of middleware to use.
Integrator/Administrator:
Evaluate the employment of specific middleware
Evaluate scheduling and resource allocation algorithms.

4. Benchmarking the Grid: A Hierarchical Framework
Targets:
Individual Resources (cluster nodes, Storage Elements)
Sites (clusters, SMPs)
Grid Constellations (i.e. multiple sites / VOs)
Middleware (e.g. mpich-g2)
Benchmarks:
Micro-benchmarks –isolate basic performance characteristics
Micro-kernel Benchmarks –synthetic codes
Application Benchmarks –derived from real applications deployed on grids

5. A Simple Model VO
SITE
Storage
NODE …
Site

6. GridBench Design: Software Architecture
Components:
RSL/JDL Compiler (XML benchmark descriptions to job description languages)
Orchestrator (benchmark execution, collection of results)
Benchmark component (the benchmark executable)
Monitoring component (collects monitoring information)
Archive Database (benchmark result archive)
Information provider (results to information services)
Benchmark Definition GUI
Benchmark Browser GUI

7. GridBench Design: Software Architecture

8. GridBench Design: Archival & Publication of Results
Benchmark results are archived in a native-XML database (Apache Xindice)
The Benchmark results are stored together with their GBDL definition.
Results are meaningless without the specific parameters etc. that were used
(Reference to) Monitoring data also archived.
Comprehension/Analysis of results is (hopefully considerably) enhanced when combined with infrastructure monitoring during the run.
Benchmark results related to a Site are published to the Site’s MDS for easy access by users and schedulers

9. Micro-benchmarks & Micro-kernels
Micro-benchmarks at the Worker-node level
EPWhetstone: embarrassingly parallel adaptation of the serial whetstone benchmark.
BlasBench: evaluate serial performance of the BLAS routines.
Micro-benchmarks at the Site level
Bonnie++: Storage I/O performance
MPPTest: MPI performance measurements
Micro-benchmarks at the VO level
MPPTest: MPI performance measurements (spanning sites)
gb_ftb: File Transfer Benchmark
Micro-kernels at the Site level
High-Performance Linpack
Selected kernels from the NAS Parallel Benchmarks
Micro-kernels at the VO level
Computationally Intensive Grid Benchmarks

10. GridBench Design: Meta-data
GBDL Main Parts:
Components
Metrics
Parameters
Location
Co/pre-requisites
Monitors

11. Metrics representation
Benchmark-level metrics Vs component-level metrics (e.g. CIGB completion time Vs EP component completion time)
Single-value metrics
<metric name="epwhetstone_IPS“ type="value">
<vector unit="MIPS">53</vector>
</metric>
Multi-value metrics
<metric name="xfer-rate" type="list">
<vector unit="bps">
, , , ,...</vector>
<vector unit="second" toffset=" ">
0,10,20,30,...</vector>

12. Monitoring GBDL can contain monitoring definitions.
<component name="data-transfer" ID="xfer01">...</component>
<monitor type="RGMA" source="ccwp71.in2p3.fr:3306"
query="select * from NetworkTCPThroughput
where NMIdSource='adc0003.cern.ch'
and NMIdDestination='ccwp7.in2p3.fr'
<parameter name="begin">comp-begin="xfer01"<ameter>
<parameter name="end">comp-end="xfer01"<ameter>
</monitor>
Implies that a monitoring client must be implemented/adapted (potentially several monitoring clients can be employed)

13. Compilation to RSL/JDL
GBDL
XML-based GBDL to Job Description
Support for simple jobs can be through the use of simple templates. (executable, parameters and locations are transformed to simple RSL/JDL ‘scripts’)
Most benchmarks need special command-line parameter formatting, or parameter files. -> ParameterHandler for each benchmark that needs it.
Param handler
compiler
RSL
JDL …

14. Compilation to RSL (example)
GBDL
<benchmark date= name="nbody">
<component id="A"
name="nbody“ type="mpi">
<location type="single">
<resource cpucount="2”
name="ce1.grid.ucy"/>
name="ce2.grid.ucy"/>
</location>
<parameter name="executable"
type="attribute">
/bin/nbody.exec </parameter> <parameter name="nparticles"
type="value">
</parameter> </component>
</benchmark>
RSL
+(&(resourceManagerContact=
“ce1.grid.ucy")
(label="subjob 0")
(environment=
(GLOBUS_DUROC_SUBJOB_INDEX 0))
(count=2)
(arguments="-n 1000")
(executable="/bin/nbody.exec" ))
(&(resourceManagerContact=
"ce2.grid.ucy ")
(label="subjob 1")
(GLOBUS_DUROC_SUBJOB_INDEX 1))
(arguments="-n 1000")

15. Experiments User Interface Compiler GridBench UI Monitor CE
Benchmark Component Executable
RSL
GBDL
GRAM
Orchestrator
std out
GRAM
std out CE
Benchmark Component Executable
Archiver SE
Xindice

16. Results

17. WIP and Future Work Complete GBDL specification
Revise Metric and Monitoring representations
Implementation of more benchmarks.