E-science grid facility for Europe and Latin America Porting a program to run on the Grid Marcello Iacono Manno Consorzio COMETA JOINT EELA2-EGEE3 TUTORIAL4TRAINERS Catania, 2008, July 4 th

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Overview Relevant Issues Applications classification Data Management Computing Schema Overview

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – The four pillars of Grid Computing

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Why running a program on the Grid? – Faster (hundreds of processors) – Greater (TBs of storage capacity) – Cheaper (currently is free, but NOT forever) Relevant Issues – Platform: UNIX (LINUX) / Windows – Software: COMPILER / LIBRARY AVAILABILITY – Interactivity: COMMAND LINE / GRAPHICAL USER INTERFACE – Legal Issues: LICENCE / DATA SECURITY Basics

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – The Grid “native” environment is open source … but heterogeneity is a Grid feature … so: – legacy middleware is based on Scientific Linux (SLC) (similar to Red Hat 6 / 7) with a 2.4.x kernel running on a i686 (32 bit) architecture – migration to SLC 4 with a 2.6 Linux kernel on a x86_64 architecture (64 bit) – migration is complete for applications:  COMETA (Consortium) is a “production” infrastructure supporting the gLite 3.0 (edg flavor) middleware version  Worker Nodes are full 64-bit (4 cores) processors  PI2S2 (Project) User Interface is 64-bit SLC4.4 (build machine)  Migration to glite-3.1 (glite-wms commands) is under way … about Windows applications: – a strong demand from the industry / business world Update Operations

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Windows new users can use the Grid Presently on gLite –Grid users interact via the gLite middleware from Linux-based User Interfaces via CLI –(almost) all gLite resources are Linux-based This implies –Grid users need to be trained –Only Linux-based applications can be deployed onto the Grid Porting of gLite to Windows –User Interface –Computing Element (farm)

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – On a Cometa WN the PGI (Portland Group Inc.) (64 bit target), Intel 9.0, gcc4 – /opt/share/pgi Cc, CC, f77, f90 – Java Runtime Environment Libraries – Static compilation:  The executable “contains” all the libraries Large file (slows down data transfer) Non need for “external” calls (faster execution) – Dynamic Libraries  Call to external libraries Small executable file (easy to transfer) Less robust solution Compiler / Library Availability

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – – Two ways to install the software:  Static “Public” installation as a stable extension of the m/ware Advantage: speeds up job execution oLess work to do at run time Disadvantages: ocompatibility problems with middleware and/or other applications omore complex modifying and updating the application orequires Software Manager (SWM) role privileges Usage: ohuge and/or stable SW packages oonly MPI binary executable available (difficultly modifiable) Static Installation

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, –  Dynamic “Private” installation during job pre – processing Advantages: omore robust and flexible jobs oeasier modifying and updating oNo SWM privileges required Disadvantages: oslows down job execution (more work to do during job execution) Usage: olittle and/or frequently modified SW packages oMPI launching script (easily modifiable) Dynamic Installation

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Interactivity – The “standard” use of Grid is the “batch” mode – Interactive Jobs are also available but … – Grid integration of “foreign” GUIs requires modifications to the application code (deprecated for both difficulties and “danger”) – Data Inspection is required by long-running jobs (>1 day)  by a watchdog script (user application unchanged)  check - pointable jobs (available on Cometa but not yet tested, require a library called from “inside” the application)  customized solutions (see Computing Schema) Graphical User Interface – The Genius portal is the “natural” GUI for the gLite m/ware  many functions already implemented (i.e. authentication)  development is faster due to standardization Interactivity & GUI

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – License Issues – In the future will act as a service provider selling calculus and storage resources on demand – The accounting system is under construction – Licensed software can be already installed at site level – A Grid License Server is being built Data Security – Data Catalog, replicas enhance data system robustness – Security is based on X509 certificates – Virtual Organization membership, groups, roles are recognized and Access Control List (ACL) are supported – Cryptography is also available (GFAL library) License Issues & Data Security

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Classification: Simulation Characteristics –Jobs are CPU-intensive –Large number of independent jobs –Run by a few (expert) users –Small input; large output Needs –Batch-system services –Minimal data management for result storage Examples: LHC Monte Carlo simulation, Fusion

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Classification: Bulk Processing Characteristics –Widely-distributed input data –Significant amount of input and output data Needs –Job management tools (workload management) –Meta-data services –More sophisticated data management Examples: HEP processing of raw data, analysis, Earth observation data Processing.

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Classification: Parallel Processing Characteristics –Critical information exchange among processors during job execution –Significant amount of input and output data Needs –Job management tools (workload management) –Special libraries for communication net layer (Low-latency InfiniBand) –Paradigm matching (shared memory vs. distributed computing) Examples: Finite Elements Mesh (Partial Derivative Equation numerical solvers)

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Classification: Responsive Apps (I) Characteristics –Small amounts of input and output data –Not CPU-intensive –Short response time (few minutes) Needs –Configuration which allows “immediate” execution (QoS) –Services must treat jobs with minimum latency Examples: Prototyping new applications, Monitoring grid operations

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Characteristics –Rapid response: a human waiting for the result! –Many small but CPU-intensive tasks –User is not aware of “grid”! Needs –Interfacing (data & computing) with non-grid application or portal –User and rights management between front-end and grid Examples: Appls that use Grid as a backend infrastructure (gMOD, gLibrary, Hadrontherapy, GATE, Interactive Analysis of Medical images, Volcano Sonification) Classification: Responsive Apps (II)

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Workflow Characteristics –Use of grid and non-grid services –Complex set of algorithms for data analysis –Complex dependencies between individual tasks Needs –Tools for managing the workflow itself –Standard interfaces for services (I.e. web-services) Examples: Flood prediction

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Data Management Data distribution must be carefully planned: –When huge amounts of data are involved –When a single file is large (1 GB) –When data transfer impacts on computing performance –When security issues are relevant (data integrity) Metadata –May help in data management (updating) –Are useful to add more information to raw data

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Computing Schema –The way the computation is actually performed on the Grid –May be critical for Grid effectiveness –Requires cooperation between Grid and application experts Guidelines –Leave the application unchanged as far as possible –Adapt the Grid (especially extending the m/ware by customization scripts) –Clearly separate application form Grid “domains” –If a massive exploitation is needed, reach it step by step

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Questions…