E-science grid facility for Europe and Latin America Porting a program to run on the Grid Marcello Iacono Manno Consorzio COMETA

Slides:



Advertisements
Similar presentations
Setting up of condor scheduler on computing cluster Raman Sehgal NPD-BARC.
Advertisements

Chapter 9: Moving to Design
FESR Consorzio COMETA - Progetto PI2S2 Using MPI to run parallel jobs on the Grid Marcello Iacono Manno Consorzio COMETA
Copyright Arshi Khan1 System Programming Instructor Arshi Khan.
MCTS Guide to Microsoft Windows Server 2008 Network Infrastructure Configuration Chapter 2 Installing Windows Server 2008.
Lecture The Client/Server Database Environment
The Client/Server Database Environment
Client/Server Architectures
Enabling Grids for E-sciencE Medical image processing web portal : Requirements analysis. An almost end user point of view … H. Benoit-Cattin,
Sergey Belov, Tatiana Goloskokova, Vladimir Korenkov, Nikolay Kutovskiy, Danila Oleynik, Artem Petrosyan, Roman Semenov, Alexander Uzhinskiy LIT JINR The.
Porting applications to EU-IndiaGrid: EGEE Marco Verlato EU-IndiaGrid Workshop April 2007 Bangalore, India.
Bright Cluster Manager Advanced cluster management made easy Dr Matthijs van Leeuwen CEO Bright Computing Mark Corcoran Director of Sales Bright Computing.
E-science grid facility for Europe and Latin America Installation and configuration of a top BDII Gianni M. Ricciardi – Consorzio COMETA.
Nicholas LoulloudesMarch 3 rd, 2009 g-Eclipse Testing and Benchmarking Grid Infrastructures using the g-Eclipse Framework Nicholas Loulloudes On behalf.
Grid Initiatives for e-Science virtual communities in Europe and Latin America DIRAC TEAM CPPM – CNRS DIRAC Grid Middleware.
03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America GENIUS server installation and configuration.
E-science grid facility for Europe and Latin America Watchdog: A job monitoring solution inside the EELA-2 Infrastructure Riccardo Bruno,
BLU-ICE and the Distributed Control System Constraints for Software Development Strategies Timothy M. McPhillips Stanford Synchrotron Radiation Laboratory.
Enabling Grids for E-sciencE ENEA and the EGEE project gLite and interoperability Andrea Santoro, Carlo Sciò Enea Frascati, 22 November.
Carrying Your Environment With You or Virtual Machine Migration Abstraction for Research Computing.
DataNet – Flexible Metadata Overlay over File Resources Daniel Harężlak 1, Marek Kasztelnik 1, Maciej Pawlik 1, Bartosz Wilk 1, Marian Bubak 1,2 1 ACC.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Code Applications Tamas Kiss Centre for Parallel.
Tool Integration with Data and Computation Grid GWE - “Grid Wizard Enterprise”
E-science grid facility for Europe and Latin America E2GRIS1 Gustavo Miranda Teixeira Ricardo Silva Campos Laboratório de Fisiologia Computacional.
E-science grid facility for Europe and Latin America Setting up a YUM Repository Gianni M. Ricciardi – Consorzio COMETA Joint EELA-2/EGEE-III.
9 Systems Analysis and Design in a Changing World, Fourth Edition.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
E-science grid facility for Europe and Latin America GridwWin: porting gLite to run under Windows Fabio Scibilia – Consorzio COMETA 30/06/2008.
FESR Consorzio COMETA - Progetto PI2S2 The COMETA consortium and its activities for Grid adoption by Industry in the context of.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Applications.
Interactive Workflows Branislav Šimo, Ondrej Habala, Ladislav Hluchý Institute of Informatics, Slovak Academy of Sciences.
1 Grid2Win: porting of gLite middleware to Windows Dario Russo INFN Catania
CASE (Computer-Aided Software Engineering) Tools Software that is used to support software process activities. Provides software process support by:- –
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE Site Architecture Resource Center Deployment Considerations MIMOS EGEE Tutorial.
Introduction Why are virtual machines interesting?
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Grid2Win: Porting of gLite middleware to.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Grid2Win : gLite for Microsoft Windows Roberto.
Tool Integration with Data and Computation Grid “Grid Wizard 2”
FESR Consorzio COMETA - Progetto PI2S2 Porting a program to run on the Grid Marcello Iacono Manno Consorzio COMETA
INFSO-RI Enabling Grids for E-sciencE Using of GANGA interface for Athena applications A. Zalite / PNPI.
Grid Execution Management for Legacy Code Architecture Exposing legacy applications as Grid services: the GEMLCA approach Centre.
Tutorial on Science Gateways, Roma, Catania Science Gateway Framework Motivations, architecture, features Riccardo Rotondo.
Breaking the frontiers of the Grid R. Graciani EGI TF 2012.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Overview of gLite, the EGEE middleware Mike Mineter Training Outreach Education National.
Consorzio COMETA - Progetto PI2S2 UNIONE EUROPEA Grid2Win : gLite for Microsoft Windows Elisa Ingrà - INFN.
Instituto de Biocomputación y Física de Sistemas Complejos Cloud resources and BIFI activities in JRA2 Reunión JRU Española.
G. Russo, D. Del Prete, S. Pardi Kick Off Meeting - Isola d'Elba, 2011 May 29th–June 01th A proposal for distributed computing monitoring for SuperB G.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
CNAF - 24 September 2004 EGEE SA-1 SPACI Activity Italo Epicoco.
OSCAR Symposium – Quebec City, Canada – June 2008 Proposal for Modifications to the OSCAR Architecture to Address Challenges in Distributed System Management.
Towards a High Performance Extensible Grid Architecture Klaus Krauter Muthucumaru Maheswaran {krauter,
Gri2Win: Porting gLite to run under Windows XP Platform
Grid2Win: Porting of gLite middleware to Windows platform
Workload Management Workpackage
Grid2Win Porting of gLite middleware to Windows XP platform
Lecture 1-Part 2: Operating-System Structures
Current Generation Hypervisor Type 1 Type 2.
Dag Toppe Larsen UiB/CERN CERN,
Dag Toppe Larsen UiB/CERN CERN,
The Client/Server Database Environment
Grid2Win: Porting of gLite middleware to Windows XP platform
Grid2Win: Porting of gLite middleware to Windows XP platform
Gri2Win: Porting gLite to run under Windows XP Platform
Grid2Win: Porting of gLite middleware to Windows XP platform
Module 01 ETICS Overview ETICS Online Tutorials
Lecture 1-Part 2: Operating-System Structures
Laura Bright David Maier Portland State University
How To Integrate an Application on Grid
gLite The EGEE Middleware Distribution
Presentation transcript:

E-science grid facility for Europe and Latin America Porting a program to run on the Grid Marcello Iacono Manno Consorzio COMETA JOINT EELA2-EGEE3 TUTORIAL4TRAINERS Catania, 2008, July 4 th

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Overview Relevant Issues Applications classification Data Management Computing Schema Overview

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – The four pillars of Grid Computing

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Why running a program on the Grid? – Faster (hundreds of processors) – Greater (TBs of storage capacity) – Cheaper (currently is free, but NOT forever) Relevant Issues – Platform: UNIX (LINUX) / Windows – Software: COMPILER / LIBRARY AVAILABILITY – Interactivity: COMMAND LINE / GRAPHICAL USER INTERFACE – Legal Issues: LICENCE / DATA SECURITY Basics

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – The Grid “native” environment is open source … but heterogeneity is a Grid feature … so: – legacy middleware is based on Scientific Linux (SLC) (similar to Red Hat 6 / 7) with a 2.4.x kernel running on a i686 (32 bit) architecture – migration to SLC 4 with a 2.6 Linux kernel on a x86_64 architecture (64 bit) – migration is complete for applications:  COMETA (Consortium) is a “production” infrastructure supporting the gLite 3.0 (edg flavor) middleware version  Worker Nodes are full 64-bit (4 cores) processors  PI2S2 (Project) User Interface is 64-bit SLC4.4 (build machine)  Migration to glite-3.1 (glite-wms commands) is under way … about Windows applications: – a strong demand from the industry / business world Update Operations

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Windows new users can use the Grid Presently on gLite –Grid users interact via the gLite middleware from Linux-based User Interfaces via CLI –(almost) all gLite resources are Linux-based This implies –Grid users need to be trained –Only Linux-based applications can be deployed onto the Grid Porting of gLite to Windows –User Interface –Computing Element (farm)

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – On a Cometa WN the PGI (Portland Group Inc.) (64 bit target), Intel 9.0, gcc4 – /opt/share/pgi Cc, CC, f77, f90 – Java Runtime Environment Libraries – Static compilation:  The executable “contains” all the libraries Large file (slows down data transfer) Non need for “external” calls (faster execution) – Dynamic Libraries  Call to external libraries Small executable file (easy to transfer) Less robust solution Compiler / Library Availability

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – – Two ways to install the software:  Static “Public” installation as a stable extension of the m/ware Advantage: speeds up job execution oLess work to do at run time Disadvantages: ocompatibility problems with middleware and/or other applications omore complex modifying and updating the application orequires Software Manager (SWM) role privileges Usage: ohuge and/or stable SW packages oonly MPI binary executable available (difficultly modifiable) Static Installation

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, –  Dynamic “Private” installation during job pre – processing Advantages: omore robust and flexible jobs oeasier modifying and updating oNo SWM privileges required Disadvantages: oslows down job execution (more work to do during job execution) Usage: olittle and/or frequently modified SW packages oMPI launching script (easily modifiable) Dynamic Installation

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Interactivity – The “standard” use of Grid is the “batch” mode – Interactive Jobs are also available but … – Grid integration of “foreign” GUIs requires modifications to the application code (deprecated for both difficulties and “danger”) – Data Inspection is required by long-running jobs (>1 day)  by a watchdog script (user application unchanged)  check - pointable jobs (available on Cometa but not yet tested, require a library called from “inside” the application)  customized solutions (see Computing Schema) Graphical User Interface – The Genius portal is the “natural” GUI for the gLite m/ware  many functions already implemented (i.e. authentication)  development is faster due to standardization Interactivity & GUI

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – License Issues – In the future will act as a service provider selling calculus and storage resources on demand – The accounting system is under construction – Licensed software can be already installed at site level – A Grid License Server is being built Data Security – Data Catalog, replicas enhance data system robustness – Security is based on X509 certificates – Virtual Organization membership, groups, roles are recognized and Access Control List (ACL) are supported – Cryptography is also available (GFAL library) License Issues & Data Security

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Classification: Simulation Characteristics –Jobs are CPU-intensive –Large number of independent jobs –Run by a few (expert) users –Small input; large output Needs –Batch-system services –Minimal data management for result storage Examples: LHC Monte Carlo simulation, Fusion

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Classification: Bulk Processing Characteristics –Widely-distributed input data –Significant amount of input and output data Needs –Job management tools (workload management) –Meta-data services –More sophisticated data management Examples: HEP processing of raw data, analysis, Earth observation data Processing.

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Classification: Parallel Processing Characteristics –Critical information exchange among processors during job execution –Significant amount of input and output data Needs –Job management tools (workload management) –Special libraries for communication net layer (Low-latency InfiniBand) –Paradigm matching (shared memory vs. distributed computing) Examples: Finite Elements Mesh (Partial Derivative Equation numerical solvers)

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Classification: Responsive Apps (I) Characteristics –Small amounts of input and output data –Not CPU-intensive –Short response time (few minutes) Needs –Configuration which allows “immediate” execution (QoS) –Services must treat jobs with minimum latency Examples: Prototyping new applications, Monitoring grid operations

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Characteristics –Rapid response: a human waiting for the result! –Many small but CPU-intensive tasks –User is not aware of “grid”! Needs –Interfacing (data & computing) with non-grid application or portal –User and rights management between front-end and grid Examples: Appls that use Grid as a backend infrastructure (gMOD, gLibrary, Hadrontherapy, GATE, Interactive Analysis of Medical images, Volcano Sonification) Classification: Responsive Apps (II)

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Workflow Characteristics –Use of grid and non-grid services –Complex set of algorithms for data analysis –Complex dependencies between individual tasks Needs –Tools for managing the workflow itself –Standard interfaces for services (I.e. web-services) Examples: Flood prediction

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Data Management Data distribution must be carefully planned: –When huge amounts of data are involved –When a single file is large (1 GB) –When data transfer impacts on computing performance –When security issues are relevant (data integrity) Metadata –May help in data management (updating) –Are useful to add more information to raw data

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Computing Schema –The way the computation is actually performed on the Grid –May be critical for Grid effectiveness –Requires cooperation between Grid and application experts Guidelines –Leave the application unchanged as far as possible –Adapt the Grid (especially extending the m/ware by customization scripts) –Clearly separate application form Grid “domains” –If a massive exploitation is needed, reach it step by step

Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Questions…