Installation and evaluation of the Globus toolkit WP 1 INFN-GRID Workload management WP 1 DATAGRID WP 2.1 INFN-GRID Massimo Sgaravatto INFN Padova

Globus To make productions and analyses on a Grid some basic services (security, information service, resource management, …) must be implemented as first step Globus identified as possible Grid framework providing these services … but it has been developed mainly for traditional computing, different from computing in HEP High throughput vs. High performance PC farms vs Supercomputers Distributed data intensive computing Need to assess what can be used for HEP environment (PC clusters, distributed PB of data, …) WP1 Installation and Evaluation of the Globus Toolkit of the INFN- GRID Project Goal: evaluation of the Globus toolkit Which services can be useful ? What is necessary to integrate/modify ? What is missing ? Duration: 6 months Results of this first evaluation used to plan future activities

Globus: Tasks Globus deployment Reduce complexity and manpower for Globus installation and maintenance Security To access GRID resources mechanisms for user authentication needed Evaluation of GSI service (based on certificates) Information Service To discover the GRID resources (CPU, storage, network, …) mechanisms to publish them must be defined Analysis of GIS service to publish information using a uniform and standard interface Resource Management Necessary a uniform interface to submit jobs on GRID resources Uniform standard interface to different resource management systems Uniform standard language for task management Assessment of Globus GRAM service for resource allocation and process management

Globus: Tasks Data Access and Migration High performance and reliable tools needed to manage data (data transfers, wide area replica between regional centers, …) Assessment of Globus tools for data management (GASS, Globus ftp) Fault Monitoring Faults in a GRID environment must be promptly detected and recovery mechanisms must be implemented Evaluation of HBM service for fault detection Execution Environment Management Code migration (moving the application where the job will actually be executed) as a possible implementation strategy Evaluation of GEM service to support code migration

Globus: Deliverables & Milestones Deliverables Tools, documentation and operational procedures for Globus deployment (6 Months) Final report on suitability of the Globus toolkit as basic Grid infrastructure (6 Months) Milestones Basic deployment Grid infrastructure for the INFN GRID (6 months)

Globus: Personnel Contribution (FTE) Bari M. DAmato (0.2), XXX (0.5) Bologna F. Semeria (0.2) Catania R. Barbera(0.1), E. Cangiano (0.1), C. Rocca (0.1) CNAF L. Fonti (0.2), F. Giacomini (0.1), A. Italiano(0.1), G. Vita Finzi (0.4) Lecce M. Cafaro (0.1), L. Depaolis (0.1) LNL M. Biasotto (0.1) Milano G. Lo Biondo (0.1), F. Prelz (0.1) Padova M. Sgaravatto (0.2) Pisa G. Bagliesi (0.3), A. Controzzi (0.5), F. Donno (0.3), A. Sciaba` (0.3), Z. Xie (0.3) Roma 1 D. Anzellotti (0.1), M. De Rossi (0.1), E. Majorana (0.1), C. Palomba (0.1), D. Rossetti (0.1), A. Spanu (0.1), E. Valente (0.1) Torino C. Anglano (0.1), A. Forte (0.1), L. Gaido (0.1) Total: 5.4 FTE

Globus: First Activities and Results Globus deployment Globus installed on ~ 30 machines in 11 different sites Bari, Bologna, Catania, CNAF, Ferrara, LNL, Milano, Padova, Napoli, Pisa, Torino First tools to quasi-automatically install and deploy Globus Bug fixes and INFN customizations included Released for DataGrid Interest from Globus team on these activities Information Service INFN MDS server (for Globus and installations, that use a centralized model) Definition and on-going implementation of test architecture for Grid Information Service (for Globus installations, that consider a distributed model) Web interface for browsing Preliminary activities on integrating the default schema with other info

Dc=bo, Dc=infn, dc=it,o=grid Bologna GIIS INFN CMS GIIS GIIS Dc=pd,Dc=infn, dc=it,o=grid Exp=cms, o=grid Top Level INFN GIIS Dc=infn,dc=it, o=grid Padova INFN GIS Architecture (test phase)

Globus: First Activities and Results Security Globus architecture for authentication evaluated Reliable, but some improvements needed … Use of INFN Certification Authority to issue Globus certificates: ok Resource Management (activities strictly related with workload management WP) Job submission tests on remote GRID resources considering different underlying resource management systems (Condor, LSF, PBS) Evaluation of Globus Resource Specification Language as uniform language to specify resources Cooperation with Grid Information Service Information on characteristics and status of local resources must be published

Workload Management in the DataGrid project (WP 1) Goal: define and implement a suitable architecture for distributed scheduling and resource management in a GRID environment Large heterogeneous environment Large numbers (thousands) of independent users Many challenging issues : Optimizing the choice of execution location based on the availability of data, computation and network resources Optimal co-allocation and advance reservation of CPU, data, network Uniform interface to possible different local resource management systems Priorities, policies on resource usage Reliability Fault tolerance Scalability … INFN responsibility in DataGrid

DataGrid Workload Mgmt: Effort breakdown (mm) FundedUnfunded INFN DATAMAT1080 CESnet PPARC

Workload Management in the INFN-GRID project (WP 2.1) Integration, adaptation and deployment of middleware developed within the DataGrid project GRID software must enable physicists to run their jobs using all the available GRID resources in a transparent way HEP applications classified in 3 different classes, with incremental level of complexity Workload management system for Monte Carlo productions Goal: throughput maximization Implementation strategy: code migration (moving the application where the processing will be performed) Workload management system for data reconstruction and production analysis Goal: throughput maximization Implementation strategy: code migration + data migration (moving the data where the processing will be performed, and collecting the outputs in a central repository) Workload management system for individual physics analysis Chaotic processing Goal: latency minimization Implementation strategy: code migration + data migration + remote data access (accessing data remotely) for client/server applications (dynamical decision)

Workload Mgmt: Personnel Contribution (FTE) Bologna F. Semeria (0.5) Catania F. Barbanera (0.2), S. Cavalieri (0.4), E. Commis (0.4), L. Lo Bello (0.4), O. Mirabella (0.4), S. Monforte (0.4), V. Sassone (0.2), L. Vita (0.4) CNAF P. Ciancarini (0.5), T. Ferrari (0.2), L. Fonti (0.3), F. Giacomini (1), A. Ghiselli (0.3), C. Vistoli (0.5) Ferrara M. Gambetti (0.2), A. Gianoli (0.2), E. Luppi (0.1) Lecce G. Aloisio (0.2), M. Cafaro (0.4), S. Campeggio (0.2), L. Depaolis (0.1). E. Fasanelli (0.2), XXX (1) Milano F. Prelz (0.3) Padova S. Orlando (0.3), M. Sgaravatto (0.4) Roma1 G. Mirabelli (0.3) Torino C. Anglano (0.3), S. Donatelli (0.3), L. Gaido (0.3), A. Welbrouck (0.3), YYY (0.9), ZZZ (1) Total: 13.1 FTE INFN participation to Workload Mgmt in DataGrid included

Workload Mgmt: First Activities and Results CMS-HLT use case analyzed in terms of GRID requirements and GRID tools availability Discussions with Globus team and Condor team Good and productive collaborations already in place Definition of a possible high throughput workload management system architecture Use of Globus and Condor mechanisms But major developments needed On going activities in putting together the various building blocks Deep tests on GRAM functionalities already done and Globus-Condor interface already in place

High throughput workload management system architecture (simplified design) Globus GRAM CONDOR Globus GRAM LSF Globus GRAM PBS globusrun Site1 Site2Site3 condor_submit (Globus Universe) Condor-G Master Grid Information Service (GIS) Submit jobs (using Class-Ads) Resource Discovery Information on characteristics and status of local resources Local Resource Management Systems Globus GRAM as uniform interface to different local resource management systems Condor-G able to provide reliability Use of Condor tools for job monitoring, logging, … Master chooses in which Globus resources the jobs must be submitted Farms

Other info INFN-GRID WP 1 (Globus) web site: INFN-GRID/Globus