1. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks The gLite middleware Middleware comparison questionnaire Gabriel Zaquine EGEE Industry Task Force Coordinator CS SI gabriel.zaquine@cern.ch EchoGRID, Beijing, 29-31 October 2007

2. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 2 Outline EGEE / gLite overview Middleware comparison questionnaire

3. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 3 240 sites 45 countries 41,000 CPUs 5 PetaBytes >10,000 users >150 VOs >100,000 jobs/day Archeology Astronomy Astrophysics Civil Protection Comp. Chemistry Earth Sciences Finance Fusion Geophysics High Energy Physics Life Sciences Multimedia Material Sciences … Flagship grid infrastructure project co-funded by the European Commission Now in 2 nd phase with 91 partners in 32 countries EGEE Production Service

4. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 4 EGEE working with collaborating infrastructure projects

5. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 5 gLite Middleware Distribution Combines components from different providers –Condor and Globus (via VDT) –LCG –EDG/EGEE –Others After prototyping phases in 2004 and 2005 convergence with LCG-2 distribution reached in May 2006 –gLite 3.0 Focus on providing a deployable MW distribution for EGEE production service LCG-2 prototyping product 2004 2005 product gLite 2006 gLite 3.0

6. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 6 Middleware structure Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory Foundation Grid Middleware will be deployed on the EGEE infrastructure –Must be complete and robust –Should allow interoperation with other major grid infrastructures –Should not assume the use of Higher-Level Grid Services Foundation Grid Middleware Security model and infrastructure Computing (CE) and Storage Elements (SE) Accounting Information and Monitoring Higher-Level Grid Services Workload Management Data Management....... Applications Overview paper http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-001.pdf

7. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 7 Workload ManagementData Management Security Information & Monitoring Access gLite Services API Computing Element Workload Management Metadata Catalog Storage Element Data Movement File & Replica Catalog Authorization Authentication Information & Monitoring Application Monitoring Auditing Job Provenance Package Manager CLI AccountingSite Proxy Overview paper http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-001.pdf

8. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 8 Middleware comparison questionnaire –Tools for Application developers –Basic programming models supported by the middleware –Data Management –Resource management & SLA –Standards and interoperability –Orthogonal questions –Application usage –Distribution model –Future plans

9. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 9 Tools for Application developers Integrated development environments (IDE) - Problem solving environments (PSE) –Several tools have been developed by user communities – These tools are not part of gLite distribution, but are part of the RESPECT program (Recommended External Software for EGEE CommuniTies) –The RESPECT program aims to publicize grid software and services that work well in concert with the EGEE gLite software. Selects software are recommend based on feedback from the EGEE user community.RESPECT  Job Management supporting workflow semantics (GridWay, GANGA/DIANE)GridWay GANGADIANE  Workflow Tools (P-GRADE, MOTEUR, +Taverna)P-GRADEMOTEUR  Integrated environments/projects (K-Wf Grid, A-Ware, gCube)*K-Wf GridA-WaregCube  … more infomore info

10. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 10 Basic programming models supported by the middleware GUIXLM format Proced ural API Object API WorkflowYes SkeletonYes (GANGA*) Components Master worker slave Yes (DIANE*) Branch & Bound optimisation SPMB : MPI, object SPMDYes Active Objects (actors) –MPI on multi-machines: security problems & issues –GUI for Active Objects: just for monitor * External tools (See RESPECT PROGAM)

11. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 11 Data Management File transfer: Yes –In addition to GridFTP, FTS provides a reliable, scalable and customizable file transfer  Multi-VO service, used to balance usage of site resources according to the SLAs agreed between a site and the VOs it supports  Manages transfers through channels File transfers handled as jobs Prioritization Retries in case of failures Automatic discovery of services Virtual file system : Yes –To the user of the EGEE data services the abstraction that is being presented is that of a global file system. –The catalog services keep track of the data location as well as relevant metadata –The Storage Element (SE) provides the virtualization of a storage resource (which can reach from simple disk servers to complex hierarchical tape storage systems) –Data movement services allow for efficient managed data transfers between SEs. –Encrypted Data Storage file /grid /vo /data

12. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 12 Resource management & SLA (1/3) Virtualisation of multi-core, multi-processors, servers, clusters: Yes Meta scheduler, Broker: Yes Scheduler (batch system): Yes –Support for commonly used submission systems:  PBS/TORQUE-MAUI, LSF and Condor –Support for commonly used submission systems:  gLite WMS, Condor-G, pre-WS GRAM (In future BES) Peer to peer support for auto-organization Hardware resources handled by the middleware (computers, networks, storage devices, scientific instruments) : Yes (Core middleware services)

13. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 13 Security (2/3) AIC: Authentification, Integrity, confidentiality : Yes –Authentication based on X.509 PKI infrastructure –Proxies can be delegated to a service such that it can act on the user’s behalf  Include additional attributes (like VO information via the VO Membership Service VOMS)  Be stored in an external proxy store (myProxy)  Be renewed (in case they are about to expire) –Support Shibboleth mechanism: Long lived certificates may be replaced by short lived certificates provided by a Shibboleth identity Provider Role management : Yes –VOMS is now a de-facto standard  Attribute Certificates provide users with additional capabilities defined by the VO. Users belong to VO’s, to groups inside a VO and may have special roles.  Basis for the authorization process

14. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 14 VO management (3/3) Full support for construction & administration of VO (we need a definition of VO – static or dynamic ?) –Static VO management –Static VO management means semi-automatic facilities to manage VO ( e.g Creation of the VO, adding a new user within the VO, ….)

15. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 15 Orthogonal questions (1/2) Globus based or not: Globus (via VDT) Approach to SOA: Yes Main languages API: (C++, Java, C#, Fortran...): C++, Java Portability across platforms: –gLite 3.0 is working on Scientific Linux 3 – x86-32 –The new components part of the gLite 3.1 stack will initially be working on Scientific Linux 4 – x86-32 –Porting gLite to Windows environment is foreseen by partners Configuration tool –Now using YAIM 4.0 –Supports component based configuration –Enables software providers to maintain configuration independently

16. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 16 Orthogonal questions (1/2) Interoperability –with other Grid middleware: Globus, Condor –with other Grid Infrastructure: OSG, NorduGrid, DEISA (on- going) –Use of standards  Security Certificates for AuthN and VOMS for AuthZ; SAML assertions in future  Information systems GLUE schema (1.3 now 2.0 in future) and adapters to interoperate  Data Management SRM 2.2 interface for data access and GridFTP for file transfers  Job Management BES interface in future (CREAM). Legacy pre-WS GRAM deployed in parallel. gLite WMS, Condor-G and BLAH to build gateways. OGF-RUS will be used for accounting

17. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 17 Applications List of successfully executed applications (real applications, success stories) –Disciplines: 10; Sub-disciplines: 36 –See growth and diversification of applications (Reported apps. only  underestimate!) –What kinds (e-science, e-business, e-engineering,…)  Mainly e-science  About 10 Business applications 6/20062/2007 Astronomy & Astrophysics28 Computational Chemistry627 Earth Science16 Fusion23 High-Energy Physics911 Life Sciences2339 Others414 Total62118 Condensed Matter Physics Comp. Fluid Dynamics Computer Science/Tools Civil Protection Finance Mission-critical for the LHC (Large Hadron Collider) - Starting 2008 WISDOM drugs discovery for neglected and emerging diseases

18. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 18 Distribution model Open Source: Apache-2 license Close Source: Commercial support: YES –Supported by EGEE for Business “early adopters” applications

19. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 19 Future plans Complete the release of the gLite 3.1 components –Including full adoption of ETICS, support for SL4, migration to VDT 1.6.0 (GT4) –Support for 64-bit and other platforms will follow according to TCG decisions Address service manageability issues from site managers –common format for log files of services –hooks for local/global monitoring –improved service management interface –standardized error messages Prototype SLA management (EGEE-III)

20. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 20 How to contribute/influence gLite process driven by application and operational requirements –New components added based on their requests and overall importance RESPECT – Program to collect useful tools that work with gLite –See EGEE application portal: http://egeena4.lal.in2p3.fr/index.php http://egeena4.lal.in2p3.fr/index.php

21. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Gabriel Zaquine - EchoGRID - October 2007 21 www.glite.org