1. First South Africa Grid Training 16-26 June 2008, Catania (Italy) OVERVIEW of the gLite COMPONENTS Marcello Iacono Manno marcello.iacono@ct.infn.it FIRST SOUTH AFRICA GRID TRAINING IN CATANIA Catania, June 17 th, 2008

2. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 2 Overview gLite Overview Site-level gLite elements – Computing Element (CE) – Storage Element (SE) – Site Berkeley Database Information Index (BDII) Virtual Organization-level gLite elements – Virtual Organization Management Service (VOMS) – Workload Management System (WMS) – LCG File Catalogue (LFC) – Top BDII Job state diagram

3. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 3 Job submission gLite services Computing Element Storage Element Site X Information System submit query retrieve Workload Management Logging & Bookkeeping User Interface publish state File and Replica Catalogs Authorization Service query update credential publish state discover services WNs

4. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 4 Foundation Grid Middleware Security model and infrastructure Computing (CE) & Storage Elements (SE) Accounting Information providers and monitoring EGEE Middleware: gLite Applications Higher-Level Grid Services Workload Management Replica Management Visualization Workflows Grid economies etc. Specific solutions allowed Solution exportability More adaptable compared to Foundation middleware Integration into applications Independent from applications Adhesion/evaluation to/of new standards Robustness emphasized Distributed as site support software Service Oriented Approach: “Light” services Allow different compatible solutions Easy and fast deploying Integration with existing services by R.Jones, 24/10/05

5. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 5 gLite Services Decomposition API Access Job Mgmt. Services Computing Element Workload Management Metadata Catalog Data Services Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Service Discovering AccountingAuditing Job Provenance Package Manager CLI Network Monitoring

6. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 6 Grid Access API Access Job Mgmt. Services Computing Element Workload Management Metadata Catalog Data Services Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Service Discovering AccountingAuditing Job Provenance Package Manager CLI Network Monitoring

7. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 7 User Interface (UI) EGEE middleware is accessed by a User Interface (UI) Supplies the CLI tools to interact with gLite services Basic operations on the Grid: – Create a proxy for authentication/authorization – Query the Information System about the resource status – Copy, duplicate, erase file to/from the Grid Data Management Sys. – List resources available for job execution – Submit a job – Cancel a job – Show job status – Retrieve job output – Retrieve information about job execution (L&B) Supplies APIs to be integrated into Grid applciations

8. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 8 Computing Element (CE) API Access Job Mgmt. Services Computing Element Workload Management Metadata Catalog Data Services Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Service Discovering AccountingAuditing Job Provenance Package Manager CLI Network Monitoring

9. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 9 Resource Access  LCG-CE (GT2 GRAM) – Soon becoming obsolete  gLite-CE (Condor-C+GSI) – Released (GT2 version) but not yet stable  CREAM (WS-I) – Prototypical under certification – EGEE contribution to the Open Grid Forum – Basic Execution Services for a standard WS-I based interface to CE  BLAH is the interface resource to the local manager (via plug-ins) – CREAM & gLite-CE – Information pass-through: sends parameters to LRMS for scheduling enhancements Condor-G Globus client gLite WMSUser CREAM CEMon ICE CREAM client EGEE authZ, InfoSys, Accounting In production Existing prototype gLite component non-gLite component Batch System LCG-CE (GT2) gLiteCE BLAH User / Resource UI Site GT2 GRAM jobmanager

10. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 10 ICE-CREAM CE evolution CREAM: Web Service Computing Element – Cream WSDL allows a personalized user interface – The C++ CLI interface allows direct submission Light software Fast notification about job state evolution – via CEMon Enhanced security – No more “fork-scheduler” Ready for per “bulk” job – Allows data input sharing ICE: an interface to CREAM – Currently under integration

11. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 11 Storage Element (SE) API Access Job Mgmt. Services Computing Element Workload Management Metadata Catalog Data Services Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Service Discovering AccountingAuditing Job Provenance Package Manager CLI Network Monitoring

12. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 12 A Storage Element (SE) is a pool of services for data management Data are stored on disk pool servers or Mass Storage Systems Storage resource management tasks: – Transparent access to files (migration to/from disk pool) – File management – Storage space reservation – File status notification – File control The Storage Resource Manager (SRM) also connects the local Data Management to remote Grid sites Storage Element (SE) Management

13. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 13 gLite 3.0 data access protocols: – File transfer protocol:GSIFTP (GridFTP) – File I/O (Remote File access):gsidcap insecure RFIO secured RFIO (gsirfio) Classic SE: – GridFTP server – Unsafe RFIO (rfiod) demon – files accessible only from local net – Single disk or disk array supported – No quotas – No SRM gLite SE (1)

14. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 14 gLite SE (2) Mass Storage Systems (Castor, dCache) – System hierarchy: front-end disks and back-end tapes – Server GridFTP – Unsafe RFIO (Castor), safe gsidcap (dCache) – Interface compliant Disk pool managers (dCache, DPM, StoRM) – Centralized storage system management – Virtual file system covering several different physical systems – On-line resource add-on – GridFTP server – Secure remote access protocols (gsidcap for dCache, gsirfio for DPM) – SRM interface

15. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 15 Information System API Access Job Mgmt. Services Computing Element Workload Management Metadata Catalog Data Services Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Service Discovering AccountingAuditing Job Provenance Package Manager CLI Network Monitoring

16. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 16 BDII top-level BDII site-level BDII resource MDS GRIS provider WMS WN UI FTS Queries Site - LDAP based - Standard Information provider (GIP) - GLUE-1.3 schema - Top level BDII tested on >230 sites - About 60 versions inside EGEE 2 minutes Grid Resource Information System GRIS and BDIIs Berkeley Database Information Index

17. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 17 R-GMA monitor R-GMA users see it as a relational database Implements Grid Monitoring Architecture (GMA) from the Open Grid Forum (OGF) Several different APIs (WebBrowsers, Java, C/C++, Python) Based on a Producer and Consumer Services on peer-to-peer base (MON box) and a centralized Registry and Schema Producer Service Registry Service Consumer Service API Schema Service Consumer application Producer application Publish Tuples Send Query Receive Tuples Register Locate Query Tuples SQL “CREATE TABLE” SQL “INSERT” SQL “SELECT”

18. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 18 GridICE Monitor - MON box also hosts the GridICE extended GRIS (on port 2136) - Usually deployed together a classic SE

19. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 19 API Access Job Mgmt. Services Computing Element Workload Management Metadata Catalog Data Services Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Service Discovering AccountingAuditing Job Provenance Package Manager CLI Network Monitoring Security Services

20. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 20 VO Membership Service: VOMS Original personal certificates are not enough to describe user privileges over the Grid User belonging to the same VO may be divided into groups and/or play special roles VOMS allows to add attributes to a certificate proxy: – For mutual authentication among users and services – VOMS produces an Attribute Certificate (AC) signed with the user’s Digital Signature – User certificates includes information about privileges – Attributes grant special privileges according to VO policy VOMS client Query Authentication Request Author. DB C=IT/O=INFN /L=Padova /CN=Marco Verlato /CN=proxy VOMS AC === VO cms extension information === VO: cms subject: /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Marco Verlato/Email=Marco.Verlato@pd.infn.it issuer: /C=CH/O=CERN/OU=GRID/CN=host/voms.cern.ch attribute: /cms/prod/ROLE=manager/Capability=NULL attribute: /cms/Role=NULL/Capability=NULL timeleft : 11:59:45 OK

21. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 21 Job request (submission, deletion) codified by Job Description Language (JDL) Keeps track of the resource requests until availability Finds a CE matching job request, the status of the Grid and the VO policies Collects information about the status of the Grid by either push or poll Job submission and control WMS as RB/1

22. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 22 WMS as RB/2 WMS helps the user accessing Grid resources – Resource Brokering, job I/O management,... LCG-RB: GT2 + Condor-G – To be replaced by WMS gLite gLite WMS: Web service (WMProxy) + Condor-G – More complex jobs (DAG) Bulk submission with shared data input Scattered input sandbox supported – Re-submission supported – Job File Perusal: on-line file inspection – Information collection from several sources (CEMon, BDII, R- GMA and DLI interface) – Parallel jobs supported (MPI)

23. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 23 Workflows Direct Acyclic Graph (DAG) is a composed job where the output of the first job is the input for the “intermediate layer” jobs and finally a job collects the outputs A Collection is a composed job formed by independent jobs A parametric job is a collection of jobs identical but for one or more JDL attributes varying according to a _PARAM_ (usually spanning over a numeric interval or a set of string values) Composed (DAG, Collection or parametric) jobs allow the user to submit many jobs by issuing a single command Submission time reduction WMProxy supported A single authentication/authorization for many jobs File sharing among different jobs Group JobId available added to the single JobId nodeE nodeC nodeA nodeD nodeB

24. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 24 Globally unique identifier Site URL Transport URL: includes protocol user need only see these LCG File Catalog

25. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 25 Job Submission Workflow JDL Logging & Bookkeeping WMS Job Submission Service Storage Element ComputingElement Information Service Job Status LCG File Catalog Job Submit Event Input Sandbox JDL Job Input Sandbox Output Sandbox User Interface Author. Service voms-proxy-init GSI data acc/transf glite-wms-job-submit -a myjob.jdl Myjob.jdl Executable = "$(CMS)/exe/sum.exe"; InputData = "lfn:/testbed0-00019"; DataCatalogType="DLI”; DataAccessProtocol = "gsiftp"; InputSandbox = {"/home/user/WP1testC","/home/file1” }; OutputSandbox = {“sim.err”, “test.out”, “sim.log"}; Requirements = other.GlueCEArchitecture == "INTEL"; Rank = other.FreeCPUs;

26. 16-26 June 2008, Catania (Italy) First South Africa Grid Training in Catania 26 Questions…