1. The Catania Grid Engine Mr. Riccardo Rotondo Consortium GARR, Rome, Italy riccardo.rotondo@garr.it.

2. Outline Requirements jSaga The Catania Grid Engine: o Authentication o Job Engine o Data Engine OGF35, Delft (NL), 18 Jun 20122

3. EGI Portal & Traceability Policies (1/2) 3 Portal Classes Portal ClassExecutableParametersInput Simple one- click provided by portal provided by portal Parameter provided by portal chosen from enumerable and limited set chosen from repository vetted by the portal Data processing provided by portal chosen from enumerable and limited set provided by user Job management provided by user provided by user Science Gateways

4. The Portal, the VO the Portal is associated to, and the Portal manager are all individually and collectively responsible and accountable for all interactions with the Grid The Portal must be capable of limiting the job submission rate The Portal must keep audit logs for all interactions with the Grid as defined in the Traceability and Logging Policy (minimum 90 days) The Portal manager and operators must assist in security incident investigations Where relevant, private keys associated with (proxy) certificates must not be transferred across a network, not even in encrypted form 4 EGI Portal & Traceability Policies (2/2)

5. SAGA is made of: SAGA Core Libraries: contain the SAGA base system, the runtime and the API packages (job management, data management, etc.); SAGA Adaptors: provide access to the underlying grid infrastructure (adaptors are available for gLite, ARC, Globus, UNICORE and other middleware); SAGA defines a standard We then need an implementation! A Simple API for Grid Applications (SAGA) Beijing, Asia4, 11.04.2012 5

6. JSAGA JSAGA is a Java implementation of SAGA developed at CCIN2P3; JSAGA: o Enables uniform data and job management across different grid infrastructures/middleware; o Makes extensions easily: adaptor interfaces are designed to minimize coding effort for integrating support of new technologies/middleware; o Is OS independent: most of the provided adaptors are written in full Java and they are tested both on Windows and Linux. Beijing, Asia4, 11.04.2012 6

7. JSAGA Adaptors Beijing, Asia4, 11.04.2012 7 JSAGA supports gLite, Globus, ARC, UNICORE, etc.

8. In order to strong reduce the risks to have the robot certificate compromised, the INFN CA decided to store this new certificate on board of the SafeNet eToken smart cards [6]; The AeToken smart card can support many certificates; A token PIN is prompted every time the user needs to interact with the smart card;e-Token 8

9. Users Client Applications Grid Portals / Science Gateways e-Token Server 9 Host based mutual authentication

10. The eToken server working scenario OGF35, Delft (NL), 18 Jun 201210 eTokenServer MyProxy Server ask for VOMS AC attributes VOMS Server store long proxy (*) SSL encryption get results ask for a service list/create request execute a service get the results back retrieve serials/proxy (*)

11. User Tracking DB OGF35, Delft (NL), 18 Jun 201211 GRID USAGE TRACEABILITY Common NamePortal User Name as stored in LDAP IP + PortIP address and TCP port used by the requester TimestampIdentify the grid operation date/time Grid InteractionGrid Interaction Identification (Job “X” submission, file upload/download). The portal MUST classify all the grid operations allowed. This value will allow to identify both applications used and operation performed. Grid IDStore the actual GRID Interaction ID (Job ID for job submission and some other relevant information for data transfer) Robot CertificateIdentify the Robot Certificate used for the Grid Operation Two Tables: one for active Jobs and File Transfersand one for the finished ones. ID70 Common Namefpistagna IP + TCP Port193.206.208.183:8162 Timestamp2011-07-06 14:16:29 Grid Interaction1 Grid ID[wms://infn-wms-01.ct.pi2s2.it:7443/glite_wms_wmproxy_server]- [https://infn-lb-01.ct.pi2s2.it:9000/7rQ458xozactEEjoXMlxQg] Robot Certificate/C=IT/O=INFN/OU=Robot/L=COMETA/CN=Robot: ViralGrid Science Gateway - Roberto Barbera Virtual Organisationcometa Example of entry in the Users Tracking DB

12. The Grid Engine 12 Grid Engine Users Tracking DB Science GW Interface SAGA/JSAGA API Job Engine Data Engine Users Track & Monit. Science GW 1 Science GW 2 Science GW 3 Grid MWs Liferay Portlets eToken Server DONE By end of JuneDONE

13. Job Engine Architecture OGF35, Delft (NL), 18 Jun 201213 WT Worker Threads for Job Submission WT Worker Threads for Status Checking USER TRACKING DB MONITORING MODULE GRID INFRASTRUCTURE(S) Job Queue WT Job Submission Job Check Status/ Get Output

14. Job Engine Features OGF35, Delft (NL), 18 Jun 201214 FeatureDescriptionStatus Middleware Independent Capacity to submit job to resources running different middleware DONE EasinessCreate code to run applications on the grid in a very short time DONE ScalabilityManage a huge number of parallel job submissions fully exploiting the HW of the machine where the Job Engine is installed DONE PerformanceHave a good response timeDONE Accounting & Auditing Register every grid operation performed by the users DONE Fault ToleranceHide middleware failure to end usersALMOST DONE WorkflowProviding a way to easily create and run workflows IN PROGRESS

15. Job Engine Scalability OGF35, Delft (NL), 18 Jun 201215 Submission time scales linearly with number of jobs >10,000 jobs a hour 15 40,000 jobs submitted in parallel ! Time to submit 10,000 jobs (h) Job submission time (h)

16. Job Engine Interoperability Interoperability is a property referring to the ability of diverse systems and organizations to work together (inter-operate). The term is often used in a technical systems engineering sense, or alternatively in a broad sense, taking into account social, political, and organizational factors that impact system to system performance; According to ISO/IEC 2382-01 (Information Technology Vocabulary, Fundamental Terms), interoperability is "The capability to communicate, execute programs, or transfer data among various functional units in a manner that requires the user to have little or no knowledge of the unique characteristics of those units". OGF35, Delft (NL), 18 Jun 201216

17. gLite-based e-Infrastructures/Projects EUAsiaGrid EUChinaGRID EU-IndiaGrid EUMEDGRID GISELA IGI (Italy) SAGrid (South Africa) 17 Job Engine Interoperability

18. 18 MyJobsMap (1/3)

19. 19 MyJobsMap (2/3)

20. 20 Both sequential and MPI-enabled jobs successfully executed The CHAIN project is preparing a demo of worldwide interoperability among gLite, Unicore, OurGrid, GOS, and GARUDA to be presented at the next EGI Technical Forum MyJobsMap (3/3)

21. Usage Workflow Bogotá, 04.06.2012 21 1. Sign in eTokenServer User Track- ing DB 3. Proxy request 4. Proxy transfer 5. Grid Interactions 5. Tracking 2. Grid Request 6. Getting Results

22. Data Enginge Requirements Grid Storage complexity hidden to end users o Users move files from/to a portal and see it as simple external storage accessible from a web interface and do not care about grid (or any other) technologies behind File management smoothly integrated with all the services provided in the SG Underlining architecture exposes a file-system- like view (i.e., a Virtual File System or VFS) through which users can perform the following actions: o Create, move, delete files/directories with the desired structure o Share files with other users o Set the number of backup copies desired EGICF 2012, Munich22

23. Data On Grid Services: DOGS A file browser shows Grid files in a tree File system exposed by the SG is virtual Easy transfer from/to Grid (by SG) is done in a few clicks Users do not need to care about how and where their files are really located EGICF 2012, Munich23

24. Upload workflow EGICF 2012, Munich24 1. Sign in eTokenServer User Tracking DB DOGS DB 5. File Upload 3. Proxy request 4. Proxy transfer 7. Update DB 6. Upload on Grid 7. Tracking 2. Upload request

25. EGICF 2012, Munich25 Back-end technical details JSAGA API used to transfer data from/to storage elements Hibernate to manage the VFS collecting information on files stored on Grid; any changes/actions in the user view affect the VFS MySQL as underlying RDBMS An additional component has been developed in order to keep track of each transaction in the users tracking DB (to be compliant with the EGI Portal and User Traceability Policies)

26. Front-end technical details A portlet has been created to be deployed in a Liferay-based portal to which access is provided only to federated users with given roles and privileges o http://www.liferay.com http://www.liferay.com The portlet view component includes elFinder, a web-based file manager developed in Javascript using jQuery UI for a dynamic and user friendly interface o http://elrte.org/elfinder http://elrte.org/elfinder EGICF 2012, Munich26

27. The Data Engine in action EGICF 2012, Munich27

28. The Data Engine in action EGICF 2012, Munich28 «Share» to be added soon

29. The Data Engine in action EGICF 2012, Munich29

30. The Data Engine in action EGICF 2012, Munich30

31. Thank you for your kind attention OGF35, Delft (NL), 18 Jun 201231