1. Co-ordination & Harmonisation of Advanced e-INfrastructures CHAIN Worldwide Interoperability Test Roberto Barbera – Univ. of Catania and INFN Diego Scardaci - INFN CHAIN Workshop – Taipei, 27 Feb. 2012 Research Infrastructures – Grant Agreement n. 260011

2. Outline  Science Gateway Architecture  World-wide interoperability test with Science Gateways  Summary and conclusions 2

3. ....... Science Gateway Science Gateway App. 1 App. 2 App. N Embedded Applications Administrator Power User Basic User Users from different organisations having different roles and privileges Access: Science Gateway model Standard-based middleware-independent Grid Engine Standard-based middleware-independent Grid Engine 3

4. AuthN/AuthZ Infrastructure  Portal Engine  Liferay Community Edition 6.0.6  Authentication  Shibboleth-2.4.3-2.1/log4shib-1.0.3-2.3/simpleSAMLphp  Implementations of SAML for AuthN (Security data exchange among different Security Domains)  OpenLDAP  Takes care of user roles and privileges (AuthZ)  Single Sign-On  Users can only do what the SG allows them to do  All transactions are tracked Authorization Authentication Science Gateway 4

5. Standards  The framework for Science Gateways proposed and fostered by CHAIN is fully web-based and adopts official worldwide standards and protocols, through their most common implementations:  Web interface: JSR 168 and JSR 286 standards (also known as "portlet 1.0" and "portlet 2.0" standards)JSR 168JSR 286  Authentication: OASIS Security Assertion Markup Language (SAML) standard and its Shibboleth and SimpleSAMLphp implementations;OASISSecurity Assertion Markup LanguageShibbolethSimpleSAMLphp  Authorisation: Lightweight Direct Access Protocol, and its OpenLDAP implementationOpenLDAP  Digital certificate management: Cryptographic Token Interface Standard (PKCS#11) standard and its Cryptoki implementationCryptographic Token Interface Standard  Application interface: Open Grid Forum (OGF) Simple API for Grid Applications (SAGA) standard and its JSAGA implementationOpen Grid ForumSimple API for Grid ApplicationsJSAGA 5

6. Interoperability (http://en.wikipedia.org/wiki/Interoperability)  Interoperability is a property referring to the ability of diverse systems and organizations to work together (inter-operate). The term is often used in a technical systems engineering sense, or alternatively in a broad sense, taking into account social, political, and organizational factors that impact system to system performance  According to ISO/IEC 2382-01 (Information Technology Vocabulary, Fundamental Terms), interoperability is defined as follows: "The capability to communicate, execute programs, or transfer data among various functional units in a manner that requires the user to have little or no knowledge of the unique characteristics of those units" 6

7. CHAIN interoperability test: objectives  To demonstrate that:  e-Infrastructures can be made interoperable to each other using standards (with the meaning of interoperability given above)  VRC-specific applications can be submitted from anywhere and run everywhere 7

8. 1. User interface is only web based, for simplicity 2. Users must be transparently authenticated & authorised on all e-Infrastructures without any additional human/machine intervention 3. There must be the smallest possible interaction with both site managers and e-Infrastructure operators 4. No modification of the middleware should be requested to developers CHAIN interoperability test: requirements 8

9. 9 Grid Engine User Tracking DB Science GW Interface SAGA/JSAGA API Job Engine Data Engine User Track. & Monit. Science GW 1 Science GW 2 Science GW 3 Grid MWs Liferay Portlets eToken Server DONEIn progress Catania Grid Engine 9 By mid April DONE

10. Job Engine  The Job Engine is made of a set of libraries to develop applications able to submit and manage jobs on a grid infrastructure  It is compliant with the OGF SAGA standard;  It is optimized to be used in a web portal running an application server (e.g., Glassfish, Tomcat, etc.) based on J2EE  It can be used also in stand-alone mode  JSAGA is the SAGA implementation adopted 10

11. Job Engine -Architecture WT Worker Threads for Job Submission WT Worker Threads for Status Checking USER TRACKING DB MONITORING MODULE GRID INFRASTRUCTURE(S) Job Queue WT Job Submission Job Check Status/ Get Output

12. 12 Job Engine - Features  The Job Engine has been designed with the following features in mind: FeatureDescriptionStatus Middleware Independent Capacity to submit job to resources running different middleware DONE EasinessCreate code to run applications on the grid in a very short time DONE ScalabilityManage a huge number of parallel job submissions fully exploiting the HW of the machine where the Job Engine is installed DONE PerformanceHave a good response timeDONE Accounting & Auditing Register every grid operation performed by the usersDONE Fault ToleranceHide middleware failure to end usersALMOST DONE WorkflowProviding a way to easily create and run workflowsIN PROGRESS

13. 13 Job Engine – Scalability  Submission time scales linearly with number of jobs;  Actual time depends on HW capabilities and thread-pools configuration 40000 Jobs submitted at the same time! Time to submit 10000 jobs (h) Job submission time (h)

14. Job Engine – Accounting & Auditing  A powerful accounting & auditing system is included in the Job Engine  It is fully compliant with EGI VO Portal Policy and EGI Grid Security Traceability and Logging Policy  The following values are stored in the DB for each job submitted:  User ID  Job Submission timestamp  Job Done timestamp  Application name  Job ID  Robot certificate ID  VO name  Execution site (name, lat, long) 14 Overall usage (Dec. 2011 → today)

15. gLite-based e-Infrastructures/Projects EUAsiaGrid EUChinaGRID EU-IndiaGrid EUMEDGRID GISELA IGI (Italy) SAGrid (South Africa) CHAIN interoperability test: e-Infrastructures involved so far 15

16. Step 1 – Grid authorisation  User VO(s) are authorized at all sites of the various e-Infrastructures  This would break requirement 3 and could require quite some time to be done  The certificates of the Science Gateway(s) to be used for the test are registered in the gLite VOMS servers and Unicore XUUDB 16

17. Step 2 – Resource discovery  Create a (super)-topBDII that gathers all the topBDII’s of the various e-Infrastructures  This would break requirement 3 and require a grid service to be managed by CHAIN  Information can become outdated  Insert the list of the topBDIIs/WMSs/TargetSystems of the various e-Infrastructures/middleware in the configuration of the portlets of the test applications and let the Grid Engine choose at job submission resource managers and services to be used 17

18. Multi-infra/multi-mw Job Engine EUMEDGRID e-Infrastructure Juelich Supercomputing Centre GISELA e-Infrastructure Multi-infrastructure/ Multi-middleware Science Gateway gLite Infrastr. Info (BDII,VO, etc.) gLite Infrastr. Info Unicore Infrastr. Info User Submit 18

19. MyJobsMap (1/4) 19

20. MyJobsMap (2/4) 20

21. MyJobsMap (3/4) 21

22. MyJobsMap (4/4) 22 Both sequential and MPI-enabled jobs successfully executed

23. Summary and conclusions  The adoption of standards (JSR 286, SAML, SAGA, etc.) in the Science Gateway proposed and fostered by CHAIN represents a first step towards a global interoperable architecture  The model is used to demonstrate CHAIN global harmonisation of e-Infrastructures world-wide  The model nicely implements HTC/HPC interoperability at the user/application level 23

24. Co-ordination & Harmonisation of Advanced e-INfrastructures Thank you