1. 1 WS-PGRADE/gUSE generic DCI gateway framework for EGI user communities http://www.sci-bus.eu Zoltan Farkas and Peter Kacsuk MTA SZTAKI SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481

2. 2 Motivations There are many user communities who would like to access several DCIs (grids, clouds, clusters) in a transparent way They do not want to learn the peculiar features of the used DCIs They want to concentrate on their scientific application Therefore they need a science gateway

3. Who are the members of an e-science community? End-users (e-scientists) (50.000-500.000) Execute the published WF applications with custom input parameters by creating application instances using the published WF applications as templates WF Application Developers (500-1.000) Develop WF applications Publish the completed WF applications for end-users SG Instance Developers (50-100) Develop application domain specific SG instance Writes technical, user and installation manuals Science Gateway (SG) Framework Developers (5-10) Develop generic SG framework Writes technical, user and installation manuals

4. What do WF developers need? WF App. Repository Access to a large set of ready-to-run scientific WF applications Accessing a large set of various DCIs to make these WF applications run Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems E-science infrastructure Clouds SG framework Using a SG framework to develop applications and share them with others

5. What do e-scientists need? App. Repository SG instance E-scientists Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems E-science infrastructure Clouds Sharing ready-to-use appls via the repository Joint run appls via the gateway in the hidden e-science infrastructure

6. 6 How to build a science gateway? Option 1: Build from scratch –If the gateway is not extremely simple, it requires long time to develop a robust gateway –Requires substantial manpower and development cost –It is very specialized and as users start to use it and come up with new requirements it is difficult to extend in a scalable way –It typically represents an isolated development without belonging to an open source community

7. 7 How to build a science gateway? Option 2: Adapt and customize an existing gateway framework –Significantly reduces development time –Requires limited manpower and development cost –Produces a robust and usable service –The open source community is driving force for further development and extensions SCI-BUS provides the required core gateway and customization technology

8. Criteria to select an existing gateway framework 1.Robustness 2.Sustainability –Will it exist in 3 years time? –How big and trustable the community who develops it? 3.Functionalities 4.How easy to adapt for the needs of the new user community? 5.Scalability 6.Extendibility 8

9. 9 Main objectives of SCI-BUS Create a generic-purpose gateway framework Elaborate a gateway instance development technology based on a customisation methodology Establish production SG instance services both for national grids (horizontal user communities) and various science communities (vertical user communities) Develop business models to guarantee sustainability and commercial exploitation

10. Generic-purpose gateway framework 10 Based on Liferay WS-PGRADE (Web Services Parallel Grid Runtime and Developer Environment) gUSE (Grid User Support Environment) architecture General purpose Workflow-oriented portal framework Supports the development and execution of workflow-based applications Enables the multi-DCI execution of any WF Support the fast development of SG instances by a customization technology

11. Scalable architecture based on collaborating services

12. Seamless access to various types of DCIs WF Graph editor WEB-UI (HTML) Liferay WS- PGRADE portal Liferay WS- PGRADE portal Information System WF Storage File Storage Application Repository WF Interpreter Cloud submitter GT5 submitter Glite submitter ARC submitter Unicore submitter BOINC submitter WS (Axis) submitter GT5 Grid DCI-Bridge Client machine Portal Server machine DCIs BES interface BOINC Grid ARC Grid Cloud Broker

13. WS-PGRADE/gUSE main features Scalable architecture based on collaborating services Seamless access to various types of DCIs: Clusters, service grids. desktop grids, clouds Advanced data-flows and workflow concept: Embedded workflows Parametric nodes Legacy codes Web services Advance input data management: Cross product Dot product Installation and configuration wizard

14. User access modes 14 ASM API WS-PGRADE UI Customized UI Other, existing UI WS-PGRADE workflow UI

15. Creating and running WS-PGRADE workflow in WF developer view 15 Step 1: Edit workflow

16. Step 2: Configuring the workflow

17. Running workflow instance

18. WS-PGRADE/gUSE community aspects App. Repository y Gateway Application developers E-scientists End-users (e-scientists) Specify the problem/application needs Execute the published applications via the gateway with custom input parameters by creating application instances Application Developers Develop e-science applications via the gateway in collaboration with e-scientists Publish the completed applications for end-users via an application repository Support collaboration between e-scientists and application developers

19. 19 WS-PGRADE/gUSE community aspects Separated views: application developer view: (access to all the features of WS-PGRADE/gUSE end-user view: limited access defined by the gateway sys admin Built-in workflow/application repository App developer can upload and download: Workflows Templates Applications End-user can only download: Applications

20. End-user UI: To build SG instances WF editing (and many other features) are not supported, only: –Appl (WF) download –Appl Parameterization –Appl Submission –Progress monitoring –Result download E.g. Autodock gateway 20

21. Example for SG instance based on end-user view: Autodock gateway 21

22. User access modes 22 ASM API WS-PGRADE UI Customized UI Other, existing UI WS-PGRADE workflow UI

23. WS-PGRADE Internal components submitter Desktop or Service grids / cloud / Clusters DCIs gUSE Web browser Client Customized web-interface 23 Developed by ASM Placing web-app as gUSE component Concept of Application Specific Module

24. Concept in detail Web browser Client Customized web- interface Web pages (JSP, JSF or other) JAVA event handler classes Application Specific Module API Configuration files (wsdd) You develop it 24 Configure web app interface using them Configure web app interface using ASM API

25. Types of gateways 1.Generic purpose gateway for national grids –Core WS-PGRADE/gUSE (e.g. Greek NGI) 2.Generic purpose gateway for a particular DCI –EDGI gateway based on WS-PGRADE/gUSE 3.Generic purpose gateway for specific technologies –SHIWA gateway based on WS-PGRADE/gUSE 4.Application-oriented science gateway instance 25

26. 26 Application-specific SG instances in SCI-BUS Create 11 SG instances in the 1st project year for the following user communities: –International seismology community –Helio-physics communityc –Swiss systems biology community of the SystemsX.ch project –German MoSGrid computational chemistry and bioinformatics community –Biomedical researchers community of the Academic Medical Centre of the University of Amsterdam –Astrophysics community –PireGrid SMEs community –Business process modelling community –Blender rendering community –Citizen web-2 community –Public application developer community (based on ETICS-2 results) Create another 6 user community SG instances based on 6 subcontractors Create another 10 user community SG instances based on associated membership

27. ETH Proteomics Gateway 27 Using WS-PGRADE for workflow creation gUSE for workflow management and submission Heavy use of the ASM API to present a simplified interface to end-users (this works extremely well) First instance: Private Gateway for one of the largest Proteomics research groups in Europe (inside ETH): Up to 100 users Access to 1200 cores local cluster and to ARC based Swiss Grid Planned future instance: Public Gateway for high-end Proteomics

28. Recent WS-PGRADE releases History since v3.4.0 Nov 2011: v3.4.0 (DCI Bridge) Feb 2012: v3.4.1 (usage statistics portlet) March 2012: v3.4.2 (support for new EMI release) April 2012: v3.4.3 (support for Liferay 6.1) May 2012: v3.4.4 (EDGI DCI support) June 2012: v3.4.5 (integrated SHIWA repository) June 2012: v3.4.6 (installation and config. wizard) … Aug 2012: v3.5.0 (SaaS cloud access via CBP) Sep 2012: v3.5.1 (IaaS cloud access via CBP)

29. Downloads statistics at sourceforge From 48 countries

30. 30 Where to find further information? gUSE/WS-PGRADE: –http://www.guse.hu/ gUSE on sourceforge –http://sourceforge.net/projects/guse/ –http://sourceforge.net/projects/guse/forums/forum/ –http://sourceforge.net/projects/guse/develop SCI-BUS web page: –http://www.sci-bus.eu/ SHIWA web page: –http://www.shiwa-workflow.eu/

31. Conclusions If you want to develop a SG instance, use a SG framework instead of developing it from scratch Why to select WS-PGRADE/gUSE and join the SCI-BUS community? 1.Robustness Already large number of gateways used in production 2.Sustainability The SCI-BUS project and its sustainability and commercialization plan guarantees it 3.Functionalities Rich functionalities that are growing according to the SCI-BUS and sourceforge community needs 4.How easy to adapt for the needs of the new user community? Already large number of gateways customized from gUSE/WS-PGRADE 5.You can influence the progress of WS-PGRADE/gUSE 31 Join SCI-BUS as associated member