Connecting Workflow-Oriented Science Gateways to Multi-Cloud Systems Zoltán Farkas, Péter Kacsuk, Ákos Hajnal MTA SZTAKI

Aim of the paper Cloud technology has been there now for a while Continuously deprecating grid middleware technologies Science gateways enabling easy processing of scientific computation should be able to make use of cloud technologies as well This transition should be completely transparent for the scientists IWSG’15, 7 th International Workshop on Science Gateways2

Concept of cloud-enabling workflow- oriented science gateways Three areas: job submission, data management and security Abstract architecture for discussing integration possibilities: IWSG’15, 7 th International Workshop on Science Gateways3 Workflow Design UI Workflow Design UI Workfllow Storage Workfllow Storage Workflow Enactor Job Submission Component Plugin

Job submission possibilities Necessary actions to enable submission to a given cloud middleware: o Workflow Design UI extension: new UI components should be created that enable the cloud-specific configuration of the workflow node (ideally automatically generated based on a template) o Workflow Storage extension: the Storage should store the cloud-specific properties of workflow nodes as well (ideally automatically serialized) o Job Submission Component extension: this component should have a new plugin implemented which is able to arrange job submission to the target middleware based on the node’s description in the Storage All this performed as many times as many cloud services implementing different services we’d like to support IWSG’15, 7 th International Workshop on Science Gateways4

Job submission to multiple cloud systems – Direct method IWSG’15, 7 th International Workshop on Science Gateways5 Workflow Design UI Workflow Design UI Workfllow Storage Workfllow Storage Workflow Enactor Job Submission Component Plugin 2 Plugin 1 UI 1 UI 2 Cloud 1 Cloud 2 Data 1 Data 2 Cloud 1’ API 1 API 2

Job submission to multiple cloud systems – Intermediate component Instead of interfacing with multiple cloud APIs, interface with a single Generic Cloud Access Service (GCAS) which hides the access details of different cloud technologies One single extension of the Workflow Design UI is necessary Single extension of Workflow Storage One additional plugin in the Job Submission Component IWSG’15, 7 th International Workshop on Science Gateways6

Job submission to multiple cloud systems – Intermediate component IWSG’15, 7 th International Workshop on Science Gateways7 Workflow Design UI Workflow Design UI Workfllow Storage Workfllow Storage Workflow Enactor Job Submission Component GCAS Plugin GCAS UI Cloud 1 Cloud 2 GCAS Cloud 1’ API 1 API 2 GCAS Plugin 1 Plugin 2

Data handling Data sooner or later needs to be transferred between the science gateway and the cloud infrastructure Direct job submission: o The data is transferred directly between the cloud storage and the science gateway GCAS-based job submission: o Option 1: data is transferred through GCAS (e.g. if no plain HTTP-based access is available) o Option 2: data transfer is initiated through GCAS, but actual transfer happens between the cloud storage and the science gateway (e.g. after redirecting HTTP GET requests from GCAS to the cloud storage) IWSG’15, 7 th International Workshop on Science Gateways8

Data handling overview IWSG’15, 7 th International Workshop on Science Gateways9 Workflow Design UI Workflow Design UI Workfllow Storage Workfllow Storage Workflow Enactor Job Submission Component GCAS Plugin Cloud 1 Cloud 2 Cloud 1’ API 1 API 2 GCAS Plugin 1 Plugin 2 Opt1 Opt2

Security considerations Direct access to cloud services: o The credentials need to be defined at the science gateway by the user o OR hidden, with the help of robot credentials GCAS-based access: o Option 1: GCAS can hide the access credentials of the cloud infrastructures it supports, thus users need to authenticate only against the GCAS, so one single auth token needs to be provided by the user o Option 2: GCAS can accept credentials from users, and forward them to the target cloud, so the user needs to define as many credentials as many cloud infrastructures she/he wants to access o OR, robot credentials can be used to hide GCAS/cloud access credentials IWSG’15, 7 th International Workshop on Science Gateways10

Direct vs. GCAS cloud access IWSG’15, 7 th International Workshop on Science Gateways11 DirectGCAS Time to implementM*N (M=number of different APIs) N Difficulty from user’s perspective IntermediateSimple (one credential to access)

Gateway and cloud access use-cases In an academic environment, the gateway and/or the cloud infrastructure being used might not be publicly available In such case applying a single GCAS service might be problematic IWSG’15, 7 th International Workshop on Science Gateways12 Portal AccessCloud AcccessGCAS usedSolution Public Yes or NoNo special setup needed Public (or Private, but external) PrivateYes or NoOpen firewall, use proxy Private YesOpen firewall, use proxy Private NoNo special setup needed

Public gateway, public cloud access IWSG’15, 7 th International Workshop on Science Gateways13 Workflow Design UI Workflow Design UI Workfllow Storage Workfllow Storage Workflow Enactor Job Submission Component GCAS Plugin Cloud 1 API 1 GCAS Plugin 1 Services publicly accessible No special setup needed

Public (or private external) gateway, private cloud access IWSG’15, 7 th International Workshop on Science Gateways14 Workflow Design UI Workflow Design UI Workfllow Storage Workfllow Storage Workflow Enactor Job Submission Component GCAS Plugin Cloud 1 API 1 GCAS Plugin 1 Cloud API on a private IP Proxy or firewall setup needed

Private gateway, private cloud access IWSG’15, 7 th International Workshop on Science Gateways15 Workflow Design UI Workflow Design UI Workfllow Storage Workfllow Storage Workflow Enactor Job Submission Component GCAS Plugin Cloud 1 API 1 GCAS Plugin 1 Portal and Cloud API in the same network Direct access: no setup needed GCAS: proxy/firewall needed

Cloud access implementetion based on GCAS Science gateway: WS-PGRADE/gUSE GCAS: CloudBroker Platform IWSG’15, 7 th International Workshop on Science Gateways16

WS-PGRADE/gUSE Overview Based on Liferay General purpose Workflow-oriented gateway framework Supports the development and execution of workflow-based applications Supports the fast development of domain- specific gateways by customization methodologies Most important design aspect is flexibility IWSG’15, 7 th International Workshop on Science Gateways17

WS-PGRADE/gUSE Overview IWSG’15, 7 th International Workshop on Science Gateways18 DCI Bridge: o Job submission service of WS-PGRADE/gUSE o Accepting job submission requests from the workflow interpreter through a standardized interface o Offering access to different types of middlewares Data Avenue: o Data management component o Offers convenient interfaces (web based and API) for managing data located on different types of storages o Also used by DCI Bridge and WS-PGRADE to provide data for workflow nodes

The CloudBroker Platform Easy, scalable, secure, integrable and pay-per-use access to scientific and technical applications in the cloud High performance computing application store and marketplace with accounting and billing Deployment and execution of applications in the cloud with the same parameters and files as for local execution Accessible through the internet via a web browser UI Integration possibilities into third party tools via web service-based APIs Public version available under Using infrastructure as a service (IaaS) from cloud providers Offering platform as a service (PaaS) for software vendors Providing software as a service (SaaS) to end users Available compute resources: Amazon, IBM, OpenStack, OpenNebula, CloudSigma Available storage resources: Amazon, Walrus, Rados S3

CloudBroker Platform Architecture CloudBroker Platform Amazon Cloud IBM Cloud … Cloud Chemistry Appli- cations Biology Appli- cations Medicine Appli- cations Web Browser UI Engineering Appli- cations Web Service API Generic Workbenches CloudBroker Integration Domain-Specific Gateways R&D End Users and Software Vendors CLI … Appli- cations

gUSE and CBP integration DCI Bridge: o Extended with a CloudBroker plugin o Making use of the Java API of CBP WS-PGRADE: o Authentication portlet: allows users to enter their CBP credentials for communicating with the CBP on behalf of the user o Workflow portlet: workflow node configuration extended with CBP-specific interfaces Workflow Interpreter: o Extended with generating JSDL for DCI Bridge based on CBP-specific node properties IWSG’15, 7 th International Workshop on Science Gateways21

WS-PGRADE gUSE CloudBroker Platform Seismology Proteomics Rendering Simulation Astrophysic s Workflow Portlet Authentication Portlet Portal Cache Portal Cache WFI DCI Bridge Repository Rest API Java API Web Interface Gromacs Blender Blast AutoDock OpenStack Adapter OpenStack Adapter OpenNebula Adapter OpenNebula Adapter Amazon EC2 Adapter Amazon EC2 Adapter Amazon S3 Adapter Amazon S3 Adapter Rados S3 Adapter Rados S3 Adapter OpenStack Cloud OpenNebula Cloud Amazo n EC2 Rados S3 Amazon S3

Integration features I. Support for accessing: Amazon EC2, OpenStack, OpenNebula, CloudSigma Data transfer happens between WS-PGRADE/gUSE and the cloud storage – no need to flow data through the CloudBroker Platform PaaS-like mode: users can run their own application in the cloud inside a workflow node SaaS-like mode: users can select from a set of predeployed applications to be run in the cloud inside a workflow node

Integration features II. Slight modification in the fully featured UI Also available in every customization level, completely hiding the fact that cloud is being used: o ASM o Remote API o End-user view o DCI Bridge

Integration features III. Support for commercial clouds with costs (prices configured in CloudBroker Platform): o Estimated job cost before submission o Actual job and workflow cost after execution

Projects making use of WS-PGRADE/gUSE and CBP integration SCI-BUS CloudSME IWSG’15, 7 th International Workshop on Science Gateways26

Summary Presented different methodologies of connecting science gateways to cloud infrastructures Focusing on an implementation connecting the WS- PGRADE/gUSE science gateway framework to cloud infrastructures with the help of the CloudBroker Platform The presented implementation enables easy migration of already existing gateways onto the cloud platform – only workflow reconfiguration is necessary

Thank you for your attention! Questions? The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/ ) under grant agreements no (SCI- BUS) and no (CloudSME). 28IWSG’15, 7 th International Workshop on Science Gateways