1. FutureGateway Overview Riccardo Bruno INFN Catania riccardo.bruno@ct.infn.it e-Research Summer Hackfest

2. Outline Introduction – Science Gateways – From CSGF to FutureGateway Architecture – APIServer Front-end – APIServerDaemon Usage – AuthN/Z Implementations – Demonstrator – Molecular Dynamics – Climate Change 2

3. A framework to build Science Gateways – "A Science Gateway is a community-developed set of tools, applications, and data that is integrated via a portal or a suite of applications, usually in a graphical user interface, that is further customized to meet the needs of a specific community." TeraGrid-XSEDETeraGrid-XSEDE Serves Community Contains tools applications and data Provides GUI (Web, desktop and mobile applications) – FutureGateway is the evolution of another product named: Catania Science Gateway Framework (CSGF)CSGF FG improves CSGF from well identified weakness especially those things addressing the learning curve FutureGateway 3

4. – Ease the installation and the maintenance It provides installation and maintenance scripts All sources available on GitHub Easily customizable for different community needs – Flexible and uniform access to distributed computing services The physical access to the distributed infrastructures comes through the JSAGA library (Grid&Cloud Engine, SimpleTosca) FG uses executor interfaces (customizable) – Back-end portal independency The FG can operate with any existing portal technology or even linked to mobile and desktop applications – Ease SG application development It provides a set of RESTful APIs so that the FG can be easily piloted by any programming language and/or web framework Improvements from CSGF 4

5. From CSGF to FutureGateway GridEngine JSAGA Portle t … … Classic CSGF (before INDIGO) Liferay/Glassfish Communication Portlet- GridEngine-JSAGA only possible with Java libraries GridEngine JSAGA Portle t … … Liferay/Tomcat API Server Same model of the previous schema The API Server instructs the existing Grid Engine and/or other interfaces REST APIs Web/M obile Apps Web/M obile Apps *SAGA ? FutureGateway Portlet – Contain GUI for SG applications Grid&Cloud Engine (GridEngine) – System to manage jobs running on Grid and Cloud infrastructrures SAGA/JSAGA – SAGA OGF standard to manage jobs on distributed infrastructures – JSAGA a Java implementation of SAGA REST APIs – Communication protocol over HTTP to manage network applications APIServer – The engine implementing REST APIs 5

6. The API Server GridEngine JSAGA Portle t … … Liferay/Tomcat API Server REST APIs Web/M obile Apps Web/M obile Apps API Server Frontend API Server Frontend API Server Daemon ‘Modular’ API Server Engine Table based queue * This mechanism allows the backward compatibility with portals built on top of the CSGF. REST Frontend written in python, this allow fast prototyping of new features (changes on the fly, easy JSON data infrastructures handling, etc …) REST Frontend may work standalone or as WSGI process (apache in the DEMO portal) The system has a modular shape, since API Server daemon operates with ‘executor interfaces’ (EIs) API frontend may spread requests to different queues, or more daemons may read one queue in this way it is possible to support easily scalability issues Supported GridEngine JSAGA adaptors: ssh, OCCI, wms, unicore,… Other Executors * FIFO type but elements are dequeued by the daemon policies or from APIs Target architecture *SAGA ? SimpleTosca 6 GridEngine

7. Architecture (figure) Target Executors (Executor Interfaces) Software components responsible to target a specific distributed infrastructure Database Manage the task queue Keep SG application definition Keep application infrastructure parameters Stores information about SG users and their groups and roles API Front-endAPIServerDaemon GridEngine If. Infrastructures Task queue JSAGA Adaptors Target executors Mobile and Web apps REST MySQL Applications Infrastructures SimpleTosca If.Other If. GridEngine JSAGA Users,Groups and Roles 7 xSAGA/Other

8. Architecture (two components) – APIFrontEnd (fgAPIServer) Python code on top of Flask microframework – Very easy to develop/manage/configure Uses a MySQL DB to configure: – Applications – Application parameters – Application I/O files – Application Infrastructures and their parameters – Tasks (Submitted applications) with params, args, etc. – Task queue (used to communicate with the APIServerDaemon) – APIServerDaemon Java servlet running as a daemon on top of Tomcat application server It timely polls over the task queue table and executes requested tasks It instructs the right ‘executor interface’ to perform the requested task – Actually only the Grid&Cloud Engine and TOSCA interfaces are available: » Grid&Cloud Engine: exploits JSAGA standard to execute on distributed infrastructures: gLite/EMI wms, ssh, occi (rOCCI), (Others: Globus, Unicore,...) » SimpleTosca: Simplified version of TOSCA based resource management: It accepts only one kind of input yaml file (used for demonstrative purposes) 8

9. APIServer front-end (fgAPIServer) Available on GIT: https://github.com/indigo-dc/fgAPIServerhttps://github.com/indigo-dc/fgAPIServer Written in python using Flask microframework http://flask.pocoo.orghttp://flask.pocoo.org This component listens any FutureGateway API REST call in compliancy with specs defined at: http://docs.csgfapis.apiary.io/#referencehttp://docs.csgfapis.apiary.io/#reference This service may run as: – Standalone service (Good for development environments or small requests traffic rate) – WSGI application (Suggested for production environments and high requests traffic rate) The front-end uses a MySQL database to store: – Applications, Application files, Application parameters, Applicaton infrastructures, Application infrastructure parameters (Instruct JSAGA adaptors) – Tasks, Task arguments, task IO files, rutime data and the task queue – FG Users/Groups/Roles, Log and Access tokens 9

10. APIServerDaemon Servlet that runs a daemon on top of Tomcat application server The Java application was necessary since JSAGA is available only via java language. A python wrapper exists but it is not fully supported as the native java version (pyjsaga) APIServer daemon polls over the task queue table (as_queue) Polling timing and other settings can be configured by a dedicated.properties file APIServerDaemon reads tasks requests from the queue, book them as ‘to process’ and then instruct the correct target interface executor for real processing This service timely checks executed task with a simple consistency check algorithm. It re-tries failed requests up to a fixed amount of times. FAILED requests can be reported to the administrator Upon task termination, the APIServerDaemon manages the task output and updates the DB task tables accordingly 10

11. APIServerDaemon ControlPanel This feature exists but is not yet consolidated, it aims to provide a complete overview of daemon activity and eventually configure/manage it https:// : /APIServerDaemon/ 11

12. Typical usages Liferay Portal APIServer FrontEnd/Daemon APIServer FrontEnd/Daemon APIServer FrontEnd/Daemon APIServer FrontEnd/Daemon Community Portal Portlet Portlet/WebApp/Web+Ajax APIServer FrontEnd/Daemon APIServer FrontEnd/Daemon Final Users Mobile&Desktop Apps Requires (INDIGO) AAI PrivateNet/Firewall/SSL Protection Used for: Molecular Dynamics Climate change Used for: INDIGO SG Demonstrator https://sgw.indigo- datacloud.eu Used for: Desktop & Mobile apps (Coming soon) The portal manages user access to portlets. CSGF-like setup 12 PTV

13. AuthN/Z in CSGF Portal 1 AuthN The user is redirected to IdP Login page 2 AuthZ User credentials are retrieved from authenticated users 3 Token The portal releases a Token that will be used to contact the APIServer. The API server needs a validy check (to be implemented) that identifies the Scope as well Robot proxy To be implemented 4 Robot Proxy The APIServer has the responsibility to generate and use Robot-Proxy to access the resources IdP LDAP AuthN AuthZ 1 2 APIServer eTokenserver Resources 5 4 5 Validity Check 3 R R R R 13

14. FG AAI in INDIGO https://www.indigo-datacloud.eu/documents/software-architecture-and-work-plan-wp6-d61 FutureGateway Portal AAI 1 2 3 4 1 Portal Access Users can access from remote desktop or mobile applications, or access from a web page. In any case the user will be redirected to a web portal capable to provide the AAI login page 2 Login The AAI component will provide the login interface to the user and will release the authentication token 3 REST Any REST API call will provide the authentication token inside the header section 4 REST-Engine Verify call Before to process any user’ REST call the REST engine verifies the authentication token which will grant user access in case of successful check REST Engine Interface Token check 14

15. FG AuthN/Z in INDIGO https://www.indigo-datacloud.eu/documents/software-architecture-and-work-plan-wp6-d61 Portal 1 IAM The user is redirected to IAM Login page 2 Token Once authenticated the user receives also AuthZ information, stored inside the Token 3 Portal and APIServer The portal contacts the ApiServer providing the IAM Token and the APIServer checks the validity of the incoming Token. The Scope and Policies will be applied accordingly to stored AuthZ information To be implemented IAM AuthN AuthZ 1 APIServer 3 Resources 4 AuthN/Z Validity Check 2 4 Resources IAM Token will be used to access the Resources 15

16. FutureGateway Baseline AuthN/Z Log-In 1 The user log-in providing its own credentials; in the baseline this uses (username and password). This log-in can be handled also with OAUTH authentication (INDIGO AAI, Facebook, Google, …) 2 LogToken The baseline Log-In code encrypts username, password and a timestamp in a LogToken string. In OAUTH cases the OAUTH token will be returned after the authentication 3 Token check Incoming LogTokens are processed by the TokenCheck function which in the baseline authentication just maps decrypted credentials with stored users in APIServer DB In OAUTH cases the TokenCheck function will use received user info to map the user with users registered into the APIServer DB. A new session token for mapped user is returned Log Token Username Password Timestamp Username Password Timestamp Key Username, Password Key Log Token APIServer Front-End APIServer Front-End Session Token … Token Expiration Key User details User Group(s) Group(s) roles User details User Group(s) Group(s) roles 1 Token check 2 3 4 Session Token Session Token will be used to call any further API REST call till token expiration 4 Session Token identifies the user and provides AuthZ informtaiotn 16

17. FG implementations so far Demonstrator – https://sgw.indigo-datacloud.eu https://sgw.indigo-datacloud.eu – Presented at the EGI Community forum in Bari 10-13/11/2015 Helloworld application (Grid&CloudEngine+SSH adapt.) ENES Portlet (Grid&CloudEngine+SSH adapt.) Molecular Dynamics – Presented at the CloudScape 02/2016 in Brussels – Demontrator Site currently in maintenance – Uses rOCCI JSAGA adaptor with Grid&Cloud Engine on top of a StackServer with Docker based compute node Climate Change (Ophidia+Kepler) – Hosted in Poznan, Kepler workflow engine exploits FG APIs to execute jobs on Virtual Appliances – Uses SSH JSAGA adaptor with Grid&Cloud Engine 17

18. The new use case: MD with TOSCA A JSAGA TOSCA adaptor is available (SimpleTosca) Successful tests have been performed instantiating a Molecular Dynamics Docker image through TOSCA using a dedicated yaml file to request it In order to fully support the new TOSCA adaptor with existing APIServer, a new interface ‘SimpleTosca’ has been developed 18

19. MD Schema (CloudScape’16) AMBER Web GUI REST APIs FutureGateway API Server GridEngine Interface JSAGA rOCCI OpenStack Docker Containers MD Container SSH MD Job I/O Staging 19

20. MD Schema with TOSCA AMBER Web GUI REST APIs FutureGateway API Server JSAGA Tosca IM Docker Containers MD Container SSH MD Job I/O Staging Highlighted in red the changes respect the previous use case. Only the adaptor modifies the use case structure GridEngine Interface 20

21. Development All source code published on GitHub: https://github.com/FutureGateway https://github.com/FutureGateway A full running instance easily installable on: – Stand-alone Ubuntu 14.04 Server LTS – Ubuntu 14.04 Server LTS Docker image – EGI FedCloud virtual appliance: https://appdb.egi.eu/store/vappliance/futuregateway Maintenance scripts and tools – Update software packages – Keep DB schema aligned with new packages – Development tools for code customizations 21

22. Questions ? 22