1. FutureGateway Overview
Riccardo Bruno INFN Sez. Catania

2. Outline Architecture Environment Demos Next steps API FrontEnd
The APIServer DB
APIServerDaemon
Environment
Installation
Demos
1st Demo in Bari
Climate change (Cloudscape’16)
MD Molecular Dynamics (Cloudscape’16)
New MD case with TOSCA (Actually)
Next steps

3. From CSGF to FutureGateway
Classic CSGF (before INDIGO)
INDIGO Approach
An intermediate approach
Web/Mobile
Apps
Web/Mobile
Apps
Liferay/Glassfish
Liferay/Tomcat
Liferay/Tomcat
Portlet
Portlet …
Portlet
Portlet …
Portlet
Portlet …
REST APIs
REST APIs
GridEngine
API Server
API Server
Engine
GridEngine
JSAGA
JSAGA
JSAGA
Comunication Portlet-GridEngine-JSAGA
only possible with JAVA libraries
Comunication Portlet-API Server via REST APIs, this allows to serve external applications
The API Server interacts via JAVA libraries to JSAGA
Same model of the previous schema
The API Server Engine instructs the existing Grid Engine cutting development efforts in re-writing ad hoc JSAGA handlers

4. The API Server Engine (Actually for the CSGF’ GridEngine and SimpleTosca)
Web/Mobile
Apps
This mechanism allows the backward compatibility with portals built on top of the CSGF.
REST Frontend written in python, this allow fast prototyping of new features (changes on the fly, easy JSON data infrastructures handling, etc …)
REST Frontend may work standalone or as WSGI process (apache in the DEMO portal)
The system has a modular shape, since API Server daemon operates with ‘interfaces’, thus adding new queues, the daemon may target other Engines
API frontend may spread requests to different queues, or more daemons may read one queue in this way it is possible to support easily scalability issues
Supported GridEngine JSAGA adaptors: ssh, OCCI, wms, unicore,…
‘Modular’ API Server Engine
Liferay/Tomcat
Portlet
Portlet …
API Server
REST Frontend
REST APIs
Table based queue*
Target architecture
API Server
Engine
GridEngine
API Server daemon ?
*SAGA
JSAGA
SimpleTosca
GridEngine
Other Executors
Same model of the previous schema
The API Server Engine instructs the existing Grid Engine
* FIFO type but elements are dequeued by the daemon policies or from APIs

5. Architecture (two components)
The actual implementation foresees two components:
APIFrontEnd
Python code on top of Flask microframework
Very easy to develop/manage/configure
Uses a MySQL DB to configure:
Applications
Applicatin parameters
Application I/O files
Application Infrastructures and its parameters
Tasks (Submitted applications) with params args etc
Task queue (used to communicate with the APIServerDaemon)
APIServerDaemon
Java servlet running as a daemon on top of Tomcat application server
It timely polls over the task queue table and executes requested tasks
It instructs the right ‘executor interface’ to perform the requested task
Actually only the GridEngine and TOSCA interfaces are available:
GridEngine exploits JSAGA standard to execute on distributed infrastructures
GridEngine allows to execute on:
gLite/EMI wms, ssh, occi (rOCCI), (Others: Globus, Unicore, ...)

6. Architecture (figure)
API Front-end
APIServerDaemon
Target executors
Task queue
GridEngine If.
Other If.
Each task has a dedicated area in the FileSystem for I/O
APIServerDaemon interfaces are java classes
Other interfaces may connect directly to the DB (task queue)
More front-ends may use the same DB
More daemons may consume the same task queue
GridEngine
JSAGA
JSAGA Adaptors
Infrastructures

7. APIServer front-end
Available on GIT:
Written in python using Flask microframework
This component listens any Futuregateway API REST call in compliancy with specs. published at:
This service may run as:
Standalone service (Good for development environments or small requests traffic rate)
WSGI application (Suggested for production environments and high requests traffic rate)
The front-end uses a MySQL database to store:
Applications, Application files, Application parameters, Applicaton infrastructures, Application infrastructure parameters (Instruct JSAGA adaptors)
Tasks, Task arguments, task IO files, rutime data and the task queue
DB configuration is later explained

8. APIServer DB(*) APIServerDaemon
application
application_flile
application_parameter
infrastructure
Infrastructure_parameter
task
task_arguments
task_input_files
Task_output_files
runtime_data
as_queue
APIServerDaemon
(*) Tables are described in next slides

9. APIServerDaemon
Servlet that runs a daemon on top of Tomcat application server
The java application was necessary since JSAGA is available only via java language. A python wrapper exists but it is not fully supported as the native java version (pyjsaga)
APIServer daemon polls over the task queue table (as_queue)
Polling timing and other settings can be configured by a dedicated .properties file
APIServerDaemon read tasks requests from the queue, book them as ‘to process’ and then instruct the correct target interface executor to process each task
This service timely checks executed task with a simple consistency check algorithm. It re-tries failed requests up to a fixed amount of times. FAILED requests can be reported to the administrator
Upon task termination, the APIServerDaemon manages the task output and updates the DB task tables accordingly

10. APIServerDaemon ControlPanel
This feature exists but is not yet consolidated, it aims to provide a complete overview of daemon activity and eventually configure/manage it

11. First demo in November (Bari)
The first demo has been presented in Bari. At that time the APIServer was written only to support GridEngine and no target executors were available yet.
Two applications were presented:
HelloWorld
An application totally decoupled from Liferay portal demonstrating that any other portal could use the FutureGateway REST APIs
Climate data analysis
A simple GUI able to execute on top an existing Ophidia client VM through SSH JSAGA adaptor
Both applications demonstrates also modern and well interactive interfaces thanks to the adoption of jQuery and bootstrap js tools.

12. Use cases: ClimateChange
This demo has been demonstrated in Bruxelles at the CloudScape’16 conference
The demo uses the FutureGateway REST APIs from the Kepler workflow manager
The application piloted by Kepler works in the same way of the demo presented in Bari executing over SSH adaptor
This case uses only the APIs and its totally decoupled from the Liferay portal
Next version will use WP5 APIs through TOSCA adaptor

13. Use cases: Molecular Dynamics
This use case demonstrates a subset of the MD use cases as presented in WP2
This use case is capable to run MD symulations in two different fashions:
Run on Grid using JSAGA the wms adaptor
Run on a Cloud based docker container using the rOCCI adaptor (serveral changes have been applied to the adaptor to cover this case)
This case uses only the APIs. This case uses a FutureGateway installation without Liferay
This use case can has been modified to use WP5 resources through the adoption of the TOSCA adaptor

14. MD Scheme (CloudScape’16)
AMBER Web GUI
REST APIs
FutureGateway API Server
Grid Engine Interface
JSAGA
rOCCI
OpenStack
Docker Containers
MD Container
SSH
MD Job
I/O
Staging

15. The new use case: MD with TOSCA
At the time of this presentation the JSAGA TOSCA adaptor is now available.
Successful tests have been performed instantiating a Molecular Dynamics docker image through TOSCA using a dedicated yaml file to request it
In order to fully support the new TOSCA adaptor with existing APIServer, a new interface ‘SimpleTosca’ has been developed

16. FutureGateway API Server
MD Scheme with TOSCA
AMBER Web GUI
REST APIs
FutureGateway API Server
Grid Engine Interface
JSAGA
Tosca IM
Docker Containers
MD Container
SSH
MD Job
I/O
Staging
Highlighted in red the changes respect the previous use case.
Only the adaptor modifies the use case structure

17. Next steps
Thanks to its flexibility and its easy availability with the installation scripts, the present APIServer may be kept maintained for a while even at the time the new APIServer re-engineered to work without the GridEngine will be available.
The new available JSAGA Resource Manager feature may require to develop a dedicated target executor since the actual GridEngine does not supports it yet.
The introduction of the Resource Manager involves the creation of ‘virtual infrastructures’ which impact the existing REST calls; new specs are available already (not yet implemented)
AAI integration will impact the APIServer front-end
Storage APIs still to be implemented (missing Specs.)