FutureGateway Overview Riccardo Bruno INFN Sez. Catania

Outline Architecture Environment Demos Next steps API FrontEnd The APIServer DB APIServerDaemon Environment Installation Demos 1st Demo in Bari Climate change (Cloudscape’16) MD Molecular Dynamics (Cloudscape’16) New MD case with TOSCA (Actually) Next steps

From CSGF to FutureGateway Classic CSGF (before INDIGO) INDIGO Approach An intermediate approach Web/Mobile Apps Web/Mobile Apps Liferay/Glassfish Liferay/Tomcat Liferay/Tomcat Portlet Portlet … Portlet Portlet … Portlet Portlet … REST APIs REST APIs GridEngine API Server API Server Engine GridEngine JSAGA JSAGA JSAGA Comunication Portlet-GridEngine-JSAGA only possible with JAVA libraries Comunication Portlet-API Server via REST APIs, this allows to serve external applications The API Server interacts via JAVA libraries to JSAGA Same model of the previous schema The API Server Engine instructs the existing Grid Engine cutting development efforts in re-writing ad hoc JSAGA handlers

The API Server Engine (Actually for the CSGF’ GridEngine and SimpleTosca) Web/Mobile Apps This mechanism allows the backward compatibility with portals built on top of the CSGF. REST Frontend written in python, this allow fast prototyping of new features (changes on the fly, easy JSON data infrastructures handling, etc …) REST Frontend may work standalone or as WSGI process (apache in the DEMO portal) The system has a modular shape, since API Server daemon operates with ‘interfaces’, thus adding new queues, the daemon may target other Engines API frontend may spread requests to different queues, or more daemons may read one queue in this way it is possible to support easily scalability issues Supported GridEngine JSAGA adaptors: ssh, OCCI, wms, unicore,… ‘Modular’ API Server Engine Liferay/Tomcat Portlet Portlet … API Server REST Frontend REST APIs Table based queue* Target architecture API Server Engine GridEngine API Server daemon ? *SAGA JSAGA SimpleTosca GridEngine Other Executors Same model of the previous schema The API Server Engine instructs the existing Grid Engine * FIFO type but elements are dequeued by the daemon policies or from APIs

Architecture (two components) The actual implementation foresees two components: APIFrontEnd Python code on top of Flask microframework Very easy to develop/manage/configure Uses a MySQL DB to configure: Applications Applicatin parameters Application I/O files Application Infrastructures and its parameters Tasks (Submitted applications) with params args etc Task queue (used to communicate with the APIServerDaemon) APIServerDaemon Java servlet running as a daemon on top of Tomcat application server It timely polls over the task queue table and executes requested tasks It instructs the right ‘executor interface’ to perform the requested task Actually only the GridEngine and TOSCA interfaces are available: GridEngine exploits JSAGA standard to execute on distributed infrastructures GridEngine allows to execute on: gLite/EMI wms, ssh, occi (rOCCI), (Others: Globus, Unicore, ...)

Architecture (figure) API Front-end APIServerDaemon Target executors Task queue GridEngine If. Other If. Each task has a dedicated area in the FileSystem for I/O APIServerDaemon interfaces are java classes Other interfaces may connect directly to the DB (task queue) More front-ends may use the same DB More daemons may consume the same task queue GridEngine JSAGA JSAGA Adaptors Infrastructures

APIServer front-end Available on GIT: https://github.com/indigo-dc/fgAPIServer Written in python using Flask microframework http://flask.pocoo.org This component listens any Futuregateway API REST call in compliancy with specs. published at: http://docs.csgfapis.apiary.io/#reference This service may run as: Standalone service (Good for development environments or small requests traffic rate) WSGI application (Suggested for production environments and high requests traffic rate) The front-end uses a MySQL database to store: Applications, Application files, Application parameters, Applicaton infrastructures, Application infrastructure parameters (Instruct JSAGA adaptors) Tasks, Task arguments, task IO files, rutime data and the task queue DB configuration is later explained

APIServer DB(*) APIServerDaemon application application_flile application_parameter infrastructure Infrastructure_parameter task task_arguments task_input_files Task_output_files runtime_data as_queue APIServerDaemon (*) Tables are described in next slides

APIServerDaemon Servlet that runs a daemon on top of Tomcat application server The java application was necessary since JSAGA is available only via java language. A python wrapper exists but it is not fully supported as the native java version (pyjsaga) APIServer daemon polls over the task queue table (as_queue) Polling timing and other settings can be configured by a dedicated .properties file APIServerDaemon read tasks requests from the queue, book them as ‘to process’ and then instruct the correct target interface executor to process each task This service timely checks executed task with a simple consistency check algorithm. It re-tries failed requests up to a fixed amount of times. FAILED requests can be reported to the administrator Upon task termination, the APIServerDaemon manages the task output and updates the DB task tables accordingly

APIServerDaemon ControlPanel This feature exists but is not yet consolidated, it aims to provide a complete overview of daemon activity and eventually configure/manage it https://<fg_host>:<port>/APIServerDaemon/

First demo in November (Bari) https://sgw.indigo-datacloud.eu/applications The first demo has been presented in Bari. At that time the APIServer was written only to support GridEngine and no target executors were available yet. Two applications were presented: HelloWorld An application totally decoupled from Liferay portal demonstrating that any other portal could use the FutureGateway REST APIs Climate data analysis A simple GUI able to execute on top an existing Ophidia client VM through SSH JSAGA adaptor Both applications demonstrates also modern and well interactive interfaces thanks to the adoption of jQuery and bootstrap js tools.

Use cases: ClimateChange This demo has been demonstrated in Bruxelles at the CloudScape’16 conference The demo uses the FutureGateway REST APIs from the Kepler workflow manager The application piloted by Kepler works in the same way of the demo presented in Bari executing over SSH adaptor This case uses only the APIs and its totally decoupled from the Liferay portal Next version will use WP5 APIs through TOSCA adaptor

Use cases: Molecular Dynamics This use case demonstrates a subset of the MD use cases as presented in WP2 This use case is capable to run MD symulations in two different fashions: Run on Grid using JSAGA the wms adaptor Run on a Cloud based docker container using the rOCCI adaptor (serveral changes have been applied to the adaptor to cover this case) This case uses only the APIs. This case uses a FutureGateway installation without Liferay This use case can has been modified to use WP5 resources through the adoption of the TOSCA adaptor

MD Scheme (CloudScape’16) AMBER Web GUI REST APIs FutureGateway API Server Grid Engine Interface JSAGA rOCCI OpenStack Docker Containers MD Container SSH MD Job I/O Staging

The new use case: MD with TOSCA At the time of this presentation the JSAGA TOSCA adaptor is now available. Successful tests have been performed instantiating a Molecular Dynamics docker image through TOSCA using a dedicated yaml file to request it In order to fully support the new TOSCA adaptor with existing APIServer, a new interface ‘SimpleTosca’ has been developed

FutureGateway API Server MD Scheme with TOSCA AMBER Web GUI REST APIs FutureGateway API Server Grid Engine Interface JSAGA Tosca IM Docker Containers MD Container SSH MD Job I/O Staging Highlighted in red the changes respect the previous use case. Only the adaptor modifies the use case structure

Next steps Thanks to its flexibility and its easy availability with the installation scripts, the present APIServer may be kept maintained for a while even at the time the new APIServer re-engineered to work without the GridEngine will be available. The new available JSAGA Resource Manager feature may require to develop a dedicated target executor since the actual GridEngine does not supports it yet. The introduction of the Resource Manager involves the creation of ‘virtual infrastructures’ which impact the existing REST calls; new specs are available already (not yet implemented) AAI integration will impact the APIServer front-end Storage APIs still to be implemented (missing Specs.)