1. User Interfaces: Science Gateways, Workflows and Toolkits
Marcin Plociennik
On behalf of WP6 team
RIA

2. Introduction
INDIGO-DataCloud

3. Motto
A user interface is like a joke. If you have to explain it, it’s not that good!
INDIGO-DataCloud

4. Initial assumptions
Focus on use cases requirements during project lifetime !!!!
Develop Toolkits (libraries) that will allow the platform usage from the level of the Scientific Gateways, desktop and mobile applications
Provide both a general-purpose multi-domain Science Gateway framework and customized examples for selected user scenarios
Manage the execution of complex workflows using PaaS layers: Focus on interactive and batch parallel data analytics workflows, services in a Workflows-as-a-Service model
Provide workflow interfaces extensions for distributed and parallel data analytics on large volume of scientific, multidimensional data
05/05/2016
INDIGO-DataCloud

5. Highlights
05/05/2016
INDIGO-DataCloud

6. Main achievements Clear added value on top of INDIGO software stack:
provide building blocks for Scientific Gateways, desktop and mobile applications
User story driven
Released 9 new user level services/extended products/tools
Integration with 6 project application use cases,
Exploitation:
7 use cases from the Sci-GaIA project already use WP6 services
Involvement of the international climate community (scientists from USA: LLNL and ORNL) for requirements elicitation/validation and distributed testbed
Outreach and community building :
Presentations – over 30 on 20 events
Organising e-Research Summer Hackfest – 6 set of training materials!
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

7. Products Ophidia FutureGateway OMT Liferay IAM JSAGA RM Portlets
Kepler
Orchent, TTSc
05/05/2016
INDIGO-DataCloud

8. Outreach and community building
European Geosciences Union 2015 (12-17 April 2015, Vienna, Austria)
RDA Sixth Plenary Meeting (23-25 September 2015, Paris, France)
EOScience2.0 (12-14 October 2015, Frascati, Italy)
ESGF F2F Meeting 2015 (7-11 December 2015, S. Francisco, CA, USA)
AGU2015 Conference (14-18 December 2015, S. Francisco, CA, USA)
Ophidia PlayDay (29 April 2016, Bologna, Italy)
Invited presentation at LLNL (23 May 2016, Livermore, CA, USA)
Invited presentation at ORNL (26 May 2016, Oak Ridge, TN, USA)
Big Data and Extreme scale Computing (15-17 June 2016, Frankfurt, Germany)
…. (CMCC Annual Meeting, DI4R – Krakow, ICCS – San Diego, RDA workshop Hamburg) …..
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

9. Training material
Training Material (developed for the e-Research Summer Hackfest)
The FutureGateway framework - Overview : Slides - Video part 1 , Video part 2
The FutureGateway framework - APIs: Slides - Video part 1, Video part 2
The FutureGateway framework - Installation: Slides - Video part 1, Video part 2, Video part 3
The Ophidia platform: Slides - Video part 1 
The Ophidia platform tutorial: Slides - Video part 2, Video part 3
The Kepler workflow manager: Slides - Video, Video live demo
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

10. Feedback review (from Users – WP2)
version
EGI Conference- November 2015
CloudScape - March 2015
1st offical INDIGO Release – July 2016
Timeline
Prototype
(WP3)
Feedback review (from Users – WP2)
Adjustment
(WP6/WP5/WP2)
Development
Integration and test
WP2(users) WP5 WP4
High level requirements
Collect and define requirements
Architecture
Use case
ENES
Molecular Dynamics
Galaxy/Elixir
Algae Bloom- Lifewatch
EMSO
LBT

11. Team – collaboration
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

12. High-level Architecture
05/05/2016
INDIGO-DataCloud

13. High-level Architecture
Data Analitics
Workflow Portlets
Ophidia
Other Science Gateways
Mobile Apps
GUI
Clients
Kepler plugin
OMT
App Portlets
Future Gateway Portal
Workflows
Future Gateway REST API
Future Gateway Engine
JSAGA/JSAGA Adaptors
TOSCA
Template
CDMI/Web
Data Services
Kubernetes
PaaS Orchestrator
Other PaaS core Services IM
Cloud native APIs

14. Services offered
05/05/2016
INDIGO-DataCloud

15. Integrating distributed data infrastructures with INDIGO-DataCloud
List of the WP6 services
FutureGateway API Server
FutureGateway Portal, Liferay IAM and generic Portlets
JSAGA with the resource manager API
Ophidia analytics workflow interfaces extensions
INDIGO Kepler module/actors
Token Translation Service Client
Open Mobile Toolkit
Orchent
Integrating distributed data infrastructures with INDIGO-DataCloud

16. Science Gateways
05/05/2016
INDIGO-DataCloud

17. The FutureGateway (FG) – Added value
A set of components to build Science Gateways
The core is a RESTful API service
Manage applications and resources for user communities
Provide a highly customisable environment targeting different portal Frameworks and their technologies (i.e Liferay, Joomla, ...)
Modular architecture to support different back-end infrastructures (Cloud, Grid, local HPC sites)
Based on JSAGA Adaptors but not only; similar interfaces may be encapsulated in customizable modules (Executor Interfaces).
Collaborative evolution of the Catania Science Gateway Framework (CSGF)
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

18. Activity done in INDIGO
API services implemented from scratch
Multi-layer architecture
User interaction and infrastructure communication in two separate layers
Allow easier integration with other tools and portals
Token-based AuthN/AuthZ compliant with INDIGO IAM architecture
Integration with desktop and mobile applications
Not possible with the CSGF
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

19. FG Basic Components API Service LiferayIAM connector
Responsible for task management
A task can be any activity to be executed on a remote infrastructure
Three types of resources are defined:
Task
Application
Infrastructure
LiferayIAM connector
Enables authentication with IAM
Validates tokens
Generates new tokens on user behalf
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

20. Integration: Galaxy for ELIXIR use case
A FG portlet for Liferay has been developed to deploy Galaxy based application on INDIGO PaaS
Application is managed by the FG API service
Interaction with PaaS through the INDIGO Orchestrator
Integration to support the use case for ELIXIR
Demo will be shown
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

21. Integration: Kepler-based workflows
FG API service connected with a desktop application
Kepler workflow
Authentication managed by the user
User provides a valid token before task execution
Can be retrieved either from IAM or from a portal connected to the API service
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

22. Exploitation: adoption in other projects and by other communities
Sci-GaIA project (GA no )
Medical Image Processor and Repository (MIPAR)
Reproducible Automatic Speech Recognition System
iGrid Portal
Technology Transfer Alliance Collaboration Portal
Kenya National Public Health Gateway
Integration of Weather Research and Forecasting Model (WRF) in a Science Gateway
Integration of Waikato Environment for Knowledge Analysis (WEKA) in a Science Gateway
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

23. Command Line Clients
05/05/2016
INDIGO-DataCloud

24. Token Translation Service Client (TTSc) Current Status / Features
Current Release 0.4.0
Command line client for REST interface of TTS
Support the complete interface
Get list of all services supported by TTS
Get list of credentials of a user
Request credentials for a specific service
Revoke a specific credential
Completely new development for INDIGO DataCloud 25

25. Token Translation Service Client (TTSc) Roadmap
Extend help for the user on the command line
Support direct login at the command line
Use direct communication with INDIGO IAM, different OpenId Connect flow
Provide a single binary to distribute
Support all the features of TTS
e.g. public ssh-key upload 26

26. Orchent – The Orchestrator CLI Current Status / Features
Initial Release, complete new development in INDIGO DataCloud
Extensive help at the command line
Full use of the REST interface of the orchestartor
List all deployments, showing a specific deployment
List all resources, showing a specific resource
Create and update of deployments
Delete deployments
Single binary executable
Completely new development for INDIGO DataCloud 27

27. Toolkits for application developers
05/05/2016
INDIGO-DataCloud

28. Integrating distributed data infrastructures with INDIGO-DataCloud
JSAGA overview
JSAGA is a full-java implementation of the OGF SAGA (Simple API for Grid Applications) standard.
provide secured, homogeneous and reliable access to various Grid systems regarding job management and file management.
JSAGA provides a set of adaptors for current Grid systems (CreamCE, Globus Gatekeeper, SRM, LFC...) and standards (BES, HTTP, SSH...)
JSAGA provides the ability to easily develop and include customized adaptors
Integrating distributed data infrastructures with INDIGO-DataCloud

29. JSAGA in INDIGO – added value
JSAGA is used in the FutureGateway engine, the base of the INDIGO WP6 software
JSAGA does not target Cloud systems in a native way, but there is an extension to SAGA which has been defined at OGF: the Resource Management (RM) API.
The work to be done on JSAGA in to implement this extension. This is the JSAGA Resource Manager (JSAGA-RM).
JSAGA-RM will be used by the FG engine (through Resource adaptors) to access INDIGO resources.
Integrating distributed data infrastructures with INDIGO-DataCloud

30. JSAGA in INDIGO - status
Current status a first version of the JSAGA-RM is included in the latest JSAGA snapshot (1.2.0-SNAPSHOT) and ready to be used
Features included in the new jSAGA engine include wrapping to underlying technologies (cloud, pilot jobs, grid…) to an uniform API, asynchronous mode (task), timeout management, notification (metrics), security context forwarding.
The new jSAGA version which includes the Resource Management API is thus fully compatible with previous releases of jSAGA and existing Job/Data adaptors.
Source code:
Future work on JSAGA-RM: adapt/implement new features regarding FG needs
Integrating distributed data infrastructures with INDIGO-DataCloud

31. INDIGOclient Java wrapper
A Java wrapper over FutureGateway’s (FG) RESTful API
Contains data model automatically convertible in and out of JSON- encoded communication with the FG
Hides low-level REST interface under a well defined API with proper error handling and logging
Tested automatically with a mock FG implementation or a real instance for means of integration

32. Open Mobile Toolkit
Set of libraries to give mobile developers easier start to implement mobile applications based on INDIGO FG Server API
Design and implement the library for the Android platform (+apps – init. ENES)
Our main goal was to maximum simplify a start of the mobile apps development. Here are the steps for Android platform:
in the Application class you need to initialize the library (Indigo.init() method)
somewhere in your activity class invoke an Indigo API method (i.e. Indigo.getTasks()) and implement callbacks. All network and processing work will be executed in the separate threads and the results will be returned to the callbacks in the main thread, where you can update the UI
Sources:
Roadmap: Main activities in second stage: OMT for IOS + selected apps

33. Scientific workflows and big data analytics
05/05/2016
INDIGO-DataCloud

34. Ophidia in a nutshell
Big data stack for the analysis of scientific, multidimensional data
Time series analysis, data subsetting (by value/index), data aggregation, data intercomparison, predicate evaluation, OLAP support, etc.
Presently 50+ operators and 100+ primitives provided
Multiple interfaces available and programmatic access via C and Python APIs
Support for both batch & interactive data analysis
Support for analytics workflows
Link to International Working Groups
RDA Array Database Assessment WG
ESGF-Compute Working Team
Riassumendo, i principali contributi di questa tesi includono:…
e infine l’implementazione del terminale…

35. Main outcomes – INDIGO extension
Workflow extensions for big data analytics:
Iterative interface (i.e. to iterate over time, datasets, etc.)
Parallel interfaces (to exploit cluster resources, HTC tasks)
Massive interfaces (filters: both general-purpose and community-specific)
Interleaved interface (to synchronize loosely coupled workflows)
Interactive interface (to enable interactive-based scenarios)
Workflow as a Service:
Single container approach (completed) and Multiple VMs approach (ongoing)
Python modules: to ease the binding and interoperability with Python based eco-systems (e.g. IPython/Jupiter, NumPy/SciPy, etc.)
New import/export operators to support different communities (e.g. EMSO and LTB )

36. Contribution to the INDIGO Release
Ophidia is included in the 1st official INDIGO release
A new release has been also added (end of October) to INDIGO-1 Update 3
Git repos:
Git book with full documentation:
Ansible roles:
Galaxy role:

37. Exploitation
Several communities are exploiting/integrating Ophidia in their case study
ENES, EMSO, LTB, Lifewatch (support for different formats SAC, FITS, NetCDF (ongoing activity).
Case study on couple model intercomparison data analysis
Involvement of the climate community (scientists from CMCC, LLNL and ORNL) for requirements elicitation/validation (PTA experiment entirely designed with the end-user)
ESGF-oriented extensions will pave the way towards integration of the INDIGO solution into the community stack: Interaction with some key ESGF WGs (e.g. security, processing)
Two presentations (with demo) accepted at the ESGF F2F 2016 Conference (Washington December 6-9, 2016) about Ophidia and the INDIGO case study on climate change:
Most important event for the ESGF community
Community-oriented conference (at least attendees expected)
Meeting with the main ESGF Working Groups, joint roadmap
PTA = Precipitation Trend Analysis

38. INDIGO Kepler - overview
Extension of the scientific workflow Kepler ecosystem
Added value: building blocks for user workflows
Uses Ptolemy/Kepler model of actors as central units of a workflow
Every FutureGateway’s major action is a separate Kepler actor
Actors’ have strongly-typed I/O interface and error propagation to the Kepler workflow director
Exploitation: through Kepler open source community (15 domains)
Marcin Plociennik, et al. “Two-level dynamic workflow orchestration in the INDIGO DataCloud for large- scale, climate change data analytics experiments”, ICCS2016

39. INDIGO Kepler
Kepler already contains tens of useful actors, both generic and domain-specific
The INDIGO DataCloud workflows make use of many of them:
Data flow and logical actors
Local I/O
Image and text display
Kepler expression language
Various utilities (random number generation, etc.)
Sets of actors are grouped in composite actors which can be shared and re-used

40. Kepler use case on climate change data analysis

41. Release Kepler module available as part of the 1st release:
Source code available under indigo-dc GitHub organization:
Documentation in the form of GitBook:

42. Milestones and deliverables
05/05/2016
INDIGO-DataCloud

43. Short name of lead participant
Deliverables
Del. No.
Deliverable name
Short name of lead participant
Type
Dissemination level
Delivery date (month)
D6.1
Software architecture and work plan for WP6
IBCH PAS R PU 6
(Sep 2015)
D6.2
First toolkits prototype evaluation report 12
(Mar 2016)
D6.3
First release of a Science Gateways prototype and release of the selected mobile apps
INFN O 15
(Jun 2016)
D6.4
Evaluation report on first release 18
(Sep 2016)
D6.5
Second toolkits prototype evaluation report 21
(Dec 2016)
D6.6
Second release of a SG prototype and release of mobile apps 23
(Feb 2017)
D6.7
Final release
evaluation report 30
(Sep 2017)
05/05/2016
INDIGO-DataCloud

44. Milestones Mil. No. Milestone name Est. month Means of verification
Est. month
Means of verification
MS40
First release of Workflow as a Service toolkit 12
(Mar 2016)
First release of Workflow as a Service toolkit and interface extensions for scientific data analysis
MS41
First prototypes of the Science
Gateway framework and of mobile apps 15
(Jun 2016)
Delivery of the first release of the prototype of the Science Gateway framework and of the mobile apps
MS42
Second release of Workflow as a Service toolkit 21
(Dec 2016)
Second release of Workflow as a Service toolkit and interface extensions for scientific data analysis
MS43
Second prototypes of the Science 23
(Feb 2017)
Delivery of the second prototypes of the Science Gateway framework and of
mobile apps
05/05/2016
INDIGO-DataCloud

45. Implementation plan
05/05/2016
INDIGO-DataCloud

46. INDIGO-DataCloud RIA-653549
Further steps
M13-M21: Work on second release of Workflow as a Service toolkit and interface extensions for scientific data analysis
M16-M23: Work on second release of a Science Gateways prototype and release of the selected mobile apps
M17-M18: Evaluation of the first release of the WP6 components, update on the architecture
M24-M30: Patches, and updates
INDIGO-DataCloud RIA

47. Backup slides
05/05/2016
INDIGO-DataCloud

48. FutureGateway Roadmap
Optimisation of installation and deployment
Automation with DevOps common tools (e.g., Ansible)
Creation of a GUI for the management of resources in the FG
Improvement of documentation
Continuous support of the research communities and their use cases
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework

49. Ophidia roadmap
Final version of the data analytics workflow interfaces
Support for INDIGO communities exploiting Ophidia (e.g. EMSO, LTB, Lifewatch, ENES)
Support in terms of analytics operators & primitives exploited in the use cases
Support for the integration of “data” and “computing” aspects
The Python class will be extended and finalised to ease the exploitation of Ophidia in Python-based environments
Final version of the Workflow-as-a-Service support for cluster-based configurations
RDA Application regarding PID-based extensions of Ophidia to build a provenance-aware analytics eco-system (ongoing evaluation – Nov. 15)
PTA = Precipitation Trend Analysis

50. Orchent – The Orchestrator CLI Roadmap
Add support for login at the command line
Remove the need to request an access token from IAM
Follow the development of the Orchestrator
Adjust to needs of users 51

51. INDIGO Kepler roadmap
Keeping the project up-to-date with new requirements and functionalities of the FutureGateway
More user-friendly support for the valid AuthN/AuthZ session using IAM
Fully automated way of instantiation of Kepler running in a VNC- available virtual machine

52. FG Release FG release shipped with INDIGO middleware
Current release: INDIGO MidnightBlue
FG source code available on GitHub under Apache 2.0 licence
Under the indigo-dc organisation
FG REST API specification publicly available
FG official documentation
29 September Krakow, Poland
Enabling scientific applications on hybrid e-Infrastructures: the FutureGateway framework