1. The INDIGO-DataCloud contributions to the EOSC and next steps
Better Software for Better Science.
Davide Salomoni
INDIGO-DataCloud Project Coordinator
INDIGO-DataCloud Final Review
Brussels, 16-17/11/2017
RIA

2. Outline INDIGO components adopted beyond the project consortium
INDIGO in EOSC-hub
INDIGO in follow-on software development projects
The Common Dissemination Booster
Continuing the INDIGO brand
Perspectives for the Horizon 2020 Work Programme
Summary
INDIGO-DataCloud Final Review
Next steps

3. Adoption beyond the INDIGO Consortium
While difficult to track (all INDIGO products are freely available and open source), we know there are already several instances of INDIGO components adopted beyond the planned use cases and partners.
Some examples follow in the next slides
INDIGO-DataCloud Final Review
Next steps

4. INDIGO IAM
Federation of all types of access to resources under a single system, in centers typically offering local access (username/password e.g. for user interfaces or batch systems), LDAP/Kerberbos-based authentication and autorization, Cloud access
Authentication of users / groups using social logins and EduGain plugged into standard OpenStack installations
Use of IAM into applications that have originally nothing to do with Clouds, such as the Cultural Heritage Network (CHnet)
INDIGO-DataCloud Final Review
Next steps

5. udocker
udocker, being a user-level solution, is often completely invisible to system administrators or resource providers
It is being used to access e.g. HPC systems, with excellent results…
… to provision jobs on traditional Grid infrastructures such as EGI’s for bioinformatics…
… or even to provide on-demand user- interfaces, coupled to traditional batch systems
INDIGO-DataCloud Final Review
Next steps

6. The INDIGO Paas and front-ends
It is being used in the context of the Earth System Grid Federation to create a High-Performance Data Analytics stack to address performance issues, enable dynamic and automated deployment of software components and provide user-friendly web access.
This was tested and validated on CMIP5 datasets, with a large scale, international testbed involving several ESGF sites (LLNL, ORNL and CMCC), one orchestrator site (PSNC) and a site hosting INDIGO PaaS services (UPV).
Additional ESGF sites, such as NCI (Australia) and a couple more in Europe, are also joining the testbed.
To be presented with a talk at the American Geophysical Union fall meeting in December 2017.
INDIGO-DataCloud Final Review
Next steps

7. INDIGO-DataCloud in Horizon 2020
INDIGO-DataCloud Final Review
Next steps

8. Evolution of EU support to the EOSC
INDIGO-DataCloud Final Review
Next steps

9. INDIGO Advanced Components and Solutions
Mapping with INDIGO
Publicly funded e-infrastructures
(EGI, EUDAT, GEANT, PRACE, RI, etc.)
Private or Commercial
Clouds (Public, PCP-based, etc.)
Scientific Users
Adopt, Use
Deployed on
Exploiting
To produce
Scientific Results
INDIGO Advanced Components and Solutions
Datasets, Resources
INDIGO-DataCloud Final Review
Next steps

10. A key concept for INDIGO
In the EOSC, we believe that core points for both WP16-17 and WP relate to the need for better data management and for the exploitation of available, heterogeneous resources.
This concerns in particular the support of efficient, open and effective tools in cloud frameworks.
We think that the practical and realistic approach followed by research communities in their Case Studies in INDIGO, linking theoretical considerations with actionable mechanisms implemented in the Cloud, is a winning point.
INDIGO-DataCloud Final Review
Next steps

11. INDIGO & EOSC in production: >= TRL8*
Several INDIGO solutions and activities are in the new EOSC-hub proposal, submitted by a consortium of 74 partners under the coordination of EGI, EUDAT and INDIGO-DataCloud
With INDIGO components such as Identity and Access Management, Token Translation, Virtual filesystems (Onedata), Advanced IaaS Services, the Infrastructure Manager, the INDIGO PaaS and its orchestrator, web front-end services, user-level containers
INDIGO was also given the overall technical coordination of the project and will contribute to training, support, external liaison, stakeholder engagement, policy.
*: TRL = Technology Readiness Level. TRL 8 means production-ready at scale
INDIGO-DataCloud Final Review
Next steps

12. INDIGO in the EOSC-hub Thematic Services
DESCRIPTION
INDIGO SOLUTIONS specified in the proposal
ECAS
Climate Analytics Service (ECAS)
provided by ENES
Ophidia, KEPLER, FutureGateway
DARIAH SG
DARIAH science gateway
tailored for the digital arts and humanities communities
OneDock, OpenStack Nova Docker, FutureGateway, Onedata
OPENCoastS
OpenCoastS: On-demand Operational Coastal Circulation Forecast Service
INDIGO udocker
Infrastructure Manager
Orchestration (TOSCA, HEAT)
WeNMR
Structural biology services
DISVIS, POWERFIT, HADDOCK, GROMACS, AMPS-NMR, CS-ROSETTA, UNIO, FANTEN
IAM, PaaS Orchestrator, Infrastructure Manager
FutureGateway, Onedata
DODAS
Dynamic On Demand Analysis Service
IAM, TTS, PaaS Orchestrator, Orchent, IM, TOSCA, Onedata, FutureGateway
LifeWatch
PAIRQURS, Citizen science services, GBIF, Digital Knowledge preservation framework, remote monitoring and smart sensing.
EO Pillar
Earth observation services coordinated by ESA.
The tools are: MEA, EPOSAR, Sentinel playground, Datacube analytic service, Geohazards exploitation platform,OSS-X Sentinel service
Onedata. Analysis of INDIGO cloud software add-ons for OpenStack.
CMI
The Component MetaData Infrastructure
Including the Virtual Language Observatory and the Virtual Collection Registry, provided by CLARIN
Providing interoperable metadata for (digital) humanities between both CLARIN and DARIAH.
GEOSS
GEO DAB (Discovery and Access Broker) GEOSS portal -
INDIGO-DataCloud Final Review
Next steps

13. INDIGO & EOSC in evolution: < TRL8
Novel features evolving INDIGO components are a key part of two proposals to the EINFRA call (eXtreme-DataCloud and DEEP Hybrid DataCloud), both recently approved and started in November 2017:
Intelligent dataset distribution and data lifecycle management
Smart caching
Orchestrating Computing Workflows based on policy driven or adaptive data movements
Flexible metadata management for big data sets
Access to bare-metal resources on the Cloud
PaaS-Level access to HPC resources
Extensions to the INDIGO Orchestrator for hybrid IaaS deployments and scale out to 3rd party clouds
Extensions to the INDIGO Virtual Router Appliance
Real-time, streaming-based data ingestion and processing
INDIGO-DataCloud Final Review
Next steps

14. eXtreme-DataCloud (XDC)
From the DoA:
“The eXtreme DataCloud (XDC) project will develop scalable technologies for federating storage resources and managing data in highly distributed computing environments.”
“The XDC roots lie in the experience of the INDIGO- DataCloud project. It catches concepts and methodologies developed in that project, in particular those connected with Storage Quality of Service and data lifecycle management.”
INDIGO-DataCloud Final Review
Next steps

15. DEEP-Hybrid DataCloud
From the DoA:
“DEEP Hybrid DataCloud project [will] support intensive computing techniques that require specialized HPC hardware, like GPUs or low latency interconnects, to explore very large datasets. A Hybrid Cloud approach enables the access to such resources that are not easily reachable by the researchers at the scale needed in the current EU e-infrastructure.”
“The project will evolve to TRL8 existing services and technologies at TRL6+, including relevant contributions to the EOSC by the INDIGO-DataCloud H2020 project, that the project will enrich with new functionalities.”
INDIGO-DataCloud Final Review
Next steps

16. The Common Dissemination Booster
INDIGO-DataCloud is the leading project in a group of 3 projects (INDIGO-DataCloud, eXtreme-DataCloud, DEEP-HybridDataCloud), which together submitted a proposal (approved) to the Common Dissemination Booster (CDB) for all 5 CDB services.
This will help to establish branding in EU projects at the EOSC level, and will contribute to the exploitation of project results at both research and business levels
INDIGO-DataCloud Final Review
Next steps

17. Continuing the INDIGO brand
A proposal for an “INDIGO Software Stack Providers Collaboration Agreement” is being discussed by the INDIGO PMB and will be submitted soon to the INDIGO-DataCloud Collaboration Board
In this agreement, we propose that partners join in contributing, extending or maintaining INDIGO-DataCloud software components beyond the boundaries of – for example– Horizon 2020 projects.
The intended benefits are:
Support of software components as a part of a well established and known Service Catalogue and Architecture
Avail of well-established dissemination channels (e.g. CDB)
Find common exploitation channels to reach users with new services
Stronger influence on larger initiatives (such as those linked to the EOSC)
Opportunities for new common proposals
INDIGO-DataCloud Final Review
Next steps

18. INDIGO in the WP
We are closely analyzing the calls recently published in the WP There are several inputs that INDIGO could give to the EOSC vision and implementation:
continued input to the EOSC Service Catalogue
development of appropriate FAIR services and tools for depositing, curating and analyzing data
access to commercial services through the EOSC hub
develop agile, fit-for-purpose and sustainable services for the evolving needs of the scientific community
HPC and Big Data enabled large-scale test-beds and applications
Big Data technologies and extreme-scale analytics
Outputs are linked to many of the initiatives reported above, and will be pursued in the coming months
INDIGO-DataCloud Final Review
Next steps

19. Summary
In 30 months, the INDIGO-DataCloud project realized a comprehensive involvement of many Research Communities and providers for the definition and tracking of their requirements.
We identified technology gaps linked to several concrete use cases in multiple fields. The INDIGO architecture was defined, published and implemented.
We produced 2 major software versions and 14 minor updates, releasing 47 open modular components that made possible many things, that before INDIGO were either very difficult or just not doable. We exploited key European know- how, reusing and extending open source software and contributing to upstream projects. We established software development and management processes, and defined development and pre-production distributed testbeds.
Production deployment of many applications making use of the INDIGO software is well underway, and INDIGO components have been selected for production use in big infrastructures, commercial companies, external projects.
Several opportunities for further exploitation of INDIGO components are being explored and already implemented, in the context of the EOSC and beyond.
Requirements
Technology
Software &
Testbeds
App Deployment
Exploitation
INDIGO-DataCloud Final Review
Next steps

20. INDIGO-DataCloud Final Review
Next steps

21. https://www.indigo-datacloud.eu Better Software for Better Science.
Thank you
Better Software for Better Science.
@indigodatacloud
INDIGO-DataCloud Final Review
Next steps