1. The PaaS Layer in the INDIGO-DataCloud
Giacinto Donvito
INDIGO-DataCloud WP5 Leader and TC
September 2016
RIA

2. D.Salomoni - The INDIGO-DataCloud MidnightBlue Release
This is the INDIGO-DataCloud
General Architecture*
*: see details in or in
September 2016
D.Salomoni - The INDIGO-DataCloud MidnightBlue Release

3. D.Salomoni - The INDIGO-DataCloud MidnightBlue Release
INDIGO MidnightBlue
Four main “solution blocks”:
Data Center Solutions
Data / Storage Solutions
Automated Solutions
User-Oriented Solutions
And “common solutions”:
Authentication and Authorization
September 2016
D.Salomoni - The INDIGO-DataCloud MidnightBlue Release

4. D.Salomoni - The INDIGO-DataCloud Platform
PaaS Features (1)
Improved capabilities in the geographical exploitation of Cloud resources.
End users need not know where resources are located, since the INDIGO PaaS layer is hiding the complexity of both scheduling and brokering.
Standard interface to access PaaS services.
Currently, each PaaS solution available on the market is using a different set of APIs, languages, etc. INDIGO uses the TOSCA standard to hide these differences.
Support for data requirements in Cloud resource allocations.
Resources can be allocated where data is stored.
Integrated use of resources coming from both public and private Cloud infrastructures.
The INDIGO resource orchestrator is capable of addressing both types of Cloud infrastructures through TOSCA templates handled at either the PaaS or IaaS level.
July 20, Jinan Cloud School
D.Salomoni - The INDIGO-DataCloud Platform

5. D.Salomoni - The INDIGO-DataCloud Platform
PaaS Features (2)
Distributed data federations supporting legacy applications as well as high level capabilities for distributed QoS and Data Lifecycle Management.
This includes for example remote Posix access to data.
Integrated IaaS and PaaS support in resource allocations.
For example, storage provided at the IaaS layer is automatically made available to higher-level allocation resources performed at the PaaS layer.
Support for distributed data caching mechanisms and integration with existing storage infrastructures.
INDIGO storage solutions are capable of providing efficient access to data and of transparently connecting to Posix filesystems already available in data centers.
July 20, Jinan Cloud School
D.Salomoni - The INDIGO-DataCloud Platform

6. D.Salomoni - The INDIGO-DataCloud Platform
PaaS Features (3)
Deployment, monitoring and automatic scalability of existing applications.
For example, existing applications such as web front-ends or R-Studio servers can be automatically and dynamically deployed in highly-available and scalable configurations.
Support for dynamic and elastic clusters of resources.
Resources and applications can be clustered through the INDIGO APIs. This includes for example batch systems on-demand (such as HTCondor or Torque) and extensible application platforms (such as Apache Mesos) capable of supporting both application execution and instantiation of long-running services.
July 20, Jinan Cloud School
D.Salomoni - The INDIGO-DataCloud Platform

7. D.Salomoni - The INDIGO-DataCloud MidnightBlue Release
The INDIGO PaaS core is built upon a set of services (exposing REST interfaces) that are:
Deployed
Scaled
Managed
Upgraded
Monitored
Self-healed
through Kubernetes ( an open source system for managing containerized applications across multiple hosts in a cluster.
September 2016
D.Salomoni - The INDIGO-DataCloud MidnightBlue Release

8. D.Salomoni - The INDIGO-DataCloud MidnightBlue Release
September 2016
D.Salomoni - The INDIGO-DataCloud MidnightBlue Release

9. D.Salomoni - The INDIGO-DataCloud MidnightBlue Release
September 2016
D.Salomoni - The INDIGO-DataCloud MidnightBlue Release

10. The AAI common glue, implemented across the entire INDIGO architecture
September 2016
D.Salomoni - The INDIGO-DataCloud MidnightBlue Release

11. D.Salomoni - The INDIGO-DataCloud Platform
AAI Features
INDIGO provides an advanced set of AAI features that includes:
User authentication (supporting SAML, OIDC, X.509)
Identity harmonization (link heterogeneous AuthN mechanisms to a single VO identity)
Management of VO membership (i.e., groups and other attributes)
Management of registration and enrolment flows
Provisioning of VO structure and membership information to services
Management, distribution and enforcement of authorization policies
A Token Translation Service (TTS), creating credentials for services that do not natively support OpenID Connect. Services that do not support OpenID Connect are for example ssh, S3 storage, OpenNebula.
July 20, Jinan Cloud School
D.Salomoni - The INDIGO-DataCloud Platform

12. D.Salomoni - The INDIGO-DataCloud MidnightBlue Release
September 2016
D.Salomoni - The INDIGO-DataCloud MidnightBlue Release

13. Managed Services Deployment and Applications Execution through Mesos
Mesos is able to manage cluster resources (cpu, mem) providing isolation and sharing across distributed applications (frameworks)
Marathon and Chronos are two powerful frameworks that can be deployed on top of a Mesos Cluster.
INDIGO PaaS uses:
Marathon to deploy, monitor and scale Long-Running services, ensuring that they are always up and running.
Chronos to run user applications (jobs), taking care of fetching input data, handling dependencies among jobs, rescheduling failed jobs.
September 2016
G. Donvito - The INDIGO-DataCloud MidnightBlue Release

14. Managed Services Deployment and Applications Execution through Mesos
Automatic deployment through Ansible recipes embedded in TOSCA and HOT templates
All the services run in docker containers;
High-availability of the cluster components:
Leader election among master nodes managed by Zookeeper;
HA Load-balancing;
Service discovery through Consul that provides also DNS functionality and health checks;
services are automatically registered in Consul as soon as they are deployed on the cluster
The external access to the deployed services is ensured through load-balancers in HA (unique entrypoint: cluster Virtual IP)
Cluster elasticity and application auto-scaling through CLUES plugin
September 2016
G. Donvito - The INDIGO-DataCloud MidnightBlue Release

15. Distributed Data access and transfer
FTS3 implements authorized data transfer from source to destination storage endpoints using various protocols via third party transfer.  Transfers are managed via command line client, a RESTful API or a web interface.
The functionality comprises:
Transfer auto-tuning/adaptive optimization
Endpoint-centric VO configuration
Transfer multi-hop
VO activity shares
Multiple replica support
RESTful interface
Retry of failed transfers
Staging files from archive
Support for Oracle and MySQL database back-ends
Transfer and access protocols support on top of gfal2 plugin mechanism (SRM, gridFTP, HTTP, xroot)
Session/connection reuse
September 2016
G. Donvito - The INDIGO-DataCloud MidnightBlue Release

16. Distributed Data access and transfer
Dynafed implements very fast loose coupling of storage endpoints as a single name-space exposed via HTTP and WebDAV. This allows to have federation of existing storage endpoints without the need of maintaining a file catalog for global to local file name  translation.
On the fly built name-space of merged meta- data items taken on demand from a number of remote and local storage endpoints.
September 2016
G. Donvito - The INDIGO-DataCloud MidnightBlue Release

17. D.Salomoni - The INDIGO-DataCloud Platform
TOSCA
Topology and Orchestration Specification for Cloud Applications
Standardizes the language to describe:
The structure of an IT service (its topology model)
How to orchestrate operational behavior (plans such as build, deploy, patch, shutdown, etc.)
Leveraging the BPMN (Business Process Model and Notation) standard
Declarative model that spans applications, virtual and physical infrastructures.
July 20, Jinan Cloud School
D.Salomoni - The INDIGO-DataCloud Platform

18. D.Salomoni - The INDIGO-DataCloud Platform
TOSCA in a nutshell
July 20, Jinan Cloud School
D.Salomoni - The INDIGO-DataCloud Platform

19. https://www.indigo-datacloud.eu Better Software for Better Science.
Thank you
Better Software for Better Science.
September 2016
D.Salomoni - The INDIGO-DataCloud MidnightBlue Release