Overview of the global architecture Giacinto DONVITO INFN-Bari

Before we start… We distinguish different basic use-cases: Authomated IaaS PaaS-like: Application execution Long Running Services Integrating distributed data infrastructures with INDIGO-DataCloud 2

Deploying of service/applications Integrating distributed data infrastructures with INDIGO-DataCloud 3 WP6 Service WP5 Service WP4 Service IAM Future Gateway portal Home IdP Tosca template repository The user, landing on the WP6 portal, can login using his/her own credential (Home IdP) IAM generates an INDIGO Token that is used in the next steps The user could chose a TOSCA Template for an already available repository WP6 layer (both portal and APIs) provide to the user the capability to customize the parameters on the TOSCA template Size, scalibility, data requirements, etc After a TOSCA “submission” WP6 will wait for a “call-back” from the orchestrator to be updated with the status of the teplate deployment TOSCA Template Docker repository WP6 APIs PaaS Orchestrator

Deploying a “IaaS automated service” This means that the users are requiring a service that is based on an IaaS istance installed and customized by automatic scripts The user will manage the host/service by his own. This for example could be useful for having a dynamic cluster of batch resources. The scaling will be managed directly at the level of PaaS Orchestrator Integrating distributed data infrastructures with INDIGO-DataCloud 4

Deploying a IaaS Automated service Integrating distributed data infrastructures with INDIGO-DataCloud 5 Future Gateway portal TOSCA Template PaaS Orchestrator Kubernetes IAM Monitoring/ Information System Accounting Monitoring QoS/SLA Policy Mgnt Data Services IM Modified TOSCA Template Cloud native APIs Heat/IM WP6 APIs

Deploying a IaaS Automated service Integrating distributed data infrastructures with INDIGO-DataCloud 6 Future Gateway portal TOSCA Template PaaS Orchestrator Kubernetes IAM Data Services FTS DynaFed Onedata Data The PaaS orchestrator accept the TOSCA Template from WP6 layer PaaS Orch. Will ask all the µServices a pieces of information (SLA, Monitoring status, Rules, etc) in order to choose the best IaaS site. The data services are important in order to define the site. The Orchestrator could ask, for example, FTS to import data into Onedata As soon as the Orchestrator has the “IaaS name”, it provide a modified version of a TOSCA templates, to an Heat/IM endpoint. If the IaaS is not TOSCA compliant we may use IM at the PaaS level as a bridge Posix/ REST APIs WebDav WP6 APIs

Deploying a IaaS Automated service Integrating distributed data infrastructures with INDIGO-DataCloud 7 Future Gateway portal TOSCA Template INDIGO PaaS Layer User Service INDIGO IaaS Data Services FTS DynaFed Onedata WP6 APIs

DATA Access/Management/Import Integrating distributed data infrastructures with INDIGO-DataCloud 8 The WP6 layer is able to exploit the APIs provided by WP5 storage layer to export: Onedata access FTS Data import/export WebDav access/browsing The data access/import will use: Web/WebDav/GFTP Dropbox-like ?? Posix (also in the Docker) The QoS Data management will exploit CDMI (WP5) CDMI/REST APIs (WP4)

DATA Access/Management/Import Integrating distributed data infrastructures with INDIGO-DataCloud 9 Future Gateway portal FTS DynaFed Onedata Data GFTP/ Posix/ REST APIs WebDav External repository WP6 Service WP5 Service WP4 Service WebDav CDMI (QoS) Web/Posix/GFTP CDMI (Management) Posix (Access/Transfer) REST APIs WebDav WP6 APIs Data

Integrating distributed data infrastructures with INDIGO- DataCloud 10

Software as a Service Use Case Interactive console Data Analytics Services VMI repo Container repo Scheduling End userApp dev Site A TOSCA Spec Data AAI Data External application CIRMMP INAF-LBT CMCC – ENES ALGAE-BLOSSOM INGV - MOIST WN TOSCA end-point Data Analytics Services Site B Data WN TOSCA end-point INDIGO-DataCloud RIA Automated IaaS

Deploying a “PaaS Service” both LRS or Application execution Both of this use-case are implemented by means of Mesos (+ its frameworks) A Long Running Service (LRS) [Mesos + Marathon] Is a PaaS like service (MySQL, Apache Tomcat, PostgreSQL, etc) This is “managed” by the PaaS layer It is monitored, restarted, etc The elasticity could be provided automatically by the PaaS layer Application execution [Mesos + Chronos] The end-user ask for executing and application, deployed via Docker Providing information about the data to be pocessed This will help in deploying the application in the correct site. The Docker container are supported as a virtualization driver: Not as complex solution with orchestration features (Magnum or similar) The orchestration layer will be provided by WP5 in order to be portable on all the IaaS Integrating distributed data infrastructures with INDIGO-DataCloud 12

Integrating distributed data infrastructures with INDIGO-DataCloud 13 Future Gateway portal TOSCA Template PaaS Orchestrator Kubernetes IAM Monitoring/ Information System Accounting Monitoring QoS/SLA Policy Mgnt Data Services IM Modified TOSCA Template Cloud native APIs Heat/IM Mesos/Marathon /Chronos Mesos Agent Deploying a “PaaS Service”: LRS or Application execution Mesos/Marathon /Chronos WP6 APIs

Integrating distributed data infrastructures with INDIGO-DataCloud 14 Future Gateway portal TOSCA Template PaaS Orchestrator Kubernetes IAM Data Services IM Modified TOSCA Template Cloud native APIs Heat/IM Mesos/Marathon /Chronos Mesos Agent Deploying a “PaaS Service”: LRS or Application execution Mesos/Marathon /Chronos User Service CDMI/Web Clues Onedata will enable the access of the data within the Docker containers So the users can access data directly from their code very transparently Dynafed could be used both with real remote access, via Parrot, or via fuse, depending on the use-cases. The cointainer will be automatically configured accordingly WP6 APIs Mesos Agent Clues is used for managing the scalability of the clusters: IaaS Automation and Mesos Clusters This is more “flexible” then automatic Heat scalability Will be IaaS indipendent Clues will interact directly with the orchestrator Other PaaS core Services

Scientific Computational Portal “as a Service” Use Case Portal / Access services LBaaS VMI repo Container repo Scheduling End userApp dev Site A TOSCA Spec Storage AAI Use Case Portal / Access services LRMS front- end LRMS WN Site B LRMS WN Storage Site-level Scheduling TOSCA end- point VPNaaS ELIXIR Haddock CIRMMP FedCloud DARIAH INAF-LBT CMCC – ENES CTA ALGAE-BLOSSOM INGV - MOIST INDIGO-DataCloud RIA Automated IaaS PaaS Services ???

WAN bursting The main problem is the “overlay network” On top of what WP4 will be able to provide WP5 is investigating a SW solution for federated Mesos Clusters at geographical level A fist implementation of geographical batch cluster could exploit HTCondor where the network problem is solved at the “batch system level” As soon as a solution for the overlay network is usable: we may exploit inter-IaaS Mesos clusters Integrating distributed data infrastructures with INDIGO-DataCloud 16

WP4 architecture Clear interfaces both for Computing and Storage: TOSCA directly provided by Heat/IM Will allow users exploiting TOSCA also without PaaS layer The capability to exploit Docker container natively when needed/possible A better scheduling system at the level of IaaS QoS Management Standard protocols for data access/transfers The main aim is to exploit solutions/functionalities that are available on both the supported IaaS Integrating distributed data infrastructures with INDIGO-DataCloud 17

Open issues How to specify data on TOSCA Templates We need to monitor/index also Mesos Cluster deployed on the IaaS Final decision about CDMI vs REST at WP4 Level We need a Web Inteface for managing the QoS in a user-friendly way The grid job are supported by means of gateways? Not all the services have the same level of maturity … We should find, togheter with the user communities, the right steps for introducing more complexity/features as the services are ready... Integrating distributed data infrastructures with INDIGO-DataCloud 18

… with the goals of building a platform... Integrating distributed data infrastructures with INDIGO-DataCloud 19 Credit to German Moltò