Overview of the global architecture Giacinto DONVITO INFN-Bari

WP4 Computing Architecture Integrating distributed data infrastructures with INDIGO-DataCloud

WP4 Storage Architecture Integrating distributed data infrastructures with INDIGO-DataCloud

Integrating distributed data infrastructures with INDIGO-DataCloud WP4 Service WP4 COMPUTING WP4 STORAGE & NET Integrating distributed data infrastructures with INDIGO-DataCloud

Integrating distributed data infrastructures with INDIGO-DataCloud WP4 architecture Clear interfaces both for Computing and Storage: TOSCA directly provided by Heat/IM Will allow users exploiting TOSCA also without PaaS layer The capability to exploit Docker container natively when needed/possible A better scheduling system at the level of IaaS QoS Management Standard protocols for data access/transfers The main aim is to exploit solutions/functionalities that are available on both the supported IaaS Integrating distributed data infrastructures with INDIGO-DataCloud

Integrating distributed data infrastructures with INDIGO-DataCloud

Integrating distributed data infrastructures with INDIGO-DataCloud WP5 Service Service Name Kubernetes PaaS Orchestrator IAM Infrastructure Manager Monitoring Pillar QoS/SLA Accounting Service Rule Engine Mesos/Marathon Mesos/Chronos FTS Onedata Dynafed CLUES Many of those services will be available in the first release The architecture will allow the developers to improve their own services seamless. Integrating distributed data infrastructures with INDIGO-DataCloud

Other Science Gateways SG Mon Admin Portlets User Portlets Mobile Apps Ophidpia plugin Other Science Gateways GUI Clients SG Mon LONI plugin Data Analitics Workflow Portlets Open Mobile Toolkit Taverna, Kepler plugin Support services Future Gateway Portal Mobile clients Workflows Future Gateway REST API Future Gateway Engine JSAGA/JSAGA Adaptors IAM TOSCA Template Monitoring/ Information System Monitoring PaaS Orchestrator TOSCA Template repo QoS/SLA Accounting Policy Mgnt IM Data Services

Integrating distributed data infrastructures with INDIGO-DataCloud WP6 Service Service Name Future Gateway API Server JSAGA Adaptor for the TOSCA orchestrator Applications' "portlets" developed by WP6 INDIGO Kepler module/actors Open Mobile Toolkit Mobile Apps JSAGA with the resource manager API DataCloud-specific adaptors for the resource manager API Ophidia analytics workflow 'parallel' interfaces Ophidia analytics workflow 'massive' interfaces Token Translation Service Client Integrating distributed data infrastructures with INDIGO-DataCloud

Integrating distributed data infrastructures with INDIGO-DataCloud Before we start… We distinguish different basic use-cases: Authomated IaaS PaaS-like: Application execution Long Running Services Integrating distributed data infrastructures with INDIGO-DataCloud

Deploying of service/applications Home IdP The user, landing on the WP6 portal, can login using his/her own credential (Home IdP) IAM generates an INDIGO Token that is used in the next steps The user could chose a TOSCA Template for an already available repository WP6 layer (both portal and APIs) provide to the user the capability to customize the parameters on the TOSCA template Size, scalibility, data requirements, etc After a TOSCA “submission” WP6 will wait for a “call-back” from the orchestrator to be updated with the status of the teplate deployment IAM Future Gateway portal WP6 APIs TOSCA Template Tosca template repository PaaS Orchestrator Docker repository WP6 Service WP5 Service Integrating distributed data infrastructures with INDIGO-DataCloud WP4 Service

Deploying a “IaaS automated service” This means that the users are requiring a service that is based on an IaaS istance installed and customized by automatic scripts The user will manage the host/service by his own. This for example could be useful for having a dynamic cluster of batch resources. The scaling will be managed directly at the level of PaaS Orchestrator Integrating distributed data infrastructures with INDIGO-DataCloud

Deploying a IaaS Automated service WP6 APIs Future Gateway portal TOSCA Template Kubernetes Monitoring/ Information System Monitoring PaaS Orchestrator QoS/SLA IAM Accounting Policy Mgnt IM Data Services Cloud native APIs Modified TOSCA Template Integrating distributed data infrastructures with INDIGO-DataCloud Heat/IM

Deploying a IaaS Automated service WP6 APIs Future Gateway portal The PaaS orchestrator accept the TOSCA Template from WP6 layer PaaS Orch. Will ask all the µServices a pieces of information (SLA, Monitoring status, Rules, etc) in order to choose the best IaaS site. The data services are important in order to define the site. The Orchestrator could ask, for example, FTS to import data into Onedata As soon as the Orchestrator has the “IaaS name”, it provide a modified version of a TOSCA templates, to an Heat/IM endpoint. If the IaaS is not TOSCA compliant we may use IM at the PaaS level as a bridge TOSCA Template Kubernetes Data Services FTS PaaS Orchestrator IAM DynaFed Onedata Posix/ REST APIs WebDav Data Data Integrating distributed data infrastructures with INDIGO-DataCloud

Deploying a IaaS Automated service WP6 APIs Future Gateway portal TOSCA Template Data Services FTS DynaFed Onedata INDIGO PaaS Layer User Service INDIGO IaaS Integrating distributed data infrastructures with INDIGO-DataCloud

DATA Access/Management/Import The WP6 layer is able to exploit the APIs provided by WP5 storage layer to export: Onedata access FTS Data import/export WebDav access/browsing The data access/import will use: Web/WebDav/GFTP Dropbox-like ?? Posix (also in the Docker) The QoS Data management will exploit CDMI (WP5) CDMI/REST APIs (WP4) Integrating distributed data infrastructures with INDIGO-DataCloud

DATA Access/Management/Import WP6 APIs Future Gateway portal External repository REST APIs FTS WebDav CDMI (QoS) DynaFed Web/Posix/GFTP Onedata GFTP/ Posix/ REST APIs WebDav CDMI (Management) Posix (Access/Transfer) External repository WP6 Service WebDav WP5 Service Data Data Data Integrating distributed data infrastructures with INDIGO-DataCloud WP4 Service

Integrating distributed data infrastructures with INDIGO-DataCloud

Software as a Service Use Case Automated IaaS Site B Site A WN Data WN Data Data Analytics Services Interactive console Data Analytics Services WN Data WN Data TOSCA end-point TOSCA end-point External application AAI TOSCA Spec Scheduling VMI repo Container repo CIRMMP INAF-LBT CMCC – ENES ALGAE-BLOSSOM INGV - MOIST End user App dev INDIGO-DataCloud RIA-653549

Deploying a “PaaS Service” both LRS or Application execution Both of this use-case are implemented by means of Mesos (+ its frameworks) A Long Running Service (LRS) [Mesos + Marathon] Is a PaaS like service (MySQL, Apache Tomcat, PostgreSQL, etc) This is “managed” by the PaaS layer It is monitored, restarted, etc The elasticity could be provided automatically by the PaaS layer Application execution [Mesos + Chronos] The end-user ask for executing and application, deployed via Docker Providing information about the data to be pocessed This will help in deploying the application in the correct site. The Docker container are supported as a virtualization driver: Not as complex solution with orchestration features (Magnum or similar) The orchestration layer will be provided by WP5 in order to be portable on all the IaaS Integrating distributed data infrastructures with INDIGO-DataCloud

Deploying a “PaaS Service”: LRS or Application execution WP6 APIs Future Gateway portal TOSCA Template Kubernetes Monitoring/ Information System Monitoring PaaS Orchestrator IAM QoS/SLA Data Services Accounting Policy Mgnt IM Modified TOSCA Template Cloud native APIs Heat/IM Mesos/Marathon/Chronos Mesos/Marathon/Chronos Mesos Agent Mesos Agent Integrating distributed data infrastructures with INDIGO-DataCloud

Deploying a “PaaS Service”: LRS or Application execution WP6 APIs Onedata will enable the access of the data within the Docker containers So the users can access data directly from their code very transparently Dynafed could be used both with real remote access, via Parrot, or via fuse, depending on the use-cases. The cointainer will be automatically configured accordingly Future Gateway portal Clues is used for managing the scalability of the clusters: IaaS Automation and Mesos Clusters This is more “flexible” then automatic Heat scalability Will be IaaS indipendent Clues will interact directly with the orchestrator TOSCA Template CDMI/Web Kubernetes Data Services PaaS Orchestrator IAM Other PaaS core Services IM Modified TOSCA Template Cloud native APIs Heat/IM User Service Clues Mesos Agent Clues Mesos Agent Mesos/Marathon/Chronos Mesos/Marathon/Chronos Mesos Agent Mesos Agent Integrating distributed data infrastructures with INDIGO-DataCloud

Scientific Computational Portal “as a Service” PaaS Services Automated IaaS Storage Storage Site A Site B Use Case Portal / Access services LRMS WN LRMS WN LRMS WN LBaaS LRMS front-end VPNaaS TOSCA end-point Use Case Portal / Access services LRMS WN ??? TOSCA end-point ELIXIR Haddock CIRMMP FedCloud DARIAH INAF-LBT CMCC – ENES CTA ALGAE-BLOSSOM INGV - MOIST Site-level Scheduling AAI TOSCA Spec Scheduling VMI repo Container repo End user App dev INDIGO-DataCloud RIA-653549

… with the goals of building a platform ... Credit to German Moltò Integrating distributed data infrastructures with INDIGO-DataCloud