1. Overview of the global architecture
Giacinto DONVITO
INFN-Bari

2. WP4 Computing Architecture
Integrating distributed data infrastructures with INDIGO-DataCloud

3. WP4 Storage Architecture
Integrating distributed data infrastructures with INDIGO-DataCloud

4. Integrating distributed data infrastructures with INDIGO-DataCloud
WP4 Service
WP4 COMPUTING
WP4 STORAGE & NET
Integrating distributed data infrastructures with INDIGO-DataCloud

5. Integrating distributed data infrastructures with INDIGO-DataCloud
WP4 architecture
Clear interfaces both for Computing and Storage:
TOSCA directly provided by Heat/IM
Will allow users exploiting TOSCA also without PaaS layer
The capability to exploit Docker container natively when needed/possible
A better scheduling system at the level of IaaS
QoS Management
Standard protocols for data access/transfers
The main aim is to exploit solutions/functionalities that are available on both the supported IaaS
Integrating distributed data infrastructures with INDIGO-DataCloud

6. Integrating distributed data infrastructures with INDIGO-DataCloud

7. Integrating distributed data infrastructures with INDIGO-DataCloud
WP5 Service
Service Name
Kubernetes
PaaS Orchestrator
IAM
Infrastructure Manager
Monitoring Pillar
QoS/SLA
Accounting Service
Rule Engine
Mesos/Marathon
Mesos/Chronos
FTS
Onedata
Dynafed
CLUES
Many of those services will be available in the first release
The architecture will allow the developers to improve their own services seamless.
Integrating distributed data infrastructures with INDIGO-DataCloud

8. Other Science Gateways SG Mon
Admin Portlets
User Portlets
Mobile Apps
Ophidpia plugin
Other Science Gateways
GUI
Clients
SG Mon
LONI plugin
Data Analitics
Workflow Portlets
Open Mobile Toolkit
Taverna, Kepler plugin
Support services
Future Gateway Portal
Mobile clients
Workflows
Future Gateway REST API
Future Gateway Engine
JSAGA/JSAGA Adaptors
IAM
TOSCA
Template
Monitoring/ Information System
Monitoring
PaaS Orchestrator
TOSCA Template repo
QoS/SLA
Accounting
Policy Mgnt IM
Data Services

9. Integrating distributed data infrastructures with INDIGO-DataCloud
WP6 Service
Service Name
Future Gateway API Server
JSAGA Adaptor for the TOSCA orchestrator
Applications' "portlets" developed by WP6
INDIGO Kepler module/actors
Open Mobile Toolkit
Mobile Apps
JSAGA with the resource manager API
DataCloud-specific adaptors for the resource manager API
Ophidia analytics workflow 'parallel' interfaces
Ophidia analytics workflow 'massive' interfaces
Token Translation Service Client
Integrating distributed data infrastructures with INDIGO-DataCloud

10. Integrating distributed data infrastructures with INDIGO-DataCloud
Before we start…
We distinguish different basic use-cases:
Authomated IaaS
PaaS-like:
Application execution
Long Running Services
Integrating distributed data infrastructures with INDIGO-DataCloud

11. Deploying of service/applications
Home IdP
The user, landing on the WP6 portal, can login using his/her own credential (Home IdP)
IAM generates an INDIGO Token that is used in the next steps
The user could chose a TOSCA Template for an already available repository
WP6 layer (both portal and APIs) provide to the user the capability to customize the parameters on the TOSCA template
Size, scalibility, data requirements, etc
After a TOSCA “submission” WP6 will wait for a “call-back” from the orchestrator to be updated with the status of the teplate deployment
IAM
Future Gateway portal
WP6 APIs
TOSCA
Template
Tosca template repository
PaaS Orchestrator
Docker
repository
WP6 Service
WP5 Service
Integrating distributed data infrastructures with INDIGO-DataCloud
WP4 Service

12. Deploying a “IaaS automated service”
This means that the users are requiring a service that is based on an IaaS istance installed and customized by automatic scripts
The user will manage the host/service by his own.
This for example could be useful for having a dynamic cluster of batch resources.
The scaling will be managed directly at the level of PaaS Orchestrator
Integrating distributed data infrastructures with INDIGO-DataCloud

13. Deploying a IaaS Automated service
WP6 APIs
Future Gateway portal
TOSCA
Template
Kubernetes
Monitoring/ Information System
Monitoring
PaaS Orchestrator
QoS/SLA
IAM
Accounting
Policy Mgnt IM
Data Services
Cloud native APIs
Modified
TOSCA
Template
Integrating distributed data infrastructures with INDIGO-DataCloud
Heat/IM

14. Deploying a IaaS Automated service
WP6 APIs
Future Gateway portal
The PaaS orchestrator accept the TOSCA Template from WP6 layer
PaaS Orch. Will ask all the µServices a pieces of information (SLA, Monitoring status, Rules, etc) in order to choose the best IaaS site.
The data services are important in order to define the site.
The Orchestrator could ask, for example, FTS to import data into Onedata
As soon as the Orchestrator has the “IaaS name”, it provide a modified version of a TOSCA templates, to an Heat/IM endpoint.
If the IaaS is not TOSCA compliant we may use IM at the PaaS level as a bridge
TOSCA
Template
Kubernetes
Data Services
FTS
PaaS Orchestrator
IAM
DynaFed
Onedata
Posix/
REST APIs
WebDav
Data
Data
Integrating distributed data infrastructures with INDIGO-DataCloud

15. Deploying a IaaS Automated service
WP6 APIs
Future Gateway portal
TOSCA
Template
Data Services
FTS
DynaFed
Onedata
INDIGO PaaS Layer
User Service
INDIGO IaaS
Integrating distributed data infrastructures with INDIGO-DataCloud

16. DATA Access/Management/Import
The WP6 layer is able to exploit the APIs provided by WP5 storage layer to export:
Onedata access
FTS Data import/export
WebDav access/browsing
The data access/import will use:
Web/WebDav/GFTP
Dropbox-like ??
Posix (also in the Docker)
The QoS Data management will exploit
CDMI (WP5)
CDMI/REST APIs (WP4)
Integrating distributed data infrastructures with INDIGO-DataCloud

17. DATA Access/Management/Import
WP6 APIs
Future Gateway portal
External repository
REST APIs
FTS
WebDav
CDMI (QoS)
DynaFed
Web/Posix/GFTP
Onedata
GFTP/
Posix/
REST APIs
WebDav
CDMI (Management)
Posix (Access/Transfer)
External repository
WP6 Service
WebDav
WP5 Service
Data
Data
Data
Integrating distributed data infrastructures with INDIGO-DataCloud
WP4 Service

18. Integrating distributed data infrastructures with INDIGO-DataCloud

19. Software as a Service Use Case
Automated IaaS
Site B
Site A WN
Data WN
Data
Data Analytics
Services
Interactive console
Data Analytics
Services WN
Data WN
Data
TOSCA end-point
TOSCA end-point
External application
AAI
TOSCA
Spec
Scheduling
VMI repo
Container repo
CIRMMP
INAF-LBT
CMCC – ENES
ALGAE-BLOSSOM
INGV - MOIST
End user
App dev
INDIGO-DataCloud RIA

20. Deploying a “PaaS Service” both LRS or Application execution
Both of this use-case are implemented by means of Mesos (+ its frameworks)
A Long Running Service (LRS) [Mesos + Marathon]
Is a PaaS like service (MySQL, Apache Tomcat, PostgreSQL, etc)
This is “managed” by the PaaS layer
It is monitored, restarted, etc
The elasticity could be provided automatically by the PaaS layer
Application execution [Mesos + Chronos]
The end-user ask for executing and application, deployed via Docker
Providing information about the data to be pocessed
This will help in deploying the application in the correct site.
The Docker container are supported as a virtualization driver:
Not as complex solution with orchestration features (Magnum or similar)
The orchestration layer will be provided by WP5 in order to be portable on all the IaaS
Integrating distributed data infrastructures with INDIGO-DataCloud

21. Deploying a “PaaS Service”: LRS or Application execution
WP6 APIs
Future Gateway portal
TOSCA
Template
Kubernetes
Monitoring/ Information System
Monitoring
PaaS Orchestrator
IAM
QoS/SLA
Data Services
Accounting
Policy Mgnt IM
Modified
TOSCA Template
Cloud native APIs
Heat/IM
Mesos/Marathon/Chronos
Mesos/Marathon/Chronos
Mesos Agent
Mesos Agent
Integrating distributed data infrastructures with INDIGO-DataCloud

22. Deploying a “PaaS Service”: LRS or Application execution
WP6 APIs
Onedata will enable the access of the data within the Docker containers
So the users can access data directly from their code very transparently
Dynafed could be used both with real remote access, via Parrot, or via fuse, depending on the use-cases.
The cointainer will be automatically configured accordingly
Future Gateway portal
Clues is used for managing the scalability of the clusters:
IaaS Automation and Mesos Clusters
This is more “flexible” then automatic Heat scalability
Will be IaaS indipendent
Clues will interact directly with the orchestrator
TOSCA
Template
CDMI/Web
Kubernetes
Data Services
PaaS Orchestrator
IAM
Other PaaS core Services IM
Modified
TOSCA Template
Cloud native APIs
Heat/IM
User Service
Clues
Mesos Agent
Clues
Mesos Agent
Mesos/Marathon/Chronos
Mesos/Marathon/Chronos
Mesos Agent
Mesos Agent
Integrating distributed data infrastructures with INDIGO-DataCloud

23. Scientific Computational Portal “as a Service”
PaaS Services
Automated IaaS
Storage
Storage
Site A
Site B
Use Case Portal / Access services
LRMS WN
LRMS WN
LRMS WN
LBaaS
LRMS front-end
VPNaaS
TOSCA end-point
Use Case Portal / Access services
LRMS WN
???
TOSCA end-point
ELIXIR
Haddock
CIRMMP
FedCloud
DARIAH
INAF-LBT
CMCC – ENES
CTA
ALGAE-BLOSSOM
INGV - MOIST
Site-level Scheduling
AAI
TOSCA
Spec
Scheduling
VMI repo
Container repo
End user
App dev
INDIGO-DataCloud RIA

24. … with the goals of building a platform ...
Credit to German Moltò
Integrating distributed data infrastructures with INDIGO-DataCloud