EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI FedCloud Task Force State of the art and Future Plans EGI Technical Forum 2013 David Wallom Chair EGI Federated Cloud Associate Director, OeRC, University of Oxford

EGI-InSPIRE RI Demo Team Mattieu Puel Boris Parak Jan Meizner Björn Hagemeier Florian Feldhaus Piotr Kasprzak Mohammed Airaj Zeeshan Ali Shah Roberto Rosende Dopazo Emir Imamagic Alvaro Simon Michel Drescher Peter Solagna Alison Packer John Gordon Kostas Koumantaros Daniele Lezzi Elisabetta Ronchieri Ilja Livenson Viet Tran Sandor Acs Michel Jouvin Ivan Diaz Alvarez Owen Synge Andy Edmonds Matteo Turilli Stuart Pullinger Alvaro Lopez Salvatore Pinto

EGI-InSPIRE RI Where did we come from? Original Mandate: 18 months, September 2011 – March – Now a core EGI activity (TSA 2.6) Activities: divided into blocks of 6 months each. – M0 – M6 – Setup – M6 – M12 – Consolidation – M12 – M18 – Integration – M18 – M24 – Preproduction – M24 – M30 – ? Workgroups: – working units within the Task Force. – Investigate a specified set of capabilities for a federation of clouds. – Have leaders and collaborators. Workbenches: sections of the TF Wiki site collecting and documenting the activities of each workgroup. Meetings: weekly conference call. F2F every 3 months.

EGI-InSPIRE RI Task Force – Sept 2013 Members ~70 individuals ~35 institutions >13 countries Members ~70 individuals ~35 institutions >13 countries Stakeholders 23 Resource Providers 10 production 10 Technology Providers 8 User Communities 4 Liaisons Technologies OpenNebula. StratusLab. OpenStack. Synnefo. WNoDeS. PERUN SlipStream Technologies OpenNebula. StratusLab. OpenStack. Synnefo. WNoDeS. PERUN SlipStream BSC CNRS LMU OeRC Masaryk INFN-BARI IFAE Cyfronet 100%IT CESNET RADICAL SRCE DANTE FZJ GRNET GWDG STFC SARA KTH INFN- CNAF FCTSG EGI.eu Imperial CESGA CETA IFCA IGI IPHC IN2P3 SZTAKI IISAS SixSq CSC

EGI-InSPIRE RI EGI Task Force Federated Clouds Stakeholders 1.Resource Centres offer Cloud services 2.Technology Providers tender solutions 3.User communities Consume cloud services 6 scenarios = 6 key federated cloud capabilities 1.Manage VM instances 2.Data access/transfer interface 3.Cloud service information federation 4.Resource consumption management 5.Cloud service availability 6.Notification & Automation Resource Centre Cloud Management Technology Provider offer consume tender appoint 7.Federated AAI 8.VM Image Management 9.Brokering 10.Contextualisation 10 scenarios = 10 key federated cloud capabilities User Communities requirements

EGI-InSPIRE RI Federation Model

EGI-InSPIRE RI Set Up – March 2012 Resource Providers OpenNebula Resource Providers OpenNebula VM Mngmt OCCI 1.1 VM Mngmt OCCI 1.1 Information GLUE 2.0+ LDAP/BDII Information GLUE 2.0+ LDAP/BDII Monitoring Nagios Monitoring Nagios Message Bus Accounting OGF UR+ Accounting OGF UR+ AAI Login/Passwd AAI Login/Passwd curl Data CDMI 1.0 Data CDMI 1.0

EGI-InSPIRE RI Consolidation – September 2012 Resource Providers OpenNebula, OpenStack Resource Providers OpenNebula, OpenStack VM Mngmt OCCI 1.1 VM Mngmt OCCI 1.1 Information GLUE 2.0+ LDAP/BDII Information GLUE 2.0+ LDAP/BDII VM sharing Marketplace VM sharing Marketplace Message Bus AAI X509 AAI X509 Data CDMI 1.0 Data CDMI 1.0 Monitoring Nagios Monitoring Nagios Accounting OGF UR+ Accounting OGF UR+ curl

EGI-InSPIRE RI Integration – April 2013 Resource Providers OpenNebula, OpenStack, WNoDeS Resource Providers OpenNebula, OpenStack, WNoDeS VM Mngmt OCCI 1.1 VM Mngmt OCCI 1.1 Information GLUE 2.0+ LDAP/BDII Information GLUE 2.0+ LDAP/BDII VM sharing Marketplace VM sharing Marketplace Message Bus AAI X509 AAI X509 Data CDMI 1.0 Data CDMI 1.0 Monitoring Nagios Monitoring Nagios Accounting OGF UR+ Accounting OGF UR+ curl

EGI-InSPIRE RI PreProduction – September 2013 Resource Providers OpenNebula, OpenStack, StratusLab, WNoDeS, Syneffo Resource Providers OpenNebula, OpenStack, StratusLab, WNoDeS, Syneffo VM Mngmt OCCI Cloudinit VM Mngmt OCCI Cloudinit Information GLUE 2.0+ LDAP/BDII Information GLUE 2.0+ LDAP/BDII VM sharing Marketplace /AppDB VM sharing Marketplace /AppDB Message Bus AAI X509 AAI X509 Data CDMI 1.0 Data CDMI 1.0 Monitoring Nagios Monitoring Nagios Accounting OGF UR+ Accounting OGF UR+ curl

EGI-InSPIRE RI Federation testbed technologies and services, past FedCloud Core Services Information System Top-DBII Information System Top-DBII Monitoring Nagios + community probes Monitoring Nagios + community probes Image Metadata Marketplace Image Metadata Marketplace VO VOMS VO VOMS User Interfaces OCCI Clients rOCCI; WNoDeS-CLI OCCI Clients rOCCI; WNoDeS-CLI CDMI Clients Libcdmi-java CDMI Clients Libcdmi-java Resource Providers VOMS proxy Vmcatcher Vmcaster Vmcatcher Vmcaster Marketplace client SSM CDMI server LDAP OpenNebula OpenStack StratusLab WNoDeS OpenNebula OpenStack StratusLab WNoDeS OCCI server AAI VOMS Proxy EGI Core Infrastructure Service Availability SAM Service Availability SAM Accounting APEL Accounting APEL Configuration DB GOCDB Configuration DB GOCDB External Services Certification Authorities

EGI-InSPIRE RI Federation testbed technologies and services, present FedCloud Core Services Information System Top-DBII Information System Top-DBII Monitoring Nagios + community probes Monitoring Nagios + community probes Image Metadata Marketplace Image Metadata Marketplace Appliance Repository Appliance Repository VO PERUN server VO PERUN server User Interfaces OCCI Clients rOCCI; WNoDeS- CLI, Slipstream OCCI Clients rOCCI; WNoDeS- CLI, Slipstream CDMI Clients Libcdmi-java CDMI Clients Libcdmi-java Resource Providers VOMS proxy Vmcatcher Vmcaster Vmcatcher Vmcaster Marketplace client SSM CDMI server LDAP OpenNebula OpenStack StratusLab WNoDeS Syneffo OpenNebula OpenStack StratusLab WNoDeS Syneffo OCCI server AAI VOMS Proxy EGI Core Infrastructure Service Availability SAM Service Availability SAM Accounting APEL Accounting APEL Configuration DB GOCDB Configuration DB GOCDB External Services Certification Authorities

EGI-InSPIRE RI Federation testbed technologies and services, future FedCloud Core Services Information System Top-DBII Information System Top-DBII Monitoring Nagios + community probes Monitoring Nagios + community probes Image Metadata Marketplace Image Metadata Marketplace Appliance Repository Appliance Repository VO PERUN server VO PERUN server User Interfaces OCCI Clients rOCCI; WNoDeS-CLI SlipStream, VMDIRAC, CompatibleOne OCCI Clients rOCCI; WNoDeS-CLI SlipStream, VMDIRAC, CompatibleOne CDMI Clients Libcdmi-java CDMI Clients Libcdmi-java Resource Providers VOMS proxy Vmcatcher Vmcaster Vmcatcher Vmcaster Marketplace client SSM CDMI server LDAP OpenNebula OpenStack StratusLab WNoDeS Syneffo, others OpenNebula OpenStack StratusLab WNoDeS Syneffo, others OCCI server AAI VOMS Proxy EGI Core Infrastructure Service Availability SAM Service Availability SAM Accounting APEL Accounting APEL Configuration DB GOCDB Configuration DB GOCDB External Services Certification Authorities ID Management Federations

EGI-InSPIRE RI Federated Clouds Blueprint v1 Milestone for Task Force Activity Our ‘Federated Cloud Manual’ Documents –Background information on federation model –Open standards utilised –Core service integration –Scenario activities –How to connect for providers with currently known technology

EGI-InSPIRE RI Major Developments since April, 1 VM Management –Refactoring and redesign of rOCCI client and server Data Management –Deployment of multiple CDMI services within the fedcloud –Release of the first version of Stoxy server. –Release of the first version of the libcdmi-python and CLI tool –Integration work on OpenStack Keystone-based authentication. Information Discovery –Defined an extension of GLUE2 to describe the cloud resources in the EGI information system and publish them in the Top-BDII. –Integrated all the resource providers in the cloud top-bdii

EGI-InSPIRE RI Major Developments since April, 2 Accounting –Multiple technologies regularly publishing accounting data from different cloud infrastructures, –Agreed format for accounting messages - Moving towards standardisation, –Accounting data now visualised in the portal. Monitoring & Notification –Majority of RPs fully integrated: eu.egi.cloud.accounting: 9 endpoints eu.egi.cloud.information.bdii : 11 endpoints eu.egi.cloud.storage-management.cdmi: 4 endpoints eu.egi.cloud.vm-management.occi: 15 endpoints –OCCI probe extended to support OpenStack authentication and first draft of functional CDMI probe developed

EGI-InSPIRE RI Major Developments since April, 3 Federated AAI –Utilisation of PERUN for VO management with support from CESNET, –Integration of VOMS proxy authentication in most cloud platforms, –Allow for multiple authentication mechanisms to be supported by rOCCI-cli: username/password, X.509, X.509 via Keystone –Allow for membership in multiple VOs for OS Keystone based authentication VM Image Management –VMrecaster released. –VMcaster now publishes to the AppDb in beta. –VMcatcher has better interface to handlers in beta (held back till after demo even if risk is low)

EGI-InSPIRE RI Major Developments since April, 4 Brokering –Investigation of different available brokering technologies with each evaluated CompatibleOne VMDIRAC COMPSs SlipStream –Developing documentation for each of these to support users –Demonstration endpoints provided for COMPSs & VMDIRAC –In-depth testing of CompatibleOne at CESGA RP in progress –OCCI connector for Slipstream in development Contextualisation –Definition of new extension for OCCI allowing user-provided data passage to VM –Cloud-init defined as default contextualization mechanism. [However users are free to use any other mechanisms on their VMs with the data passed through OCCI] –The extension is implemented in rOCCI server and OCCI-OS. rOCCI client also supports it. The testbed is now ready to contextualize VMs. –Documentation is being updated.

EGI-InSPIRE RI Federated Clouds Blueprint v2 To do –Update Scenario activity –Update Technologies utilised –Update Provider How-to with feedback –Add User How-to

EGI-InSPIRE RI Task group Demonstrations Accounting Portal Integration of Contextualisation into OCCI through Cloudinit AppDB & VMCaster/Catcher integration

EGI-InSPIRE RI EU-Brasil OpenBio-BioVel – Ecology remote hosting of OpenModeller service. OpenModeller + Venus-C COMPS + FedCloud test bed. User Demonstrations – September 2013 ESA SlipStream web interface, with custom cluster deployment Instantiate an ESA configured Computing Cluster on the Federated Cloud Start processing jobs on the cluster

EGI-InSPIRE RI End of the first period as a task Towards production-grade cloud resources within the EGI infrastructure: – Continue transition into EGI operations. – Federated IaaS Cloud as a first class citizen alongside ‘grid/PaaS’ current infrastructure Biggest challenges: – Creation of ‘EGI Cloud Profile’ – stable standards to encourage new participants – Automation services – building the higher level services to match other providers – Data… Continued focus on use cases building on experiences with early adopters: – Support user communities as individual VOs rather than operating all within single ‘catch-all’ VO – Support for community specific user interfaces where integrated with standards – Infrastructure for tailored image management and support; Policy and operational developments – Security in collaboration with the EGI Security groups – SLA. and investigation of other provider models. Conclusions

EGI-InSPIRE RI Sept ‘13 -> Mar ‘14 PRODUCTION

EGI-InSPIRE RI Thank you Task Force resources Mailing List: Wiki site: GitHub: Indico site: