1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 1 EGI FedCloud Task Force State of the art and Future Plans EGI Technical Forum 2013 David Wallom Chair EGI Federated Cloud Associate Director, OeRC, University of Oxford david.wallom@oerc.ox.ac.uk

2. www.egi.eu EGI-InSPIRE RI-261323 Demo Team Mattieu Puel Boris Parak Jan Meizner Björn Hagemeier Florian Feldhaus Piotr Kasprzak Mohammed Airaj Zeeshan Ali Shah Roberto Rosende Dopazo Emir Imamagic Alvaro Simon Michel Drescher Peter Solagna Alison Packer John Gordon Kostas Koumantaros Daniele Lezzi Elisabetta Ronchieri Ilja Livenson Viet Tran Sandor Acs Michel Jouvin Ivan Diaz Alvarez Owen Synge Andy Edmonds Matteo Turilli Stuart Pullinger Alvaro Lopez Salvatore Pinto

3. www.egi.eu EGI-InSPIRE RI-261323 Where did we come from? Original Mandate: 18 months, September 2011 – March 2013. – Now a core EGI activity (TSA 2.6) Activities: divided into blocks of 6 months each. – M0 – M6 – Setup – M6 – M12 – Consolidation – M12 – M18 – Integration – M18 – M24 – Preproduction – M24 – M30 – ? Workgroups: – working units within the Task Force. – Investigate a specified set of capabilities for a federation of clouds. – Have leaders and collaborators. Workbenches: sections of the TF Wiki site collecting and documenting the activities of each workgroup. Meetings: weekly conference call. F2F every 3 months. https://wiki.egi.eu/wiki/Fedcloud-tf

4. www.egi.eu EGI-InSPIRE RI-261323 Task Force – Sept 2013 Members ~70 individuals ~35 institutions >13 countries Members ~70 individuals ~35 institutions >13 countries Stakeholders 23 Resource Providers 10 production 10 Technology Providers 8 User Communities 4 Liaisons Technologies OpenNebula. StratusLab. OpenStack. Synnefo. WNoDeS. PERUN SlipStream Technologies OpenNebula. StratusLab. OpenStack. Synnefo. WNoDeS. PERUN SlipStream BSC CNRS LMU OeRC Masaryk INFN-BARI IFAE Cyfronet 100%IT CESNET RADICAL SRCE DANTE FZJ GRNET GWDG STFC SARA KTH INFN- CNAF FCTSG EGI.eu Imperial CESGA CETA IFCA IGI IPHC IN2P3 SZTAKI IISAS SixSq CSC

5. www.egi.eu EGI-InSPIRE RI-261323 EGI Task Force Federated Clouds Stakeholders 1.Resource Centres offer Cloud services 2.Technology Providers tender solutions 3.User communities Consume cloud services 6 scenarios = 6 key federated cloud capabilities 1.Manage VM instances 2.Data access/transfer interface 3.Cloud service information federation 4.Resource consumption management 5.Cloud service availability 6.Notification & Automation Resource Centre Cloud Management Technology Provider offer consume tender appoint 7.Federated AAI 8.VM Image Management 9.Brokering 10.Contextualisation 10 scenarios = 10 key federated cloud capabilities User Communities requirements

6. www.egi.eu EGI-InSPIRE RI-261323 Federation Model

7. www.egi.eu EGI-InSPIRE RI-261323 Set Up – March 2012 Resource Providers OpenNebula Resource Providers OpenNebula VM Mngmt OCCI 1.1 VM Mngmt OCCI 1.1 Information GLUE 2.0+ LDAP/BDII Information GLUE 2.0+ LDAP/BDII Monitoring Nagios Monitoring Nagios Message Bus Accounting OGF UR+ Accounting OGF UR+ AAI Login/Passwd AAI Login/Passwd curl Data CDMI 1.0 Data CDMI 1.0

8. www.egi.eu EGI-InSPIRE RI-261323 Consolidation – September 2012 Resource Providers OpenNebula, OpenStack Resource Providers OpenNebula, OpenStack VM Mngmt OCCI 1.1 VM Mngmt OCCI 1.1 Information GLUE 2.0+ LDAP/BDII Information GLUE 2.0+ LDAP/BDII VM sharing Marketplace VM sharing Marketplace Message Bus AAI X509 AAI X509 Data CDMI 1.0 Data CDMI 1.0 Monitoring Nagios Monitoring Nagios Accounting OGF UR+ Accounting OGF UR+ curl

9. www.egi.eu EGI-InSPIRE RI-261323 Integration – April 2013 Resource Providers OpenNebula, OpenStack, WNoDeS Resource Providers OpenNebula, OpenStack, WNoDeS VM Mngmt OCCI 1.1 VM Mngmt OCCI 1.1 Information GLUE 2.0+ LDAP/BDII Information GLUE 2.0+ LDAP/BDII VM sharing Marketplace VM sharing Marketplace Message Bus AAI X509 AAI X509 Data CDMI 1.0 Data CDMI 1.0 Monitoring Nagios Monitoring Nagios Accounting OGF UR+ Accounting OGF UR+ curl

10. www.egi.eu EGI-InSPIRE RI-261323 PreProduction – September 2013 Resource Providers OpenNebula, OpenStack, StratusLab, WNoDeS, Syneffo Resource Providers OpenNebula, OpenStack, StratusLab, WNoDeS, Syneffo VM Mngmt OCCI 1.1 + Cloudinit VM Mngmt OCCI 1.1 + Cloudinit Information GLUE 2.0+ LDAP/BDII Information GLUE 2.0+ LDAP/BDII VM sharing Marketplace /AppDB VM sharing Marketplace /AppDB Message Bus AAI X509 AAI X509 Data CDMI 1.0 Data CDMI 1.0 Monitoring Nagios Monitoring Nagios Accounting OGF UR+ Accounting OGF UR+ curl

11. www.egi.eu EGI-InSPIRE RI-261323 Federation testbed technologies and services, past FedCloud Core Services Information System Top-DBII Information System Top-DBII Monitoring Nagios + community probes Monitoring Nagios + community probes Image Metadata Marketplace Image Metadata Marketplace VO VOMS VO VOMS User Interfaces OCCI Clients rOCCI; WNoDeS-CLI OCCI Clients rOCCI; WNoDeS-CLI CDMI Clients Libcdmi-java CDMI Clients Libcdmi-java Resource Providers VOMS proxy Vmcatcher Vmcaster Vmcatcher Vmcaster Marketplace client SSM CDMI server LDAP OpenNebula OpenStack StratusLab WNoDeS OpenNebula OpenStack StratusLab WNoDeS OCCI server AAI VOMS Proxy EGI Core Infrastructure Service Availability SAM Service Availability SAM Accounting APEL Accounting APEL Configuration DB GOCDB Configuration DB GOCDB External Services Certification Authorities

12. www.egi.eu EGI-InSPIRE RI-261323 Federation testbed technologies and services, present FedCloud Core Services Information System Top-DBII Information System Top-DBII Monitoring Nagios + community probes Monitoring Nagios + community probes Image Metadata Marketplace Image Metadata Marketplace Appliance Repository Appliance Repository VO PERUN server VO PERUN server User Interfaces OCCI Clients rOCCI; WNoDeS- CLI, Slipstream OCCI Clients rOCCI; WNoDeS- CLI, Slipstream CDMI Clients Libcdmi-java CDMI Clients Libcdmi-java Resource Providers VOMS proxy Vmcatcher Vmcaster Vmcatcher Vmcaster Marketplace client SSM CDMI server LDAP OpenNebula OpenStack StratusLab WNoDeS Syneffo OpenNebula OpenStack StratusLab WNoDeS Syneffo OCCI server AAI VOMS Proxy EGI Core Infrastructure Service Availability SAM Service Availability SAM Accounting APEL Accounting APEL Configuration DB GOCDB Configuration DB GOCDB External Services Certification Authorities

13. www.egi.eu EGI-InSPIRE RI-261323 Federation testbed technologies and services, future FedCloud Core Services Information System Top-DBII Information System Top-DBII Monitoring Nagios + community probes Monitoring Nagios + community probes Image Metadata Marketplace Image Metadata Marketplace Appliance Repository Appliance Repository VO PERUN server VO PERUN server User Interfaces OCCI Clients rOCCI; WNoDeS-CLI SlipStream, VMDIRAC, CompatibleOne OCCI Clients rOCCI; WNoDeS-CLI SlipStream, VMDIRAC, CompatibleOne CDMI Clients Libcdmi-java CDMI Clients Libcdmi-java Resource Providers VOMS proxy Vmcatcher Vmcaster Vmcatcher Vmcaster Marketplace client SSM CDMI server LDAP OpenNebula OpenStack StratusLab WNoDeS Syneffo, others OpenNebula OpenStack StratusLab WNoDeS Syneffo, others OCCI server AAI VOMS Proxy EGI Core Infrastructure Service Availability SAM Service Availability SAM Accounting APEL Accounting APEL Configuration DB GOCDB Configuration DB GOCDB External Services Certification Authorities ID Management Federations

14. www.egi.eu EGI-InSPIRE RI-261323 Federated Clouds Blueprint v1 Milestone for Task Force Activity Our ‘Federated Cloud Manual’ Documents –Background information on federation model –Open standards utilised –Core service integration –Scenario activities –How to connect for providers with currently known technology https://documents.egi.eu/public/ShowDocument?docid=1773

15. www.egi.eu EGI-InSPIRE RI-261323 Major Developments since April, 1 VM Management –Refactoring and redesign of rOCCI client and server Data Management –Deployment of multiple CDMI services within the fedcloud –Release of the first version of Stoxy server. –Release of the first version of the libcdmi-python and CLI tool –Integration work on OpenStack Keystone-based authentication. Information Discovery –Defined an extension of GLUE2 to describe the cloud resources in the EGI information system and publish them in the Top-BDII. –Integrated all the resource providers in the cloud top-bdii

16. www.egi.eu EGI-InSPIRE RI-261323 Major Developments since April, 2 Accounting –Multiple technologies regularly publishing accounting data from different cloud infrastructures, –Agreed format for accounting messages - Moving towards standardisation, –Accounting data now visualised in the portal. Monitoring & Notification –Majority of RPs fully integrated: eu.egi.cloud.accounting: 9 endpoints eu.egi.cloud.information.bdii : 11 endpoints eu.egi.cloud.storage-management.cdmi: 4 endpoints eu.egi.cloud.vm-management.occi: 15 endpoints –OCCI probe extended to support OpenStack authentication and first draft of functional CDMI probe developed

17. www.egi.eu EGI-InSPIRE RI-261323 Major Developments since April, 3 Federated AAI –Utilisation of PERUN for VO management with support from CESNET, –Integration of VOMS proxy authentication in most cloud platforms, –Allow for multiple authentication mechanisms to be supported by rOCCI-cli: username/password, X.509, X.509 via Keystone –Allow for membership in multiple VOs for OS Keystone based authentication VM Image Management –VMrecaster released. –VMcaster now publishes to the AppDb in beta. –VMcatcher has better interface to handlers in beta (held back till after demo even if risk is low)

18. www.egi.eu EGI-InSPIRE RI-261323 Major Developments since April, 4 Brokering –Investigation of different available brokering technologies with each evaluated CompatibleOne VMDIRAC COMPSs SlipStream –Developing documentation for each of these to support users –Demonstration endpoints provided for COMPSs & VMDIRAC –In-depth testing of CompatibleOne at CESGA RP in progress –OCCI connector for Slipstream in development Contextualisation –Definition of new extension for OCCI allowing user-provided data passage to VM –Cloud-init defined as default contextualization mechanism. [However users are free to use any other mechanisms on their VMs with the data passed through OCCI] –The extension is implemented in rOCCI server and OCCI-OS. rOCCI client also supports it. The testbed is now ready to contextualize VMs. –Documentation is being updated.

19. www.egi.eu EGI-InSPIRE RI-261323 Federated Clouds Blueprint v2 To do –Update Scenario activity –Update Technologies utilised –Update Provider How-to with feedback –Add User How-to

20. www.egi.eu EGI-InSPIRE RI-261323 Task group Demonstrations Accounting Portal Integration of Contextualisation into OCCI through Cloudinit AppDB & VMCaster/Catcher integration

21. www.egi.eu EGI-InSPIRE RI-261323 EU-Brasil OpenBio-BioVel – Ecology remote hosting of OpenModeller service. OpenModeller + Venus-C COMPS + FedCloud test bed. User Demonstrations – September 2013 ESA SlipStream web interface, with custom cluster deployment Instantiate an ESA configured Computing Cluster on the Federated Cloud Start processing jobs on the cluster

22. www.egi.eu EGI-InSPIRE RI-261323 End of the first period as a task Towards production-grade cloud resources within the EGI infrastructure: – Continue transition into EGI operations. – Federated IaaS Cloud as a first class citizen alongside ‘grid/PaaS’ current infrastructure Biggest challenges: – Creation of ‘EGI Cloud Profile’ – stable standards to encourage new participants – Automation services – building the higher level services to match other providers – Data… Continued focus on use cases building on experiences with early adopters: – Support user communities as individual VOs rather than operating all within single ‘catch-all’ VO – Support for community specific user interfaces where integrated with standards – Infrastructure for tailored image management and support; Policy and operational developments – Security in collaboration with the EGI Security groups – SLA. and investigation of other provider models. Conclusions

23. www.egi.eu EGI-InSPIRE RI-261323 Sept ‘13 -> Mar ‘14 PRODUCTION

24. www.egi.eu EGI-InSPIRE RI-261323 Thank you Task Force resources Mailing List: fedcloud-tf@mailman.egi.eu Wiki site: http://go.egi.eu/tf-fedclouds GitHub: https://github.com/EGI-FCTF Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=56