1. 1/17 Distributed Systems Architecture Research Group Universidad Complutense de Madrid Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula Constantino Vázquez Blanco Sun HPC Software Workshop Regensburg, Germany 7-10 September 2009

2. 2/17 Objectives Benefits of the latest OpenNebula version OpenNebula features for the different cloud types Compare OpenNebula with other similar technologies Ecosystem of the technology SGE Clusters on top of Hybrid Clouds using OpenNebula Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

3. 3/17 What is OpenNebula? Orchestrates storage, network and virtualization technologies to enable the dynamic placement of multi-tier services on distributed infrastructures, combining both data center resources and remote cloud resources, according to allocation policies Provides internal and Cloud administration and user interfaces for the full management of the IaaS Cloud platform Open-Source Toolkit for Building Cloud Infrastructures Private Cloud: Management of virtual infrastructure in the data- center or cluster Hybrid Cloud : Combination of private with Cloud resources Public Cloud: Cloud interfaces for the full management of services Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

4. 4/17 Benefits Faster response to infrastructure needs for services Centralized management of the virtual and physical distributed infrastructure Higher utilization of existing resources Operational saving with server consolidation Lower infrastructure expenses Infrastructure Manager Infrastructure User Faster delivery and scalability of services to meet dynamic demands Support for heterogeneous execution environments Full control of the lifecycle of virtualized services management System Integrators Fits into any existing data center Builds any type of Cloud deployment Open source software, Apache license Seamless integration with any product and service in the virtualization/cloud ecosystem and management tool in the data center Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

5. 5/17 Private Clouds Features – Private Clouds The infrastructure is owned and used by a single organization Private clouds enable a flexible and agile management of local infrastructure Not a new model, datacenter management has been around for a while Internal interfaces expose additional functionality for managing virtualized resources and controlling data center operation, not exposed by cloud interfaces Cloud interfaces may be also provided for users requiring higher abstraction Internal Interface Centralized management VM placement optimization Dynamic resizing and partitioning of the infrastructure Support for heterogeneous workloads Administration Domain Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

6. 6/17 Features – Private Clouds FeatureFunction Internal InterfaceUnix-like CLI for fully management of VM life-cycle and physical boxes XML-RPC API and libvirt virtualization API SchedulerRequirement/rank matchmaker allowing the definition of workload and resource-aware allocation policies Support for advance reservation of capacity through Haizea Virtualization Management Xen, KVM, and VMware Generic libvirt connector (VirtualBox planned for 1.4.2) Image ManagementGeneral mechanisms to transfer and clone VM images Network ManagementDefinition of isolated virtual networks to interconnect VMs Service Management and Contextualization Support for multi-tier services consisting of groups of inter-connected VMs, and their auto-configuration at boot time SecurityManagement of users by the infrastructure administrator Fault TolerancePersistent database backend to store host and VM information ScalabilityTested in the management of medium scale infrastructures with hundreds of servers and VMs (no scalability issues has been reported) InstallationInstallation on a UNIX cluster front-end without requiring new services Distributed in Ubuntu 9.04 (Jaunty Jackalope) Flexibility and Extensibility Open, flexible and extensible architecture, interfaces and components, allowing its integration with any product or tool Private Clouds Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

7. 7/17 Hybrid Clouds Features – Hybrid Clouds Extension of Private Clouds to combine private with public Cloud-based infrastructure to enable highly scalable hosting environments Access to remote cloud is fully transparent to private cloud users Hybrid Clouds enable the dynamic scaling of capacity to meet peak or fluctuating demands Remote Cloud Administration Domain Internal Interface Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

8. 8/17 Features – Hybrid Clouds FeatureFunction Cloud PluginsAmazon EC2 and ElasticHosts connectors FederationSupport for simultaneous access to several remote clouds FlexibilityModular approach to develop new connectors Hybrid Clouds Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

9. 9/17 Public Clouds Features – Public Clouds The infrastructure is owned by a single commercial organization and used by customers Public clouds enable the deployment of an entire IT infrastructure without the associated capital costs, paying only for the used capacity Cloud interface: Simple remote management of virtualized server instances Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula Remote Cloud Administration Domain Internal Interface Cloud Interface

10. 10/17 Features – Public Clouds FeatureFunction Cloud Interfaces for Users Implementation of a subset of the EC2 Query API Flexibility The Cloud Service allows the implementation of new Cloud interfaces (OCCI planned for 1.4.2) Public Clouds Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

11. 11/17 Comparison with Similar Technologies Platform ISFVMware VsphereEucalyptusNimbusOpenNebula Virtualization Management VMware, XenVMwareXen, KVMXen Xen, KVM, VMware Virtual Network Management Yes NoYes Image Management Yes Service Contextualizatio n No Yes SchedulingYes No Yes Administration Interface Yes No Yes Hybrid Cloud Computing No Yes Cloud InterfacesNovCloudEC2WSRF, EC2EC2 Query Flexibility and Extensibility YesNoYes Open SourceNo GPLApache Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

12. 12/17 RESERVOIR Flagship of Cloud Computing Research in FP7 Open source technology to enable deployment and management of complex IT services across different administrative domains Functionality for Service Management Definition Lifecycle Billing/accounting Elasticity/SLAs Resources and Services Virtualization without Barriers VM Focus on technologies that enable to build a federation of cooperating computing clouds A project driven by business use cases: SAP business application, Telco application, utility computing and eGov application 17-million and 3-year project partially funded by the European Commission Partners: IBM, Telefónica, UCL, Umea, SAP, Thales, SUN, Elsag Datamat, UCM, CETIC, University of Lugano, University of Messina, OGF.eeig. Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

13. 13/17 RESERVOIR RESERVOIR Architecture Service Provider Service Manager VEE Manager (OpenNebula) VEE Host Service Provider VEE Host VEE Manager VEE Host VHI VMI Infrastructure Providers Organize the placement of VEEs to meet optimization policies and constraints Support advanced new functionality for performance and relocation optimization Monitor service and enforce SLA compliance by managing number and capacity of service components (VEEs) Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

14. 14/17 Ecosystem Related Technologies Haizea (Uchicago): Open-source VM-based lease management architecture Several tools for service elasticity management, VM scheduling… being developed around OpeNebula in RESERVOIR (IBM, Telefonica I+D,…) Infrastructure Technology KVM Management Tools Xen Community Project Cloud Services Technology Partner of ElasticHosts Project in the Amazon EC2 Solutions Catalog Cloud Solutions and Tools Libvirt CLI and Desktop Applications Open-source Distributions Ubuntu 9.04 (Jaunty Jackalope) Chapter on Cloud Technologies in the Morfeo open-source community Standardization Bodies OGF Open Cloud Computing Interface Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

15. 15/17 Scaling SGE cluster in Hybrid Clouds Infrastructure Perspective Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

16. 16/17 Scaling SGE cluster in Hybrid Clouds Service Perspective Execution of SGE Clusters on top of Hybrid Clouds using OpenNebula

17. 17/17 THANK YOU FOR YOUR ATTENTION!!! More info, downloads, mailing lists at www.OpenNebula.org Acknowledgements OpenNebula is partially funded by the “RESERVOIR– Resources and Services Virtualization without Barriers” project EU grant agreement 215605 www.reservoir-fp7.eu/ Ignacio M. Llorente Rubén S. Montero Javier Fontán Rafael Moreno Scaling SGE cluster in Hybrid Clouds