Apache CloudStack An Introduction Kevin Kluge Apache CloudStack Committer Elasticsearch VP Engineering

Use CloudStack to build IaaS clouds (like EC2) What does CloudStack do? What is it? Does not do PaaS or SaaS but those can run on it. Does not connect to Amazon. Use CloudStack to build IaaS clouds (like EC2) Create VMs, disks, networks, network services Self service Meter usage Written in Java Scalable Many vendor integrations Native and EC2 API

How did Amazon build EC2? Amazon eCommerce Platform AWS API (EC2, S3, …) Amazon Orchestration Software Open Source Xen Hypervisor Networking Commodity Servers Commodity Storage

How can you build your cloud? Amazon eCommerce Platform Your Portal (Optional) AWS API (EC2, S3, …) CloudStack or AWS API Amazon Orchestration Software CloudStack Orchestration Software Open Source Xen Hypervisor ESXi, KVM, XenServer/XCP Networking Servers Storage

Project history 2008/2009: closed-source development via Cloud.com First deployments in late 2009 May 2010: ~98% open source as GPLv3 (open core) August 2011: 100% open source GPLv3 April 2012: Switch to Apache License v2 Submit code to Apache Software Foundation March 2013: Graduate to Apache TLP

Project current state TLP within Apache Software Foundation Apache CloudStack 4.0 and 4.1 delivered 4.2 under development; target August release Uses ASF infra for bugs, wiki, mailing lists ~50% non-Citrix committers and PMC members Across contributors Citrix percentage much lower

Messages = Posts Source: http://www.qyjohn.net/?p=3321

Source: http://www.qyjohn.net/?p=3321

Workloads in the Cloud

Different workloads have different needs Clouds can support both types of workloads But architecture to deliver SLA is different Different workloads have different needs Amazon-style Workload Traditional Workload Design for failure Expect reliability Ephemeral resources Back-up everything Multi-site redundancy HA, Fault tolerance Self-service recovery Admin controls recovery

CloudStack supports both workload types Amazon-style Workloads Traditional Workloads CloudStack Mgmt Server Traditional Zone vSphere Cloud-era Availability Zone Cloud-era Availability Zone Cloud-era Availability Zone Enterprise Networking (e.g., VLAN) ESXi Cluster ESXi Cluster ESXi Cluster Object Storage Enterprise Storage (e.g., SAN)

Object store is critical for Cloud-era workloads Amazon-Style Region CloudStack Mgmt. Server Workloads are distributed across availability zones No guarantee on zone reliability DBs and Templates snapped to object store. For small failures, recreate instance in same zone For DR, recreate instance in different zone Dramatically less expensive Availability Zone Availability Zone Availability Zone Object Store

Features

Open platform for compute, storage, network XCP/XS VMware KVM Oracle VM Bare metal Hypervisor Storage Local Disk iSCSI NFS Fiber Channel Object Stores Block & Object Network Network Type Isolation Load balancer Firewall VPN Network & Network Services

Virtual machine management Users Start Stop Restart Destroy VM Operations Console Access CPU Utilized Network Read Network Writes VM Status Change Service Offering 2 CPUs 1 GB RAM 20 GB 20 Mbps 4 CPUs 4 GB RAM 200 GB 100 Mbps

Volume and snapshot management Add / Delete Volumes VM 1 Volume Create Templates from Volumes Volume Template Schedule Snapshots Hourly Daily Weekly Monthly Now View Snapshot History 12/2/2012 7.30 am …. 2/2/2012 7.30 am

Multi-tenancy and account management Admin Org A Reseller A Domain VMs, IPs, Snapshots… Resources Domain is a unit of isolation that represents a customer org, business unit or a reseller Domain can have arbitrary levels of sub-domains A Domain can have one or more accounts An Account represents one or more users and is the basic unit of isolation Admin can limit resources at the Account or Domain levels Admin Org C Sub-Domain Group B Account Group A User 1 User 2

Service offerings Specify Resource Levels Configure Properties Define Scope CPU Cores CPU (MHz) Memory (MB) Name Compute Custom Disk Size Disk Size (GB) Storage Tag Public Name Disk Network Rate Redundant VR Network Firewall Load balancer CPU Cap Host Tag Enable HA Public

Network offering Provides cloud operator defined service features Isolation Load Balancing VPN Firewall Supports Physical Devices NetScaler F5 BIG-IP Juniper SRX Zone 1 Zone N Pod 1 Pod N Pod 1

Network and network services Create Networks and attach VMs Acquire public IP address for NAT & load balancing Control traffic to VM using ingress and egress firewall rules Set up rules to load balance traffic between VMs

Layer-3 guest network Network Services Managed Externally Network Services Managed by CS Public Network 65.11.0.0/16 Security Group 1 Security Group 1 Public Network/Internet 65.11.1.2 65.11.1.2 Guest VM 1 Guest VM 1 65.11.1.3 Guest VM 2 Physical Load Balancer 65.11.1.3 Guest VM 2 EIP, ELB 65.11.1.4 65.11.1.4 Guest VM 3 Guest VM 3 65.11.1.5 65.11.1.5 Guest VM 4 Guest VM 4 CS Virtual Router CS Virtual Router Security Group 2 DHCP, DNS Security Group 2 DHCP, DNS

Layer-2 guest virtual network CS Virtual Router provides Network Services External Devices provide Network Services Guest Virtual Network 10.0.0.0/8 VLAN 100 Guest Virtual Network 10.0.0.0/8 VLAN 100 Public Network/Internet Public Network/Internet 10.1.1.1 Guest VM 1 Public IP 6.37.1.12 Private IP 10.1.1.111 10.1.1.1 Guest VM 1 Juniper SRX Firewall CS Virtual Router Gateway address 10.1.1.1 6.37..1.11 10.1.1.3 Guest VM 2 10.1.1.3 Guest VM 2 Private IP 10.1.1.112 DHCP, DNS NAT Load Balancing VPN Physical Load Balancer 10.1.1.4 Guest VM 3 10.1.1.4 Guest VM 3 Public IP 6.37.1.11 10.1.1.5 Guest VM 4 10.1.1.5 Guest VM 4 CS Virtual Router DHCP, DNS

Comparison of guest network options Layer-2 Layer-3 Isolation VLAN/SDN Security Groups Performance Better Network setup Moderate Easy Support broadcast Yes No Scalability Good Best Interoperability with physical servers Poor

Storage Primary Storage Secondary Storage (Object Storage) Configured at Cluster-level. Close to hosts for better performance Stores all disk volumes for VMs in a cluster Cluster can have one or more primary storages Local disk, iSCSI, FC or NFS Primary Storage L3 switch Pod 1 Secondary Storage L2 switch Host 2 Cluster 1 Host 1 Configured at Zone-level Stores all Templates, ISOs and Snapshots Zone can have one or more secondary storages NFS, S3 interfaces Secondary Storage (Object Storage) Primary Storage

Deployment and Software Architecture

Management Servers managing a Region Single Management Server can manage multiple zones Zones can be geographically distributed but low latency links are expected for better performance Single MS node can manage up to 10K hosts. Multiple MS nodes can be deployed as cluster for scale or redundancy Data Center 1 Data Center 2 Zone 3 Zone 2 Data Center 2 Zone 3 Zone 2 Management Servers Data Center 3 Zone 4 Zone1 Data Center 2 Zone 3 Zone 2 Data Center 2 Zone 3 Zone 2 Data Center 2 Zone 3 Zone 2

… Cloud-era zone deployment … … … … Region 2 Mgmt Server Cluster Admin Account Replication Internet Zone in Region 2 Region 1 Mgmt Server Cluster Router Primary MySQL Replica MySQL Load Balancer Site-to-Site VPN L3 Core Switch Top of Rack Switch … Object Store Servers … … … … Availability Zone 1 Pod 1 Pod 2 Pod 3 Pod N

Traditional zone deployment … Load Balancer Object Store Core Switch Internet 10Gbps Storage & Mgmt 1Gbps Guest … Load Balancer Core Switch Aggregation Switch TOR Switch Compute Nodes NFS Primary Storage Object Store Pod 1   Pod 2 Pod 200

Service VMs CloudStack-owned VMs Three main types Runs Debian Squeeze Virtual Router Console Proxy Secondary Storage VM Runs Debian Squeeze Runs for system or user Solve datapath issues Scale vertically and horizontally

Management Server internals

VM provisioning process When a user requests a VM instance, there are several steps performed. The user logs in and selects the desired availability zone for their instance, and then selects the desired template from the list of templates available to them. This is the trigger for the provisioning process. Depending on the instance and zone requirements, optional network services such as routing, dhcp and load balancing are provisioned for the zone. If these services are already provisioned, and can be shared by the user, then shared instances are used; otherwise isolated instances of the network services are used. The template representing the root disk of the VM is copied from the secondary storage for the zone to the primary storage for the cluster. CloudStack attempts to localize services for accounts to as few clusters as possible. This is done partly for security reasons, and partly to ensure optimal performance for provisioned services. If the instance requires any data volumes, the data volumes are created on primary storage for the cluster. Note that the storage preferences for the root volume and data volumes may be different resulting in the volumes occupying different primary storage devices within a given cluster. For example, data disks may have attributes which place them on a primary storage device which is continuously backed up while the root volume might be located on local storage. CloudStack then instructs the host to create and start the instance VM VM provisioning process User Requests Instance Provision Optional Network Services Copy instance template from secondary storage to primary storage on appropriate cluster Create any requested data volumes on primary storage for the cluster Create instance Start instance Zone Secondary Storage Pod Cluster Host Primary Storage VM Template

CloudStack software modules Presentation OAMP API End User API AWS API S3 API End User Services Accounts/ACL Policies Offerings Templates Console Proxy Domains & Projects Virtual Resource Management HA Usage Statistics Collection Alerts VM Sync Data Center Abstraction Layer Orchestration Deployment Planning Templates SDN Snapshots Configuration / Mappings Hardware Resource Management Storage Pools Hypervisor Clusters L2/L3 Networks Network Services Object Storage

Management Server interaction with hypervisors XAPI HTTP vCenter XenServer KVM Agent OVM Agent XCP ESX XS 5.6, 5.6FP1, 5.6 SP2, 6.0.2, XCP 1.1 Incremental Snapshots VHD NFS, iSCSI, FC & Local disk Storage over-provisioning: NFS ESX 4.1, 5.0, 5.1 Full Snapshots VMDK NFS, iSCSI, FC & Local disk Storage over-provisioning: NFS, iSCSI RHEL 6.0, 6.1, 6.2, 6.3 Ubuntu 12.04 Full Snapshots (not live) QCOW2 NFS, iSCSI & FC Storage over-provisioning: NFS OVM 2.2 No Snapshots RAW NFS & iSCSi No storage over-provisioning

Scalability to 30,000 hosts in production Mgmt Server CPU Util. Seconds to deploy 25,000 …. to …. 30,000 VMs 0 …. to …. 30,000 VMs Simulator developed to test massive scale Four Management Servers can manage 30,000 hosts Scale to hundreds of thousands of hosts possible with multiple management server clusters (regions)

Futures

Expanding orchestration control Apache CloudStack API Apache CloudStack API Apache CloudStack API Apache CloudStack API Hypervisor Firewall Load Bal Baremetal Storage Security Switches

Futures (mostly in ACS 4.2) Object storage technologies via S3 SDN integrations Blade orchestration Region support Zone wide primary storage Improved CLI Hyper-V Additional API support(?)

The future needs you! Project web site: http://cloudstack.apache.org Mailing lists: dev-subscribe@cloudstack.apache.org users-subscribe@cloudstack.apache.org IRC: #CloudStack on irc.freenode.net Join your local CloudStack user group!

Thanks