1. Apache CloudStack An Introduction Kevin Kluge
Apache CloudStack Committer
Elasticsearch VP Engineering

2. Use CloudStack to build IaaS clouds (like EC2)
What does CloudStack do? What is it?
Does not do PaaS or SaaS but those can run on it.
Does not connect to Amazon.
Use CloudStack to build IaaS clouds (like EC2)
Create VMs, disks, networks, network services
Self service
Meter usage
Written in Java
Scalable
Many vendor integrations
Native and EC2 API

3. How did Amazon build EC2? Amazon eCommerce Platform
AWS API (EC2, S3, …)
Amazon Orchestration Software
Open Source Xen Hypervisor
Networking
Commodity Servers
Commodity Storage

4. How can you build your cloud?
Amazon eCommerce Platform
Your Portal (Optional)
AWS API (EC2, S3, …)
CloudStack or AWS API
Amazon Orchestration Software
CloudStack Orchestration Software
Open Source Xen Hypervisor
ESXi, KVM, XenServer/XCP
Networking
Servers
Storage

5. Project history 2008/2009: closed-source development via Cloud.com
First deployments in late 2009
May 2010: ~98% open source as GPLv3 (open core)
August 2011: 100% open source GPLv3
April 2012: Switch to Apache License v2
Submit code to Apache Software Foundation
March 2013: Graduate to Apache TLP

6. Project current state TLP within Apache Software Foundation
Apache CloudStack 4.0 and 4.1 delivered
4.2 under development; target August release
Uses ASF infra for bugs, wiki, mailing lists
~50% non-Citrix committers and PMC members
Across contributors Citrix percentage much lower

7. Messages = Posts
Source:

8. Source: http://www.qyjohn.net/?p=3321

9. Workloads in the Cloud

10. Different workloads have different needs
Clouds can support both types of workloads
But architecture to deliver SLA is different
Different workloads have different needs
Amazon-style Workload
Traditional Workload
Design for failure
Expect reliability
Ephemeral resources
Back-up everything
Multi-site redundancy
HA, Fault tolerance
Self-service recovery
Admin controls recovery

11. CloudStack supports both workload types
Amazon-style Workloads
Traditional Workloads
CloudStack Mgmt Server
Traditional Zone
vSphere
Cloud-era Availability Zone
Cloud-era Availability Zone
Cloud-era Availability Zone
Enterprise Networking (e.g., VLAN)
ESXi Cluster
ESXi Cluster
ESXi Cluster
Object Storage
Enterprise Storage (e.g., SAN)

12. Object store is critical for Cloud-era workloads
Amazon-Style Region
CloudStack
Mgmt. Server
Workloads are distributed across availability zones
No guarantee on zone reliability
DBs and Templates snapped to object store.
For small failures, recreate instance in same zone
For DR, recreate instance in different zone
Dramatically less expensive
Availability Zone
Availability Zone
Availability Zone
Object Store

13. Features

14. Open platform for compute, storage, network
XCP/XS
VMware
KVM
Oracle VM
Bare metal
Hypervisor
Storage
Local Disk
iSCSI
NFS
Fiber Channel
Object Stores
Block & Object
Network
Network Type
Isolation
Load balancer
Firewall
VPN
Network & Network Services

15. Virtual machine management
Users
Start
Stop
Restart
Destroy
VM Operations
Console Access
CPU Utilized
Network Read
Network Writes
VM Status
Change
Service Offering
2 CPUs
1 GB RAM
20 GB
20 Mbps
4 CPUs
4 GB RAM
200 GB
100 Mbps

16. Volume and snapshot management
Add / Delete
Volumes
VM 1
Volume
Create Templates
from Volumes
Volume
Template
Schedule
Snapshots
Hourly
Daily
Weekly
Monthly
Now
View Snapshot
History
12/2/ am ….
2/2/ am

17. Multi-tenancy and account management
Admin
Org A
Reseller A
Domain
VMs, IPs, Snapshots…
Resources
Domain is a unit of isolation that represents a customer org, business unit or a reseller
Domain can have arbitrary levels of sub-domains
A Domain can have one or more accounts
An Account represents one or more users and is the basic unit of isolation
Admin can limit resources at the Account or Domain levels
Admin
Org C
Sub-Domain
Group B
Account
Group A
User 1
User 2

18. Service offerings Specify Resource Levels Configure Properties
Define Scope
CPU Cores
CPU (MHz)
Memory (MB)
Name
Compute
Custom Disk Size
Disk Size (GB)
Storage Tag
Public
Name
Disk
Network Rate
Redundant VR
Network
Firewall
Load balancer
CPU Cap
Host Tag
Enable HA
Public

19. Network offering Provides cloud operator defined service features
Isolation
Load Balancing
VPN
Firewall
Supports Physical Devices
NetScaler
F5 BIG-IP
Juniper SRX
Zone 1
Zone N
Pod 1
Pod N
Pod 1

20. Network and network services
Create Networks and attach VMs
Acquire public IP address for NAT & load balancing
Control traffic to VM using ingress and egress firewall rules
Set up rules to load balance traffic between VMs

21. Layer-3 guest network Network Services Managed Externally
Network Services Managed by CS
Public Network
/16
Security Group 1
Security Group 1
Public Network/Internet
Guest VM 1
Guest VM 1
Guest VM 2
Physical
Load Balancer
Guest VM 2
EIP, ELB
Guest VM 3
Guest VM 3
Guest VM 4
Guest VM 4 CS
Virtual Router CS
Virtual Router
Security Group 2
DHCP, DNS
Security Group 2
DHCP, DNS

22. Layer-2 guest virtual network
CS Virtual Router provides Network Services
External Devices provide Network Services
Guest Virtual Network /8
VLAN 100
Guest Virtual Network /8
VLAN 100
Public Network/Internet
Public Network/Internet
Guest VM 1
Public IP
Private IP
Guest VM 1
Juniper SRX
Firewall CS
Virtual Router
Gateway address
Guest VM 2
Guest VM 2
Private IP
DHCP, DNS
NAT
Load Balancing
VPN
Physical
Load Balancer
Guest VM 3
Guest VM 3
Public IP
Guest VM 4
Guest VM 4 CS
Virtual Router
DHCP, DNS

23. Comparison of guest network options
Layer-2
Layer-3
Isolation
VLAN/SDN
Security Groups
Performance
Better
Network setup
Moderate
Easy
Support broadcast
Yes No
Scalability
Good
Best
Interoperability with physical servers
Poor

24. Storage Primary Storage Secondary Storage (Object Storage)
Configured at Cluster-level. Close to hosts for better performance
Stores all disk volumes for VMs in a cluster
Cluster can have one or more primary storages
Local disk, iSCSI, FC or NFS
Primary Storage
L3 switch
Pod 1
Secondary
Storage
L2 switch
Host 2
Cluster 1
Host 1
Configured at Zone-level
Stores all Templates, ISOs and Snapshots
Zone can have one or more secondary storages
NFS, S3 interfaces
Secondary Storage (Object Storage)
Primary
Storage

25. Deployment and Software Architecture

26. Management Servers managing a Region
Single Management Server can manage multiple zones
Zones can be geographically distributed but low latency links are expected for better performance
Single MS node can manage up to 10K hosts.
Multiple MS nodes can be deployed as cluster for scale or redundancy
Data Center 1
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Management Servers
Data Center 3
Zone 4
Zone1
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2

27. … Cloud-era zone deployment … … … … Region 2 Mgmt Server Cluster Admin
Account
Replication
Internet
Zone in
Region 2
Region 1 Mgmt Server Cluster
Router
Primary MySQL
Replica MySQL
Load Balancer
Site-to-Site VPN
L3 Core Switch
Top of Rack Switch …
Object Store
Servers … … … …
Availability Zone 1
Pod 1
Pod 2
Pod 3
Pod N

28. Traditional zone deployment … Load Balancer Object Store Core Switch
Internet
10Gbps Storage & Mgmt
1Gbps Guest …
Load Balancer
Core Switch
Aggregation Switch
TOR Switch
Compute Nodes
NFS Primary Storage
Object Store
Pod 1
Pod 2
Pod 200

29. Service VMs CloudStack-owned VMs Three main types Runs Debian Squeeze
Virtual Router
Console Proxy
Secondary Storage VM
Runs Debian Squeeze
Runs for system or user
Solve datapath issues
Scale vertically and horizontally

30. Management Server internals

31. VM provisioning process
When a user requests a VM instance, there are several steps performed.
The user logs in and selects the desired availability zone for their instance, and then selects the desired template from the list of templates available to them. This is the trigger for the provisioning process.
Depending on the instance and zone requirements, optional network services such as routing, dhcp and load balancing are provisioned for the zone. If these services are already provisioned, and can be shared by the user, then shared instances are used; otherwise isolated instances of the network services are used.
The template representing the root disk of the VM is copied from the secondary storage for the zone to the primary storage for the cluster. CloudStack attempts to localize services for accounts to as few clusters as possible. This is done partly for security reasons, and partly to ensure optimal performance for provisioned services.
If the instance requires any data volumes, the data volumes are created on primary storage for the cluster. Note that the storage preferences for the root volume and data volumes may be different resulting in the volumes occupying different primary storage devices within a given cluster. For example, data disks may have attributes which place them on a primary storage device which is continuously backed up while the root volume might be located on local storage.
CloudStack then instructs the host to create and start the instance VM
VM provisioning process
User Requests Instance
Provision Optional Network Services
Copy instance template from secondary storage to primary storage on appropriate cluster
Create any requested data volumes on primary storage for the cluster
Create instance
Start instance
Zone
Secondary Storage
Pod
Cluster
Host
Primary Storage VM
Template

32. CloudStack software modules
Presentation
OAMP API
End User API
AWS API
S3 API
End User Services
Accounts/ACL
Policies
Offerings
Templates
Console Proxy
Domains & Projects
Virtual Resource Management HA
Usage
Statistics
Collection
Alerts
VM Sync
Data Center Abstraction Layer
Orchestration
Deployment Planning
Templates
SDN
Snapshots
Configuration / Mappings
Hardware Resource Management
Storage
Pools
Hypervisor
Clusters
L2/L3
Networks
Network
Services
Object
Storage

33. Management Server interaction with hypervisors
XAPI
HTTP
vCenter
XenServer
KVM
Agent
OVM
Agent
XCP
ESX
XS 5.6, 5.6FP1, 5.6 SP2, 6.0.2, XCP 1.1
Incremental Snapshots
VHD
NFS, iSCSI, FC & Local disk
Storage over-provisioning: NFS
ESX 4.1, 5.0, 5.1
Full Snapshots
VMDK
NFS, iSCSI, FC & Local disk
Storage over-provisioning: NFS, iSCSI
RHEL 6.0, 6.1, 6.2, 6.3 Ubuntu 12.04
Full Snapshots (not live)
QCOW2
NFS, iSCSI & FC
Storage over-provisioning: NFS
OVM 2.2
No Snapshots
RAW
NFS & iSCSi
No storage over-provisioning

34. Scalability to 30,000 hosts in production
Mgmt Server CPU Util.
Seconds to deploy
25, …. to … ,000 VMs
…. to … ,000 VMs
Simulator developed to test massive scale
Four Management Servers can manage 30,000 hosts
Scale to hundreds of thousands of hosts possible with multiple management server clusters (regions)

35. Futures

36. Expanding orchestration control
Apache CloudStack API
Apache CloudStack API
Apache CloudStack API
Apache CloudStack API
Hypervisor
Firewall
Load Bal
Baremetal
Storage
Security
Switches

37. Futures (mostly in ACS 4.2)
Object storage technologies via S3
SDN integrations
Blade orchestration
Region support
Zone wide primary storage
Improved CLI
Hyper-V
Additional API support(?)

38. The future needs you! Project web site: http://cloudstack.apache.org
Mailing lists:
IRC: #CloudStack on irc.freenode.net
Join your local CloudStack user group!

39. Thanks