1. Kenichi Kourai Kouta Sannomiya Kyushu Institute of Technology, Japan
Seamless and Secure Application Consolidation for Optimizing Instance Deployment in Clouds
I'm Kenichi Kourai from Kyushu Institute of Technology, Japan.
I'm gonna talk about seamless and secure application consolidation for optimizing instance deployment in clouds.
This is joint work with my student, who has graduated.
Kenichi Kourai
Kouta Sannomiya
Kyushu Institute of Technology, Japan

2. Cost Reduction in IaaS Clouds
IaaS clouds provide users with instances (VMs)
The cost depends on the type and number of instances
Need to optimize instance deployment
The loads of instances continue to change
Used instances should be always sufficient but minimal
Infrastructure-as-a-Service clouds provide users with instances, which are usually virtual machines.
Users can run their applications in their instances.
In this example, two application servers and a database are running in three instances.
Users can deploy instances of favorite types and the necessary number of instances.
The cost depends on the type and number of instances.
To reduce the cost, users need to always optimize instance deployment.
This is because the loads of instances continue to change like this.
So, used instances should be always sufficient but minimal.
In IaaS clouds, the optimization of instance deployment is performed according to resource utilization of instances.
instance
instance
instance
app
server
app
server DB
IaaS cloud

3. Most Popular Optimization
Scale-in/out adjust the number of instances
Scale-out increases it against over-utilization
Scale-in decreases it to reduce costs
Scale-in cannot further reduce the number when only one instance is deployed
At least one instance is required
The most popular optimization is scale-in and out.
These adjust the number of instances.
When an application becomes over-utilized, the user can increase the number of instances by scale-out.
In contrast, when the application becomes under-utilized, the user can decrease the number by scale-in to reduce costs.
However, when only one instance is deployed for an under-utilized application, the user cannot further reduce the number of instances by scale-in.
For example, the load of an intra server may become almost zero during weekends.
But at least one instance is required if the server cannot be stopped.
instance 1
instance 1
instance 2
scale-out
app
app
app
scale-in
low
load
high
load

4. Optimization for One Instance
Scale-up/down adjust the amount of resources
Switch instance types offline in most of existing clouds
Downtime occurs during the switch
Users have to restart applications in new instances
Cost reduction is limited
By the cost of the minimum instance type
The optimization for one instance is scale-up and down.
These adjust the amount of resources assigned to each instance.
When an application becomes over-utilized, the user can increase the number of virtual CPUs by scale-up.
In contrast, when the application becomes under-utilized, the user can decrease the amount of resources by scale-down to reduce costs.
Most of existing clouds achieve scale-up and down by switching instance types offline.
When the user switches instance types, he has to stop applications in the current instance, move their data to a new instance, and restart them in the new instance.
During this switch, downtime occurs.
In addition, cost reduction is limited by the cost of the minimum instance type.
instance
instance
scale-up
app
app
scale-down
low
load
high
load

5. Optimization by Consolidation
Consolidate applications into one instance
De-consolidate them under over-utilization
Cause downtime during (de-)consolidation
When applications are moved between instances
Isolation among applications becomes weaker
Consolidation across users is impossible
For further optimization, users can consolidate multiple applications into one instance.
This is called application consolidation.
For example, consider a multi-tire application, which consists of an application server and a database.
When both are under-utilized, the user can run them in one instance to reduce costs.
When the instance becomes over-utilized, the users can de-consolidate these applications to multiple instances again.
However, this optimization also causes downtime when applications are moved between instances for consolidation and de-consolidation.
In addition, isolation among applications becomes weaker than when using one instance per application because of no isolation provided by VMs.
Therefore, application consolidation across multiple users is impossible.
instance 1
instance 1
instance 2
consolidate
app 1
app 1
app 2
app 2
de-consolidate
low
load
high
load

6. FlexCapsule Enable seamless and secure application consolidation
Run each application in an app VM inside an instance
Using nested virtualization
Migrate applications between instances with negligible downtime
Guarantee security between applications by VM's isolation
instance 1
instance 2
To enable seamless and secure application consolidation even across different users, we propose FlexCapsule.
FlexCapsule runs each application in a lightweight VM called an app VM inside an existing instance.
This is achieved using the technology called nested virtualization, which enables VMs to run inside a VM.
Nested virtualization imposes extra overhead, but it is reported that the overhead is 6 to 8% for common workloads.
By introducing app VMs, FlexCapsule can migrate applications between instances with negligible downtime.
Thanks to strong isolation between app VMs, FlexCapsule can guarantee security between applications consolidated into one instance.
app VM 1
app VM 2
app 1
app 2

7. System Architecture Each app VM runs only one process and a library OS
An OS server in each instance supports multi-process
Each app VM is assigned a private IP address
NAPT translates it into the public IP address of an instance
A VPN is constructed across app VMs in multiple instances
instance 1
instance 2
This figure shows the system architecture of FlexCapsule.
Each app VM runs only one application process and a library OS.
A library OS is a library that provides OS functions and more lightweight than general-purpose OSes.
To support multi-process functions, FlexCapsule runs an OS server in each instance.
Each app VM is assigned a private IP address to reduce the cost for using a public IP address.
To provide services to the outside, FlexCapsule uses network address port translation.
NAPT translates the private IP address of an app VM into the public IP address of an instance, and vice versa.
Thanks to this NAPT, multiple app VMs running different applications, for example, a web server and a mail server, can use the same public IP address.
Also, FlexCapsule constructs a virtual private network across app VMs in multiple instances to enable seamless VM migration.
app VM 1
app VM 2
app VM 3 OS
server
app 1
app 2
app 3 OS
server
VPN
libOS
libOS
libOS
(public)

8. Optimization Using App VMs (1/2)
Application consolidation
Migrate under-utilized app VMs to one instance
Stop the source instances & re-assign their IP addresses
Application de-consolidation
Deploy new instances & migrate over-utilized app VMs
Re-assign IP addresses
When performing application consolidation, FlexCapsule migrates under-utilized app VMs to one instance.
If the source instances of the migration have no app VMs, FlexCapsule stops them and re-assigns their public IP addresses to the destination instance.
In this example, instance 2 has no app VM, it is stopped and its IP address is re-assigned to instance 1.
Thus, migrated app VMs can be reached using the same public IP address before application consolidation.
In contrast, when performing application de-consolidation, FlexCapsule deploys new instances and migrates over-utilized app VMs to them.
At the same time, it re-assigns one of the public IP addresses assigned to the source instance to the new instances.
instance 1
instance 2
instance 1
app
VM 1
app
VM 2
app
VM 1
app
VM 2
re-assign

9. Optimization Using App VMs (2/2)
Scale-up/down
Deploy a new instance & migrate app VMs
Stop the original instance & re-assign its public IP address
Scale-out
Clone app VMs & migrate them to new instances
Allow using the original public IP address by NAPT and VPN
To scale up and down an application, as shown in the left figure, FlexCapsule first deploys a new instance of appropriate type.
Then, it migrates app VMs inside the original instance to the new one.
Finally, it stops the original instance and re-assigns that public IP address to the new one.
To scale out an application, as shown in the right figure, FlexCapsule clones app VMs inside the original instance, creates new instances, and migrates the cloned app VMs to them.
The new instances are assigned different public IP addresses, but app VMs inside them can use the original public IP address using NAPT and a VPN.
original
instance
new instance
original
instance
new
instance
clone
app
VM '
app VM
app VM

10. < FlexCapsule OS A library OS running in an app VM
Reduce the memory footprint of an app VM
Enable faster VM migration
Reduce the overhead of nested virtualization
Simplify virtualization using para-virtualization (PV)
Need support for VM migration
A library OS running in an app VM is called the FlexCapsule OS.
Since only necessary functions are linked to an application, the FlexCapsule OS can reduce the memory footprint of an app VM.
So, running an app VM per application process doesn't require so much extra memory, compared with using a general-purpose OS.
The small memory footprint of an app VM enables faster VM migration because the migration time depends on the memory size of a VM.
The FlexCapsule OS reduces the overhead of nested virtualization by using para-virtualization.
Since a para-virtualized OS simplifies virtualization, the performance of app VMs can be improved.
In compensation of this performance gain, a para-virtualized OS needs to support VM migration by itself.
traditional VM
app VM
<
app
services
app
libOS OS

11. FlexCapsule Server An OS server running in each instance
Used for cooperation between app VMs
E.g., clone an app VM when it invokes fork
Manage NAPT rules to forward packets to app VMs
Register a NAPT rule when an app VM invokes listen
:80
=> :80 OS
server
app VM
app
An OS server running in each instance is called the FlexCapsule server.
Since an app VM runs only one application process, it cannot achieve functions for multiple processes.
The FlexCapsule server is used for cooperation between app VMs.
For example, when an application invokes the fork function, the FlexCapsule server clones the whole app VM.
It also manages NAPT rules to forward packets to app VMs.
When an app VM invokes the listen function, the FlexCapsule server registers a NAPT rule between public and private IP addresses like this.
fork
libOS
clone
NAPT
gateway
app VM '
app
libOS
register
listen

12. Migration of App VMs Send a suspend request to an app VM
FlexCapsule OS suspends PV devices
Transfer the states of the app VM to the destination
Memory contents and virtual CPU/device states
FlexCapsule OS resumes PV devices
source instance
destination instance
When an app VM is migrated, the FlexCapsule server sends a suspend request to the FlexCapsule OS in an app VM.
The FlexCapsule OS first suspends para-virtual devices such as network devices.
Specifically, it safely disconnects device drivers from such virtual devices.
After that, the FlexCapsule server transfers the states of the app VM to the destination instance.
The states include memory contents and virtual CPU and device states.
When the migration of the app VM is completed, the FlexCapsule OS resumes para-virtual devices.
app VM
app VM ' VM
states
suspend
app OS
server OS
server
libOS
read
write

13. Migration-transparent Networking
Apply the same NAPT rules after VM migration
Use NAPT rules in the source instance
Need to transfer no NAPT rules to the destination
Deliver packets inside a VPN across instances
Seamlessly forward packets to migrated app VMs
source instance
destination instance
FlexCapsule provides migration-transparent networking by using a VPN across instances.
Even after an app VM has been migrated, FlexCapsule can apply the same NAPT rules to the app VM.
Since it continues to use NAPT rules in the source instance, it doesn't need to transfer NAPT rules to the destination instance.
When the source instance receives packets to the migrated app VM, FlexCapsule delivers them inside a VPN including the destination instance.
As a result, it can seamlessly forward packets to the migrated app VM.
NAPT
gateway
app VM
packets
VPN

14. Fork of App VMs Duplicate the states of a parent app VM
Efficiently copy memory contents in the hypervisor
Copy virtual CPU/device states
Share the disk between parent and child app VMs
Seamlessly insert copy-on-write disks to both app VMs
instance
COW
disk
disk
When an application invokes the fork function, the FlexCapsule OS sends a fork request to the FlexCapsule server.
As an alternative method, the user can send the request to the server.
The FlexCapsule server creates a child app VM and duplicates the states of the parent app VM to the child one.
To perform this duplication efficiently, it invokes the underlying hypervisor and copies memory contents.
The hypervisor also copies virtual CPU and device states.
In addition, the FlexCapsule server shares the disk between the parent and child app VMs.
This sharing is done in a copy-on-write manner and modifications are reflected only to one of the app VMs.
To achieve this, the FlexCapsule server seamlessly inserts copy-on-write disks to both app VMs.
parent
app VM
child
app VM OS
server
fork
hypervisor
VM states

15. Load-balancing with Fork
Create a pool of app VMs
Clone an app VM in advance or on demand
All the app VMs listen on the same port
Forward packets to one app VM in round-robin
The pool can be across instances by using a VPN
instance 1
instance 2
FlexCapsule enables load balancing with fork.
For this purpose, it creates a pool of app VMs.
In advance or on demand, it repeatedly clones an app VM .
If the original app VM listens on a port, all the cloned app VMs also listen on the same port.
In this example, all the app VMs listen on port 80.
When packets are sent to the listening port, FlexCapsule forwards them to one app VM in the pool.
It selects such an app VM in a round-robin fashion.
For example, the selection is done in the order of app VM, app VM prime, and app VM double prime.
If some of the app VMs are migrated to other instances, the pool can be across instances by using a VPN.
packets
app VM
app
VM '
app
VM ''
NAPT
gateway
VPN
:80
:80
:80

16. Implementation We have implemented FlexCapsule in Xen 4.2
VMs run Xen using nested virtualization
DomU is an app VM
Dom0 runs FlexCapsule server
FlexCapsule OS is based on OSV 0.21
Fully virtualized but use para-virtual devices
Enable running many existing applications
With no or slight modifications
We have implemented FlexCapsule in Xen.
VMs run Xen using nested virtualization.
Its DomU is an app VM and Dom0 runs the FlexCapsule server.
The FlexCapsule OS is based on OSv.
OSv is fully virtualized but uses para-virtual devices.
It can run many existing applications with no or slight modification.

17. Experiments We conducted experiments to examine
Effectiveness of optimizing instance deployment using FlexCapsule
Application de-consolidation
Scale-out and scale-up
Performance of FlexCapsule
Migration of an app VM
Fork of an app VM
Host
CPU: Intel Xeon E3-1290v2
Memory: 8 GB
Instance (VM)
vCPU: 1-4
Memory: 2 GB
App VM
vCPU: 1
Memory: MB
To confirm the effectiveness of FlexCapsule, we conducted several experiments.
One of our aims is to show the effectiveness of optimizing instance deployment using FlexCapsule.
We examined the performance improvement by application de-consolidation, scale-out, and scale-up.
The other aim is to examine the performance of FlexCapsule.
We measured the performance of migration and fork of an app VM.

18. Application De-consolidation
We compared application performance under consolidation and de-consolidation
Ran three app VMs in one instance and three instances
De-consolidation could improve application performance
1.9x - 2.7x
First, we compared application performance under consolidation and de-consolidation.
We used three app VMs, which ran the lighttpd web server, the memcached caching system, and the Redis in-memory database.
When consolidating these three app VMs, we ran them inside one instance.
When de-consolidating them, we used three instances.
This figure shows relative performance based on the throughput of each application when we consolidated the app VMs.
De-consolidation could improve application performance by a factor of 1.9 to 2.7.
This shows the effectiveness of application de-consolidation using app VMs.

19. Scale-out and Scale-up
We increased the number of instances for scale- out
The total throughput of lighttpd increased
We increased the number of vCPUs for scale-up
The throughput of lighttpd increased
The performance of network processing was improved
Next, we investigated the effectiveness of scale-out and scale-up using app VMs.
For scale-out, we ran one app VM inside one instance and increased the number of instances.
The left figure shows the total throughput of the lighttpd web servers.
The total throughput in two instances became twice of that in one instance.
For scale-up, we increased the number of virtual CPUs assigned to the instance.
As shown in the right figure, the throughput of lighttpd increased.
lighttpd is single-threaded, so using multiple virtual CPUs don't increase the performance.
But the performance of network processing in a virtual network device was improved by increasing the number of virtual CPUs.

20. Migration Performance
The downtime of an app VM was shorter
FlexCapsule OS suspended fewer virtual devices
The migration time of an app VM was 1.5x shorter
In the minimum memory size
We migrated an app VM and measured the downtime and migration time.
For comparison, we also migrated a regular VM running Linux.
The left figure shows the downtime.
The downtime did almost not depend on the memory size of the VM.
That of an app VM was shorter than that of the regular VM.
This is because the FlexCapsule OS suspended fewer virtual devices.
The right figure shows the migration time.
The time was proportional to the memory size of the VM.
The app VM could run with the smaller amount of memory, thanks to the smaller memory footprint of a library OS.
So, in the minimum memory size, the migration time was 1.5 times shorter.

21. Fork Performance We measured the execution time of fork
Slightly proportional to the memory size
The bottleneck was restoring device states
We compared with fork using standard commands
Our implementation was 18x faster
Finally, we measured the execution time of the fork function in an app VM.
The left figure shows the fork time in FlexCapsule.
The time was slightly proportional to the memory size of the app VM.
The bottleneck was restoring virtual device states.
We compared with fork implementation using Xen's standard save and restore commands.
Our implementation was 18 times faster, thanks to native support by the hypervisor.

22. Related Work Picocenter [Zhang+ EuroSys'16]
Run applications in containers inside instances
Designed for long-lived, mostly idle applications
Resource as a Service [Ben-Yehuda+ HotCloud'12]
Dynamically change the amount of resources of instances
Cannot decrease the number of vCPUs to less than one
Graphene [Tsai+ EuroSys'14]
Linux-compatible library OS supporting multi-process
Process-level isolation is weaker than VM-level one
Picocenter runs applications in containers inside instances.
To optimize instance deployment, it swaps applications to and from cloud storage.
Unlike FlexCapsule, it is designed for long-lived, mostly idle applications, which can be stopped.
Resource-as-a-Service clouds can dynamically change the amount of resources of instances.
For example, VMware vCloud Air supports seamless and flexible scale-up and down.
However, it cannot decrease the number of virtual CPUs to less than one.
Graphene is a Linux-compatible library OS supporting multi-process.
It achieves process fork and migration.
But its process-level isolation is weaker than VM-level one provided by FlexCapsule.

23. Conclusion FlexCapsule Future Work Optimize instance deployment
Run each application in an app VM inside an instance
Using nested virtualization and a library OS
Enable seamless and secure application consolidation
Using VM migration and strong isolation among VMs
Future Work
Enable various applications to run in app VMs
Reduce the overhead of nested virtualization
In conclusion, we proposed FlexCapsule for optimizing instance deployment.
FlexCapsule runs each application in an app VM inside an instance.
It uses nested virtualization to run app VMs and a library OS to reduce its overhead.
Using VM migration and strong isolation among VMs, FlexCapsule enables seamless and secure application consolidation across users.
One of our future work is to enable various applications to run in app VMs.
For that, we need to advance multi-process support such as inter-process communication.
Another direction is to reduce the overhead of nested virtualization.
For example, we need to develop para-virtualized OSv.