1. Consistent Offline Update of Suspended Virtual Machines in Clouds
I'm Kenichi Kourai from Kyushu Institute of Technology.
I'm gonna talk about Consistent Offline Update of Suspended Virtual Machines in Clouds.
Kenichi Kourai and Yuji Shiota
Kyushu Institute of Technology, Japan

2. Security Updates Software vulnerabilities are increasing
16,555 in 2018 [CVE]
Security updates are indispensable
Download software packages from the Internet
Install files in the packages
Execute package scripts
Recently, software vulnerabilities are increasing.
Ten years ago, the number of reported vulnerabilities was about 6,000.
Last year, that was more than 16,000.
Therefore, security updates are indispensable.
For security updates, first, the update manager downloads software packages from the Internet.
Then, it extracts files from the packages and installs them to the system.
In addition, it executes package scripts before and after the installation and uninstallation if necessary.
Internet
update
manager

3. Unused VMs in Clouds
There exist many unused virtual machines (VMs) in IaaS clouds [Zhou+ '10]
Unused VMs are stopped or suspended
Suspending VMs is desirable for quick resumption
Many vulnerabilities are found for unused VMs
In Infrastructure-as-a-Service clouds, users can easily create their virtual machines and install various software such as favorite operating systems and servers.
They run such VMs only when necessary.
Therefore, there exist many unused VMs in IaaS clouds.
Unused VMs are usually stopped or suspended because only running VMs are charged in IaaS clouds.
Suspending VMs is more desirable than stopping VMs because VMs can be resumed quickly.
When a VM is suspended, its current state is saved to storage.
This is analogous to hibernation of physical machines.
In either case, while VMs are not running, many vulnerabilities are often found in installed software.
running VM
stopped VM
suspended VM
vuln.
save

4. Risks of Online Update
It is at high risk to apply security updates after VMs become online
The VMs suffer from attacks immediately
Need to download packages from the Internet
It takes a long time to update the VMs completely
Such unused VMs can be booted or resumed after a long time.
After they become online, it is at high risk to apply security updates.
If attacks against found vulnerabilities are widely spread, the VMs suffer from the attacks immediately after they are connected to the Internet.
Since VMs have to connect to the Internet and download software packages, it is difficult to prevent attacks via the Internet.
If there are many security updates, it takes a long time to update the VMs completely.
The probability of attacks becomes higher.
Internet
vuln.
update
manager
download
attack
resumed VM

5. Offline Update
Update VMs in an isolated execution environment [Microsoft]
Need to permit cloud admins to do this
Modify the disks of VMs directly [Zhou+ '10]
Not applicable to suspended VMs
Often corrupt the disks by cache inconsistency
To solve this issue of online update, offline update of VMs has been proposed.
One approach is to update VMs in an execution environment isolated from the Internet.
In advance, it downloads security updates to a local server.
Then, it boots or resumes VMs in the execution environment and runs the update managers in the VMs.
However, users need to permit cloud administrators to do this.
The other approach is to directly modify the virtual disks of VMs.
It downloads security updates and writes extracted files to the disks outside VMs.
This approach enables offline update of stopped VMs, but it is not applicable to suspended VMs.
If it modifies the disks of suspended VMs, the disks are often corrupted by cache inconsistency.
Internet
update
manager
download
modify
disk
stopped VM

6. OUassister Enable consistent offline update of suspended VMs
Emulate security updates to a VM offline
Keep the integrity of the virtual disk
Apply the emulation results to the VM online
The VM itself updates its disk consistently
To enable consistent offline update of suspended VMs, we propose OUassister.
OUassister emulates security updates to a VM offline.
It prevents the update manager from directly modifying the virtual disk of the VM.
Therefore, the integrity of the virtual disk is kept.
Even if this offline task takes time, that does not increase the risk because the VM is kept offline.
When the VM is resumed and becomes online later, OUassister just applies the emulation results to the VM.
Since the VM itself updates its virtual disk consistently, the integrity of the virtual disk is kept.
This online task is much less than traditional online update.
offline task
online task
update
manager
emulation
results
emulate
updates
transfer
disk
resumed VM

7. Emulation Environment
Construct an emulation environment outside a target VM
Execute the update manager as in the VM
Download and install packages
Provide access to the virtual disk of the VM
Read package information and write updated files
For the update emulation, OUassister constructs an emulation environment outside a target VM.
In the emulation environment, it executes the update manager as if the update manager ran inside the VM but more securely.
The update manager first downloads software packages from the Internet and then installs them as usual.
To enable this, OUassister provides access to the virtual disk of the VM.
The update manager can read information on installed packages in the VM and write updated files.
However, as mentioned before, actual writes can corrupt the virtual disk if the VM is suspended.
update
manager
emulate
access
disk
suspended VM
emulation environment

8. Shadow Filesystem
Use the union filesystem to prevent the corruption of the virtual disk
Layered filesystem stacking two filesystems
Upper layer: an empty filesystem for updated files
Lower layer: the filesystem used in the virtual disk
Give an illusion of updating the virtual disk
To prevent the corruption of the virtual disk, OUassister provides the shadow filesystem using the union filesystem.
The union filesystem can create a layered filesystem by stacking two filesystems.
In the shadow filesystem, the upper layer is an empty filesystem for writing updated files.
The lower layer is the filesystem used in the virtual disk.
When the update manager reads files from the shadow filesystem, it accesses the lower layer.
In contrast, when it writes files to the shadow filesystem, it accesses the upper layer.
Using the shadow filesystem, OUassister gives an illusion of updating the virtual disk to the update manager.
upper layer
lower layer
disk

9. Accurate Emulation Provide the shadow proc filesystem
Return dynamic system information on the OS in the target VM
E.g., pseudo files for configs, process status, etc.
Use VM introspection
Analyze OS data structure in the memory of a VM
For more accurate emulation, OUassister provides the shadow proc filesystem.
This filesystem returns dynamically generated system information on the operating system in the target VM.
For example, it provides pseudo files containing system configurations, process status, and so on.
To obtain necessary information in the VM and construct the shadow proc filesystem, OUassister uses VM introspection.
VM introspection is a technique for securely accessing the internal state of a VM from the outside.
It analyzes operating system data structure in the memory of a VM.
update
manager VM
introspection
procfs
shadow procfs VM
emulation environment

10. Extracting Updated Files
Extract updated files using the union filesystem
Create a list of removed files
Removed files are recorded as whiteout files
Create an archive of updated files
Not include whiteout files
list of
remoted
files
After the update emulation, OUassister extracts updated files using the union filesystem.
First, it creates a list of removed files.
In the union filesystem, removed files are recorded in the upper layer as special files called whiteout files.
A whiteout file indicates that the corresponding file in the lower layer is removed.
Then, OUassister creates an archive of updated files recorded in the upper layer.
At this time, whiteout files are not included in the archive.
upper layer
whiteout
archive
lower layer

11. Extracting Package Scripts
Hook the execve system call
Obtain the script paths and save the script files
Execute no scripts offline
Extract 4 types of scripts from packages
Pre-installation, pre-removal, post-installation, and post-removal in Ubuntu
During update emulation, OUassister extracts package scripts executed by the update manager.
To do so, it hooks the execve system call issued by the update manager.
Then, it obtains the script paths and saves the script files.
Since the update manager cannot execute the scripts offline, OUassiter rewrites the path specified in the system call.
Script extraction is a task specific to the package management system.
For Ubuntu packages, OUassister extracts four types of scripts.
Pre-installation and pre-removal scripts are executed before installation and uninstallation, respectively.
For example, theses scripts stop servers.
Post-installation and post-removal scripts are executed after them, respectively.
These scripts update several databases, for example.
pre-installation
file extraction/
removal
post-installation
pre-removal
post-removal

12. Applying Emulation Results
Apply the results of offline emulation after the target VM is resumed
Extract the archive of updated files inside the VM
Remove files from the disk
Execute the saved scripts in a saved order
Temporarily disconnect the VM from the Internet
OUassister applies the results of offline emulation to the virtual disk of a target VM after the VM is resumed.
First, OUassister transfers the archive of updated files and the list of removed files to the VM.
Then, it extracts the archive and removes the specified files inside the VM.
Also, OUassister executes saved package scripts in a saved order.
Before extracting the archive, it executes saved pre-installation and pre-removal scripts.
After removing the files, it executes saved post-installation and post-removal scripts.
To prevent the resumed VM from being attacked while applying emulation results, OUassister temporarily disconnects the VM from the Internet.
Internet
list of
remoted
files
archive
disk
resumed VM

13. Experiments We executed 4 types of software updates Comparison
Install/uninstall the nginx Web server
Update the OpenSSL library
Update the package list
Comparison
Traditional online update inside a VM
We conducted several experiments to examine the effectiveness of offline update in OUassister.
We executed four types of software updates using the apt command:
installing the nginx Web server, uninstalling it, updating the OpenSSL library, and updating the package list.
For comparison, we executed traditional online update inside a VM.
We used Xen, ran a VM on top of it, and ran Ubuntu in the VM.
During the offline task in OUassister, we suspended the VM.
host
CPU: Intel Xeon E5630
Memory: 6 GB
HDD: 250 GB
Network: Gigabit Ethernet
Xen 4.1.3 VM
vCPU: 1
Memory: 512 MB
Disk: 4 GB
Ubuntu LTS

14. Update Time Online update time was reduced
11-57% of traditional online update
Breakdown of the total update time
The offline tasks occupied a large portion
First, we measured the online update time, which is the time needed for applying software updates inside a running VM.
For OUassister, the online update time is the time for only the online task performed after a VM is resumed.
As shown in the left-hand side figure, the online update time was reduced successfully in OUassister.
That was 11 to 57% of traditional online update.
Next, we measured the total update time, which is the sum of the offline and online update time, when we used OUassister.
The right-hand side figure shows the time with its breakdown.
The emulation time occupied the largest portion of the total update time.
The archive time was proportional to the size of updated files.

15. Extracted Files/Scripts
Updated files included various data
The data, cache, and log of apt and man database
Various numbers of scripts were executed
6 packages were installed/uninstalled for nginx
We examined the files and scripts extracted after update emulation.
The left-hand side figure shows the number of updated files.
Updated files included the data, cache, and log of the package management system and the database for man as well as files contained in the updated packages.
The right-hand side figure shows the breakdown of extracted scripts.
The number of scripts depended on that of updated packages.
Six packages were installed and uninstalled for nginx.
Only one package was updated for openssl.

16. Access to Shadow Proc Filesystem
Necessary for the correct execution of the update manager
OS configuration (1-3)
Process status (4) #
file
description 1
/proc/filesystems
supported filesystems 2
/proc/sys/kernel/ngroups_max
max of process groups 3
/proc/sys/net/ipv6/
related to IPv6 4
/proc/[pid]/stat
process status 5
/proc/1/root/
link to the init process 6
/proc/self/
link to the update manager
To show that our accurate emulation is necessary, we examined accessed files in the shadow proc filesystem.
For the first two files, supported filesystems and the maximum number of process groups, and one directory for ipv6, the contents depend on the kernel configuration.
Since the kernel can be different between the inside and outside of a VM, emulation is required for the correct execution of the update manager.
For the fourth file, process status, it is also necessary that the update manager examines processes running inside a VM.
In contrast, the remaining two directories do not require emulation.

17. Related Work Nüwa [Zhou+ '10] Aufs-based upgrade in Ubuntu
Enable offline execution of package scripts
Several updates fail due to incomplete emulation
Aufs-based upgrade in Ubuntu
Simulate and test release upgrades
No mechanism for merging updates
VMST [Fu+ '12]
Provide emulation environments as dedicated VMs
Need as many dedicated VMs as updated VMs
Nuwa is a tool for offline update of VMs.
It enables offline execution of package scripts as much as possible by rewriting scripts.
However, several updates fail due to incomplete emulation.
Aufs-based upgrade in Ubuntu can simulate and test release upgrades.
aufs is one implementation of the union filesystem.
If the upgrade does not work well, it is easily canceled by simply removing the upper layer of aufs.
This tool does not have a mechanism for merging the upper and lower layers.
VMST enables accurate emulation outside VMs.
Unlike OUassister, it provides emulation environments as dedicated VMs.
Therefore, it needs to prepare as many dedicated VMs as updated VMs.

18. Conclusion
We proposed OUassister for consistent offline update of suspended VMs
Accurately emulate security updates of VMs offline
Extract updated files and scripts
Apply the emulation results to VMs online
Achieved shorter online update time
Future work
Examine how many security updates are needed
Apply OUassister to other distributions and virtualized systems
In conclusion, we proposed OUassister for enabling consistent offline update of suspended VMs.
OUassister emulates security updates of VMs offline more accurately using VM introspection.
Then, it extracts updated files and scripts.
When VMs are resumed and become online, OUassister just applies the emulation results to the VMs.
As a result, it achieved shorter online update time.
One of our future work is to examine how many security updates are needed in certain periods.
The number affects the online update time of OUassister.
Another direction is to apply OUassister to other distributions, for example, CentOS.
Also, we plan to apply it to other virtualized systems such as KVM.