Consistent Offline Update of Suspended Virtual Machines in Clouds I'm Kenichi Kourai from Kyushu Institute of Technology. I'm gonna talk about Consistent Offline Update of Suspended Virtual Machines in Clouds. Kenichi Kourai and Yuji Shiota Kyushu Institute of Technology, Japan

Security Updates Software vulnerabilities are increasing 16,555 in 2018 [CVE] Security updates are indispensable Download software packages from the Internet Install files in the packages Execute package scripts Recently, software vulnerabilities are increasing. Ten years ago, the number of reported vulnerabilities was about 6,000. Last year, that was more than 16,000. Therefore, security updates are indispensable. For security updates, first, the update manager downloads software packages from the Internet. Then, it extracts files from the packages and installs them to the system. In addition, it executes package scripts before and after the installation and uninstallation if necessary. Internet update manager

Unused VMs in Clouds There exist many unused virtual machines (VMs) in IaaS clouds [Zhou+ '10] Unused VMs are stopped or suspended Suspending VMs is desirable for quick resumption Many vulnerabilities are found for unused VMs In Infrastructure-as-a-Service clouds, users can easily create their virtual machines and install various software such as favorite operating systems and servers. They run such VMs only when necessary. Therefore, there exist many unused VMs in IaaS clouds. Unused VMs are usually stopped or suspended because only running VMs are charged in IaaS clouds. Suspending VMs is more desirable than stopping VMs because VMs can be resumed quickly. When a VM is suspended, its current state is saved to storage. This is analogous to hibernation of physical machines. In either case, while VMs are not running, many vulnerabilities are often found in installed software. running VM stopped VM suspended VM vuln. save

Risks of Online Update It is at high risk to apply security updates after VMs become online The VMs suffer from attacks immediately Need to download packages from the Internet It takes a long time to update the VMs completely Such unused VMs can be booted or resumed after a long time. After they become online, it is at high risk to apply security updates. If attacks against found vulnerabilities are widely spread, the VMs suffer from the attacks immediately after they are connected to the Internet. Since VMs have to connect to the Internet and download software packages, it is difficult to prevent attacks via the Internet. If there are many security updates, it takes a long time to update the VMs completely. The probability of attacks becomes higher. Internet vuln. update manager download attack resumed VM

Offline Update Update VMs in an isolated execution environment [Microsoft] Need to permit cloud admins to do this Modify the disks of VMs directly [Zhou+ '10] Not applicable to suspended VMs Often corrupt the disks by cache inconsistency To solve this issue of online update, offline update of VMs has been proposed. One approach is to update VMs in an execution environment isolated from the Internet. In advance, it downloads security updates to a local server. Then, it boots or resumes VMs in the execution environment and runs the update managers in the VMs. However, users need to permit cloud administrators to do this. The other approach is to directly modify the virtual disks of VMs. It downloads security updates and writes extracted files to the disks outside VMs. This approach enables offline update of stopped VMs, but it is not applicable to suspended VMs. If it modifies the disks of suspended VMs, the disks are often corrupted by cache inconsistency. Internet update manager download modify disk stopped VM

OUassister Enable consistent offline update of suspended VMs Emulate security updates to a VM offline Keep the integrity of the virtual disk Apply the emulation results to the VM online The VM itself updates its disk consistently To enable consistent offline update of suspended VMs, we propose OUassister. OUassister emulates security updates to a VM offline. It prevents the update manager from directly modifying the virtual disk of the VM. Therefore, the integrity of the virtual disk is kept. Even if this offline task takes time, that does not increase the risk because the VM is kept offline. When the VM is resumed and becomes online later, OUassister just applies the emulation results to the VM. Since the VM itself updates its virtual disk consistently, the integrity of the virtual disk is kept. This online task is much less than traditional online update. offline task online task update manager emulation results emulate updates transfer disk resumed VM

Emulation Environment Construct an emulation environment outside a target VM Execute the update manager as in the VM Download and install packages Provide access to the virtual disk of the VM Read package information and write updated files For the update emulation, OUassister constructs an emulation environment outside a target VM. In the emulation environment, it executes the update manager as if the update manager ran inside the VM but more securely. The update manager first downloads software packages from the Internet and then installs them as usual. To enable this, OUassister provides access to the virtual disk of the VM. The update manager can read information on installed packages in the VM and write updated files. However, as mentioned before, actual writes can corrupt the virtual disk if the VM is suspended. update manager emulate access disk suspended VM emulation environment

Shadow Filesystem Use the union filesystem to prevent the corruption of the virtual disk Layered filesystem stacking two filesystems Upper layer: an empty filesystem for updated files Lower layer: the filesystem used in the virtual disk Give an illusion of updating the virtual disk To prevent the corruption of the virtual disk, OUassister provides the shadow filesystem using the union filesystem. The union filesystem can create a layered filesystem by stacking two filesystems. In the shadow filesystem, the upper layer is an empty filesystem for writing updated files. The lower layer is the filesystem used in the virtual disk. When the update manager reads files from the shadow filesystem, it accesses the lower layer. In contrast, when it writes files to the shadow filesystem, it accesses the upper layer. Using the shadow filesystem, OUassister gives an illusion of updating the virtual disk to the update manager. upper layer lower layer disk

Accurate Emulation Provide the shadow proc filesystem Return dynamic system information on the OS in the target VM E.g., pseudo files for configs, process status, etc. Use VM introspection Analyze OS data structure in the memory of a VM For more accurate emulation, OUassister provides the shadow proc filesystem. This filesystem returns dynamically generated system information on the operating system in the target VM. For example, it provides pseudo files containing system configurations, process status, and so on. To obtain necessary information in the VM and construct the shadow proc filesystem, OUassister uses VM introspection. VM introspection is a technique for securely accessing the internal state of a VM from the outside. It analyzes operating system data structure in the memory of a VM. update manager VM introspection procfs shadow procfs VM emulation environment

Extracting Updated Files Extract updated files using the union filesystem Create a list of removed files Removed files are recorded as whiteout files Create an archive of updated files Not include whiteout files list of remoted files After the update emulation, OUassister extracts updated files using the union filesystem. First, it creates a list of removed files. In the union filesystem, removed files are recorded in the upper layer as special files called whiteout files. A whiteout file indicates that the corresponding file in the lower layer is removed. Then, OUassister creates an archive of updated files recorded in the upper layer. At this time, whiteout files are not included in the archive. upper layer whiteout archive lower layer

Extracting Package Scripts Hook the execve system call Obtain the script paths and save the script files Execute no scripts offline Extract 4 types of scripts from packages Pre-installation, pre-removal, post-installation, and post-removal in Ubuntu During update emulation, OUassister extracts package scripts executed by the update manager. To do so, it hooks the execve system call issued by the update manager. Then, it obtains the script paths and saves the script files. Since the update manager cannot execute the scripts offline, OUassiter rewrites the path specified in the system call. Script extraction is a task specific to the package management system. For Ubuntu packages, OUassister extracts four types of scripts. Pre-installation and pre-removal scripts are executed before installation and uninstallation, respectively. For example, theses scripts stop servers. Post-installation and post-removal scripts are executed after them, respectively. These scripts update several databases, for example. pre-installation file extraction/ removal post-installation pre-removal post-removal

Applying Emulation Results Apply the results of offline emulation after the target VM is resumed Extract the archive of updated files inside the VM Remove files from the disk Execute the saved scripts in a saved order Temporarily disconnect the VM from the Internet OUassister applies the results of offline emulation to the virtual disk of a target VM after the VM is resumed. First, OUassister transfers the archive of updated files and the list of removed files to the VM. Then, it extracts the archive and removes the specified files inside the VM. Also, OUassister executes saved package scripts in a saved order. Before extracting the archive, it executes saved pre-installation and pre-removal scripts. After removing the files, it executes saved post-installation and post-removal scripts. To prevent the resumed VM from being attacked while applying emulation results, OUassister temporarily disconnects the VM from the Internet. Internet list of remoted files archive disk resumed VM

Experiments We executed 4 types of software updates Comparison Install/uninstall the nginx Web server Update the OpenSSL library Update the package list Comparison Traditional online update inside a VM We conducted several experiments to examine the effectiveness of offline update in OUassister. We executed four types of software updates using the apt command: installing the nginx Web server, uninstalling it, updating the OpenSSL library, and updating the package list. For comparison, we executed traditional online update inside a VM. We used Xen, ran a VM on top of it, and ran Ubuntu in the VM. During the offline task in OUassister, we suspended the VM. host CPU: Intel Xeon E5630 Memory: 6 GB HDD: 250 GB Network: Gigabit Ethernet Xen 4.1.3 VM vCPU: 1 Memory: 512 MB Disk: 4 GB Ubuntu 12.04 LTS

Update Time Online update time was reduced 11-57% of traditional online update Breakdown of the total update time The offline tasks occupied a large portion First, we measured the online update time, which is the time needed for applying software updates inside a running VM. For OUassister, the online update time is the time for only the online task performed after a VM is resumed. As shown in the left-hand side figure, the online update time was reduced successfully in OUassister. That was 11 to 57% of traditional online update. Next, we measured the total update time, which is the sum of the offline and online update time, when we used OUassister. The right-hand side figure shows the time with its breakdown. The emulation time occupied the largest portion of the total update time. The archive time was proportional to the size of updated files.

Extracted Files/Scripts Updated files included various data The data, cache, and log of apt and man database Various numbers of scripts were executed 6 packages were installed/uninstalled for nginx We examined the files and scripts extracted after update emulation. The left-hand side figure shows the number of updated files. Updated files included the data, cache, and log of the package management system and the database for man as well as files contained in the updated packages. The right-hand side figure shows the breakdown of extracted scripts. The number of scripts depended on that of updated packages. Six packages were installed and uninstalled for nginx. Only one package was updated for openssl.

Access to Shadow Proc Filesystem Necessary for the correct execution of the update manager OS configuration (1-3) Process status (4) # file description 1 /proc/filesystems supported filesystems 2 /proc/sys/kernel/ngroups_max max of process groups 3 /proc/sys/net/ipv6/ related to IPv6 4 /proc/[pid]/stat process status 5 /proc/1/root/ link to the init process 6 /proc/self/ link to the update manager To show that our accurate emulation is necessary, we examined accessed files in the shadow proc filesystem. For the first two files, supported filesystems and the maximum number of process groups, and one directory for ipv6, the contents depend on the kernel configuration. Since the kernel can be different between the inside and outside of a VM, emulation is required for the correct execution of the update manager. For the fourth file, process status, it is also necessary that the update manager examines processes running inside a VM. In contrast, the remaining two directories do not require emulation.

Related Work Nüwa [Zhou+ '10] Aufs-based upgrade in Ubuntu Enable offline execution of package scripts Several updates fail due to incomplete emulation Aufs-based upgrade in Ubuntu Simulate and test release upgrades No mechanism for merging updates VMST [Fu+ '12] Provide emulation environments as dedicated VMs Need as many dedicated VMs as updated VMs Nuwa is a tool for offline update of VMs. It enables offline execution of package scripts as much as possible by rewriting scripts. However, several updates fail due to incomplete emulation. Aufs-based upgrade in Ubuntu can simulate and test release upgrades. aufs is one implementation of the union filesystem. If the upgrade does not work well, it is easily canceled by simply removing the upper layer of aufs. This tool does not have a mechanism for merging the upper and lower layers. VMST enables accurate emulation outside VMs. Unlike OUassister, it provides emulation environments as dedicated VMs. Therefore, it needs to prepare as many dedicated VMs as updated VMs.

Conclusion We proposed OUassister for consistent offline update of suspended VMs Accurately emulate security updates of VMs offline Extract updated files and scripts Apply the emulation results to VMs online Achieved shorter online update time Future work Examine how many security updates are needed Apply OUassister to other distributions and virtualized systems In conclusion, we proposed OUassister for enabling consistent offline update of suspended VMs. OUassister emulates security updates of VMs offline more accurately using VM introspection. Then, it extracts updated files and scripts. When VMs are resumed and become online, OUassister just applies the emulation results to the VMs. As a result, it achieved shorter online update time. One of our future work is to examine how many security updates are needed in certain periods. The number affects the online update time of OUassister. Another direction is to apply OUassister to other distributions, for example, CentOS. Also, we plan to apply it to other virtualized systems such as KVM.