1. UVBond: Strong User Binding to VMs for Secure Remote Management in Semi-Trusted Clouds
I'm Kenichi Kourai from Kyushu Institute of Technology.
I'm gonna talk about UVBond: Strong User Binding to VMs for Secure Remote Management in Semi-Trusted Clouds.
This is joint work with my student, who has graduated.
Keisuke Inokuchi and Kenichi Kourai
Kyushu Institute of Technology, Japan

2. Remote Management in Clouds
IaaS clouds provide users with virtual machines (VMs)
Provide remote management of VMs
Users manage their VMs via the management servers
Connect to the server and manipulate VMs
E.g., boot, shutdown, migration, and out-of-band remote management
Infrastructure-as-a-Service clouds provide users with virtual machines.
Users can install their own operating systems and applications as they like.
Clouds provide remote management of VMs via web interfaces and APIs.
Users manage their VMs via the management servers provided by clouds.
They first connect to the server and then manipulate their VMs.
For example, the management server has the ability for booting new VMs, shutting down running VMs, and migrating VMs to other hosts.
In addition, users can use a management method called out-of-band remote management and log in to VMs without relying on servers inside the VMs.
management server VM
user
cloud

3. Untrusted Cloud Operators
Not all the cloud operators are trusted in semi-trusted clouds
28% of cybercrimes are caused by insiders [PwC'14]
35% of admins have accessed sensitive information [CyberArk'09]
Untrusted cloud operators can abuse the management server and its ability
The management server is managed by cloud operators, but not all the operators are trusted in semi-trusted clouds.
In semi-trusted clouds, the providers are trusted but some of the operators may be untrusted.
In fact, it is reported that 28% of cybercrimes are caused by insiders.
Malicious operators actively attack VMs.
Also, it is revealed that 35% of system administrators have accessed sensitive information without authorization.
Curious but honest operators may eavesdrop on attractive information that they can easily obtain from VMs.
As such, untrusted cloud operators can abuse the management server and its ability.
management server VM
operators

4. Issue 1: Illegal Command Execution
Cloud operators can execute arbitrary management commands to VMs
Perform out-of-band remote management
Access virtual serial consoles and GUI consoles of VMs
Perform VM introspection
Access the memory, disks, and network of VMs
Eavesdrop on and tamper with sensitive information
First, untrusted cloud operators can execute arbitrary management commands to VMs.
In particular, using out-of-band remote management, they can access virtual serial consoles and virtual GUI consoles of VMs by directly accessing virtual devices of VMs.
They can log in to user's VMs and eavesdrop on and tamper with sensitive information inside them.
In addition, using a technique called VM introspection, cloud operators can directly access the memory, disks, and network of VMs.
They can obtain and modify sensitive information included in these resources. VM
illegal
command
operators

5. Issue 2: VM Redirection Attack
Cloud operators can redirect user's commands to malicious VMs
Redirect out-of-band remote management
Steal login passwords by a malicious login program or a key logger
Redirect VM introspection
Return fake information even for compromised VMs
Second, untrusted cloud operators can redirect user's commands to their malicious VMs.
We call this attack the VM redirection attack.
For example, they can steal login passwords in out-of-band remote management by using a malicious login program or a key logger installed in the malicious VMs.
Since these malicious activities are done inside VMs, they are difficult to prevent even if console input and output are encrypted end to end.
In addition, the VM redirection attack can be used for preventing users from using VM introspection and detecting malicious activities inside VMs.
If cloud operators prepare a VM with a legitimate memory image and return fake memory data to users, users are fooled as normal even when their VMs are compromised.
redirect
command
management server
user's VM
malicious VM
user

6. The Root Cause
The binding of users to their VMs is weak in traditional clouds
Cloud operators can access user's VMs
Necessary privilege for cloud management
Users can access cloud operator's VMs
Difficult to distinguish VMs
The root cause of these issues is that the binding of users to their VMs is weak in traditional clouds.
Since only users are not bound to their VMs, cloud operators can freely access any user's VMs.
This privilege is necessary for cloud management, but it should be restricted.
Conversely, users can access cloud operator's VMs if cloud operators allow that.
When cloud operators prepare VMs similar to user's, it is difficult for users to distinguish those VMs.
operator's VM
user's VM
operators
user

7. UVBond Strongly bind users to their VMs in the trusted hypervisor
Securely boot user's VM in semi-trusted clouds
Issue a VM descriptor to the user after the boot
Permit only the user to execute commands only to the VM corresponding to the VM descriptor
Prevent illegal command execution and VM redirection attacks
We propose UVBond, which strongly binds users to their VMs in the trusted hypervisor.
The hypervisor is software running underneath VMs and the management server.
First, UVBond securely boots user's VM in semi-trusted clouds.
After the boot, UVBond issues a VM descriptor to the user.
Then, it permits only the user to execute commands only to the VM corresponding to the VM descriptor.
Cloud operators cannot execute any illegal commands to user's VMs.
They cannot perform the VM redirection attack or forward user's commands to their malicious VMs.
command
management server
user's VM
malicious VM
operators
user
hypervisor

8. Threat Model The hypervisor are trusted
By remote attestation with TPM at boot time
By security checks with hardware at runtime
Cloud operators are not trusted
Abuse privileged components including the management server
We assume that only the hypervisor and hardware are the trusted computing base.
To trust hardware, we assume that cloud providers themselves are trusted.
This assumption is widely accepted.
The trustworthiness of the hypervisor can be confirmed by various techniques.
At boot time, remote attestation with TPM guarantees that the hypervisor is booted correctly.
At runtime, security checks with hardware can detect attacks against the hypervisor.
In contrast, we do not trust cloud operators or privileged components managed by them.
Privileged components include the management server and virtual devices used for out-of-band remote management.
We assume that untrusted cloud operators can abuse privileged components.
management server
virtual
devices VM
operators
hypervisor

9. VM Boot with UVBond
Share a disk encryption key between a user and the hypervisor
The hypervisor associates the key with a created VM
Boot the VM by decrypting its encrypted disk in the hypervisor
Guarantee that user's own VM is booted with the correct disk
In UVBond, first, a user securely shares his disk encryption key with the hypervisor at the boot time of his VM.
Then the hypervisor associates the key with the VM.
Using the registered disk encryption key, UVBond boots the VM by decrypting its encrypted disk inside the hypervisor.
Since the VM cannot be correctly booted using the other disks, it is guaranteed that user’s own VM is booted with the correct disk.
Cloud operators can still boot their VM with a malicious disk and its encryption key instead of user’s VM.
To prevent this attack, UVBond enables the user to confirm that his disk encryption key is correctly registered to the hypervisor.
encrypted
disk
VM boot
management server VM
disk encryption
key
user
hypervisor
decrypt

10. VM Descriptors
Issue a VM descriptor to the user after the boot of a VM
The hypervisor associates the descriptor with the VM
Securely send the descriptor by encrypting it
The user specifies the descriptor on executing commands to his VM
The hypervisor permits access to the VM only when the descriptor matches the VM
After the boot of user’s VM, UVBond issues a VM descriptor to the user.
The hypervisor associates the descriptor with the VM.
It encrypts the generated descriptor and securely sends it to the user via the untrusted management server.
When the user executes commands to his VM, he specifies the VM descriptor.
On the basis of the descriptor, the hypervisor determines whether the specified command can be executed to the specified VM or not.
It permits access to the VM only when the descriptor matches the VM.
command
management server VM
VM descriptor VM
descriptor
user
hypervisor

11. VM Migration with UVBond
Obtain the disk encryption key from the hypervisor
Transfer it to the destination host
It is encrypted by the public key of the destination hypervisor
Re-register the key to the destination hypervisor
The key is decrypted by the private key of that hypervisor
Users can use the same VM descriptor
When a VM is migrated, the disk encryption key should be registered again to the hypervisor at the destination host.
However, the user cannot manually register his key because he just sends the migrate command only to the source host.
So, at the source host, UVBond obtains the disk encryption key from the hypervisor and transfers it to the destination host.
The key is encrypted by the public key of the destination hypervisor.
At the destination host, UVBond automatically registers the received key to the hypervisor again.
The key is decrypted by the private key of that hypervisor.
Note that the user can use the same VM descriptor as before VM migration.
VM migrate
management server VM VM
transfer
user
hypervisor
hypervisor

12. Semantic Gap
The hypervisor cannot directly recognize user's high-level commands
It can recognize only low-level hypercalls issued by commands
Each command usually consists of a set of hypercalls
Difficult to associate a VM descriptor only with correct hypercalls
The hypervisor could not detect hypercall injection
A VM descriptor should be able to control the execution of each command completely, but this is not easy.
The hypervisor cannot directly recognize user's high-level commands.
It can recognize only low-level hypercalls issued to the hypervisor by commands.
If each command is exactly equivalent to one hypercall, the hypervisor can securely execute the hypercall to the VM corresponding to the descriptor by combining the hypercall and the descriptor.
However, each command usually consists of a set of hypercalls and the other tasks that cannot be executed inside the hypervisor.
So, it is difficult to associate a VM descriptor only with the hypercalls issued by the specified command.
For example, the hypervisor could not detect illegal hypercalls injected by untrusted cloud operators.
VM descriptor VM
command
hypercall
user
hypervisor

13. Hypercall Automaton Identify each command by a sequence of hypercalls
Different commands may have the same sequence
The same sequence means the same effect on VM management
Use a finite state automaton that accepts all the possible sequences for each command
One command can have various sequences
command
state
transition
pause 9 8
unpause 10
mem-set 15
shutdown
destroy 12 13
save 20 21
To bridge this semantic gap between high-level commands and low-level hypercalls, UVBond identifies each command by a sequence of hypercalls.
Different commands may have the same hypercall sequence, but we consider that the same sequence means the same effect on VM management.
For each command, UVBond uses a finite state automaton that accepts all the possible sequences of hypercalls issued by the command, as shown in the figure.
This is called a hypercall automaton.
For example, this automaton transits from state 10 to 11 by the memory_op hypercall.
In general, one command have various hypercall sequences.
The size of the hypercall automata for several commands is shown in this table.
The save command has the most complex hypercall automaton.
xen_version
memory_op
memory_op
sysctl
sysctl 1 8 9 10 11 12
domctl
domctl x8
memory_op
sysctl 13 14 15

14. Automaton-based Permission
Permit access to a VM as long as a hypercall sequence is not rejected by a hypercall automaton
A user specifies a hypercall automaton as well as a VM descriptor
Allow even cloud operators to execute some of the commands without a VM descriptor
Trade-off between ease of management and security
When a user sends a command to the management server, he specifies a hypercall automaton as well as a VM descriptor.
The hypervisor permits access to the VM corresponding to the descriptor as long as a hypercall sequence issued by the command is not rejected by the hypercall automaton.
Using hypercall automata, UVBond allows even cloud operators to execute some of the commands without a VM descriptor.
For this purpose, a user registers the corresponding hypercall automata to his VM in advance.
While the commands corresponding to the registered hypercall automata are executed, cloud operators can access the VMs.
Users can determine permitted commands at their discretion and take a trade-off between ease of management and security.
VM descriptor VM
command
hypercall
automaton
hypercall
user
hypervisor
hypercall
automaton

15. Simultaneous Command Execution
Hypercalls can be simultaneously issued by multiple commands
Need to apply a different hypercall automaton for each command
Handle a hypercall sequence per command
Distinguish commands using processes executing them
Identify each process by its page tables [Jones+ ATC'06]
Since multiple users run VMs in one host, hypercalls can be simultaneously issued by multiple commands.
So, it is necessary to apply a different hypercall automaton for each command.
UVBond handles a hypercall sequence per command.
However, as I mentioned before, the hypervisor cannot recognize commands.
So, UVBond distinguishes commands using processes executing them.
The process is an abstraction of the operating system, but the hypervisor can identify each process by its page tables.
When a hypercall automaton is registered to a VM, the hypervisor associates the current page tables with the automaton.
When a hypercall is issued, the hypervisor searches for the hypercall automaton corresponding to the current page tables and applies it.
command 1
command 2
page
table 1
page
table 2
hypervisor
hypercall
automaton 1
hypercall
automaton 2

16. Preventing Information Leakage
Secure out-of-band remote management
The hypervisor has to encrypt I/O data [Egawa+ CloudCom'12]
UVBond prevents its key from being registered to a malicious VM
Secure VM introspection
UVBond allows only a user to map his VM's memory
The hypervisor has to encrypt the memory data [Kourai+ CLOUD'16]
In UVBond, the hypervisor can control access to VMs, but the access control of hypercalls is not often sufficient.
For out-of-band remote management, cloud operators can eavesdrop on console input and output without issuing hypercalls after the connection is established.
So, the hypervisor has to encrypt the I/O data between a remote client and the hypervisor.
UVBond can prevent its encryption key from being registered to a malicious VM by the VM redirection attack.
For VM introspection, UVBond allows only a user to map the memory of his VM onto his monitor process.
However, cloud operators can access the process memory without using hypercalls after that.
Therefore, the hypervisor has to encrypt the memory data and send it to a trusted remote host.
encrypted I/O
en/decrypt
management server VM
encrypted
memory data
remote host
hypervisor

17. Encryption of Para-virtualized Disk I/O
Paravirtual disk drivers are often used for performance
Consist of front-end and back-end drivers
Share grant pages between them for I/O buffers
The hypervisor en/decrypts grant pages
Monitor the communication between the two drivers
Even cloud operators can read decrypted data in grant pages
To access virtual disks of VMs, paravirtual disk drivers are often used in VMs because of performance improvement.
Therefore, the hypervisor has to support disk encryption for para-virtualization.
However, this is not easy because the hypervisor cannot trap all accesses to virtual disks.
The paravirtual disk driver consists of the front-end driver running in a VM and the back-end driver outside the VM.
These drivers share memory called grant pages used for I/O buffers and send and receive I/O data.
In UVBond, the hypervisor monitors the communication between the two drivers and encrypts and decrypts grant pages.
However, even cloud operators can read decrypted data in grant pages after decryption or before encryption by the hypervisor. VM
backend
driver
frontend
driver
encrypted
disk
grant
page
en/decrypt
hypervisor

18. Duplication of Grant Pages
The hypervisor duplicates grant pages
Only a guest grant page can hold decrypted data
A shadow grant page always holds encrypted data
The hypervisor synchronizes the two types of grant pages
Encrypt data in a guest grant page and stores it to a shadow one
Decrypt data in a shadow grant page and stores it to a guest one
To prevent this attack, the hypervisor duplicates grant pages.
The original grant page in a VM is called a guest grant page.
The corresponding page provided to the back-end driver is called a shadow grant page.
Only a guest grant page can hold decrypted data.
A shadow grant page always holds encrypted data.
The hypervisor synchronizes the two types of grant pages.
When a VM writes data to the disk, the hypervisor encrypts data written to a guest grant page and stores it to the corresponding shadow one.
When a VM reads data from the disk, it decrypts data read to a shadow grant page and stores it to the corresponding guest one. VM
backend
driver
shadow
grant
page
decrypt
guest
grant
page
frontend
driver
encrypted
disk
encrypt
hypervisor

19. Using AES-NI in the Hypervisor
Using XMM registers causes a hardware exception
The hypervisor defers the restoration of XMM registers on CPU scheduling
Temporarily disable a hardware exception
Save XMM registers before the use
Restore them after the use
Using a CPU instruction set called AES-NI is necessary to reduce the overhead of encryption.
However, special treatment was required to use AES-NI in the hypervisor.
AES-NI needs to use XMM registers, which causes a hardware exception in the hypervisor.
This is because the hypervisor defers the restoration of XMM registers on CPU scheduling and restores them on the exception.
But it is not assumed that a hardware exception occurs inside the hypervisor.
To prevent this exception, UVBond temporarily disable a hardware exception just before using AES-NI.
Instead, UVBond saves XMM registers before the use and restores them after the use.
UVBond
hypervisor
1. save
3. restore
2. use
XMM
registers

20. Experiments
We conducted experiments to confirm the effectiveness of UVBond
Investigate the execution of management commands
Measure the boot and migration time
Examine disk I/O performance
vCPU: 2
Memory: 2 GB
HDD: 20 GB
We conducted several experiments to confirm the effectiveness of UVBond.
First, we investigated the execution of management commands.
Seconds, we measured the time needed for booting and migrating a VM.
Finally, we examined the performance of disk I/O with encryption.
In these experiments, we used Xen 4.4 modified for UVBond.
For comparison, we used vanilla Xen without modification. VM
CPU: Intel Xeon E v2
Memory: 8 GB
HDD: 1 TB
Gigabit Ethernet

21. Command Execution with Hypercall Automata
We executed commands with various VM descriptors and hypercall automata
The command execution succeeded only when both were correct
We measured the execution time of commands
The overhead was 4 ms for short-time commands
Negligible overhead for long-time ones
long-time commands
First, we examined that UVBond could detect illegal commands.
We executed commands with various VM descriptors and hypercall automata.
As a result, we confirmed that commands could be successfully executed only when both the VM descriptor and the hypercall automaton matched the target VM and the command, respectively.
Next, we examined the execution time of commands.
The left figure shows the execution time of short-time commands and the right figure shows that of long-time ones.
For short-time commands, the overhead of UVBond was 4 ms, which included the registration and runtime check of a hypercall automaton.
In contrast, for long-time commands, the overhead was negligible because the number of issued hypercalls was relatively small.
short-time commands

22. Performance of a VM Boot
We measured the time from the execution of the create command to OS boot-up
With/without the page cache for VM's disk
The boot time was 6 sec longer in UVBond
Due to the overhead of additional operations including encryption
We examined the boot time of a VM to confirm the overhead of UVBond.
We measured the time from when we executed the create command until the operating system was booted up in the VM.
Since the page cache affected the disk performance of the VM largely, we measured the time with or without the page cache for VM's disk.
As shown in the figure, the boot time was 6 seconds longer in UVBond than in vanilla Xen without depending on the page cache.
This is due to the overhead of additional operations performed by UVBond, including disk encryption.

23. Performance of VM Migration
Migration time
We measured the execution time of the migrate command
UVBond was 1.4 sec longer than vanilla Xen
Downtime
We measured the time during which the VM stopped
UVBond was only 50 ms longer
We examined the performance of VM migration using UVBond.
We measured the migration time, which was the time from when we executed the migrate command until it was completed.
The left figure shows the result.
The migration time in UVBond was 1.4 seconds longer than that in vanilla Xen because UVBond had to securely transfer the disk encryption key.
Also, we measured the downtime, which was the time from when the VM was paused at the source host until it was resumed at the destination host.
As shown in the right figure, the downtime in UVBond became only 50 ms longer.

24. Disk I/O Performance
We measured the throughput and the latency using fio
Compared with dm-crypt inside a VM
The performance degradation was less than 10%
Thanks to AES-NI
Comparable to dm-crypt
Finally, we examined the disk I/O performance using the fio benchmark.
In addition to UVBond with or without AES-NI and vanilla Xen, we used the system using Linux dm-crypt in a VM.
dm-crypt encrypts and decrypts disks at the operating system level.
We measured read and write performance of sequential and random access.
The left figure shows the throughput and the right one shows the latency.
Compared with vanilla Xen, the throughput in UVBond degraded only by 3 to 10% and the latency increased only by 0.8 to 1.3 ms thanks to AES-NI.
The throughput in UVBond was comparable to or even better than that in dm-crypt, while the latency was slightly longer.

25. Related Work Self-service cloud [Butt+ CCS'12][Butt+ SoCC'14]
User's VMs can be securely managed in his own privileged VM
The TCB is quite large
BitVisor [Shinagawa+ VEE'09]
Encrypt the disk of a VM using a parapass-through driver
Support only fully virtualized OSes
System-call automata [Hofmeyr+ Comp.Sec.'98]
Detect intrusion on the basis of a sequence of system calls
Hypercall automata are its application to the hypervisor
Self-service cloud can prevent cloud operators from illegally accessing users’ VMs.
Since user’s VMs can be securely managed in his own privileged VM, cloud operators cannot eavesdrop on or tamper with the VMs.
However, its TCB is quite large because it includes not only the hypervisor but also several privileged VMs.
BitVisor can encrypt the disk of a VM using a parapass-through driver in the trusted hypervisor.
The hypervisor intercepts only minimum hardware access needed for disk encryption.
Unlike UVBond, BitVisor supports only fully virtualized operating systems and cannot allow paravirtual disk drivers in a VM.
System-call automata are used for intrusion detection systems.
Such IDSes detect intrusion on the basis of a sequence of system calls issued by an application.
Our hypercall automata used in UVBond are its application to the hypervisor.

26. Conclusion
We proposed UVBond for providing strong user binding to VMs in semi-trusted clouds
Boot user's VM by decrypting its encrypted disk inside the hypervisor
Securely execute management commands to his VM
With a VM descriptor and hypercall automata
Prevent illegal command execution and VM redirection attacks
Future work
Apply UVBond to large cloud management systems, e.g., OpenStack
Need to extract commands and create their hypercall automata
Need to modify the web interface and API
In conclusion, we proposed UVBond for providing strong user binding to VMs in semi-trusted clouds.
UVBond enables only a user to boot his VM by decrypting its encrypted disk inside the trusted hypervisor.
Then it allows the user to securely execute management commands to his VM with an issued VM descriptor and hypercall automata.
As such, UVBond prevents untrusted cloud operators from executing illegal commands to user’s VMs or redirecting user’s commands to their malicious VMs.
Our future work is to apply UVBond to large cloud management systems such as OpenStack.
To support UVBond in such systems, we have to extract used commands and create their hypercall automata.
Also, we need to modify the Web interface and API so as to send VM descriptors and hypercall automata as well as commands.