1. Secure Out-of-band Remote Management of Virtual Machines with Transparent Passthrough
I'm Kenichi Kourai from Kyushu Institute of Technology.
I'm gonna talk about Secure Out-of-band Remote Management of Virtual Machines with Transparent Passthrough.
This is join work with my students.
Shota Futagami, Tomoya Unoki, and Kenichi Kourai
Kyushu Institute of Technology, Japan

2. Remote Management of VMs
IaaS clouds
Run virtualized systems and provide virtual machines (VMs)
Out-of-band remote management
Enable users to manage VMs via virtual devices
E.g., virtual serial devices, keyboards, and video cards
Even on network configuration errors and at boot time
Infrastructure-as-a-Service clouds run virtualized systems and provide virtual machines to users.
Users can use a necessary number of VMs to construct large systems.
To manage their VMs provided by clouds, users access their VMs from remote hosts using remote management software such as SSH and VNC.
In addition to a management method for directly accessing VMs, clouds provide another method called out-of-band remote management.
This management method enables users to indirectly access VMs via virtual devices of their VMs, for example, virtual serial devices, keyboards, and video cards.
This method allows users to manage the systems in VMs even on their network configuration errors and at boot time.
virtualized
system
I/O
virtual
devices
user VM
user
cloud

3. Information Leakage by Insiders
Virtual devices may be managed by untrusted cloud operators
28% of cybercrimes are caused by insiders [PwC'14]
35% of admins have accessed sensitive information [CyberArk'09]
Cloud operators can easily eavesdrop on and tamper with I/O of out-of-band remote management
In clouds, virtual devices in the virtualized system may be managed by untrusted cloud operators.
Cloud operators are system administrators responsible for managing the whole virtualized systems.
Unfortunately, not all the cloud operators are trusted.
It is reported that 28% of cybercrimes are caused by insiders.
Also, it is revealed that 35% of system administrators have accessed sensitive information without authorization.
Untrusted cloud operators can easily eavesdrop on and tamper with I/O of out-of-band remote management via virtual devices of user VMs.
For example, login passwords typed by users may be stolen and illegal commands may be injected into user VMs.
Cloud operators should be granted only the least privilege, but this is often difficult.
virtualized
system
operators
user VM
password
virtual
devices
user

4. Previous Approaches
Several systems trusting the hypervisor have been proposed
Encrypt I/O between users and the hypervisor [Egawa+ CloudCom'12]
Virtual devices handle only encrypted data
Encrypt I/O in users' special VMs [Butt+ CCS'12]
These VMs are protected by the trusted hypervisor
To prevent such attacks, several systems trusting the hypervisor have been proposed.
The hypervisor is software running underneath VMs and virtual devices.
For example, FBCrypt encrypts I/O data of out-of-band remote management between a remote client and the hypervisor.
A remote client encrypts input data and sends it to a virtual device.
When a user VM attempts to obtain the data from that virtual device, the hypervisor decrypts it transparently.
For output, the hypervisor encrypts data and the remote client decrypts it.
Self-service cloud encrypts I/O data in users’ special VMs.
These VMs are protected by the trusted hypervisor.
A virtual device reads encrypted data and the special VM decrypts it.
Data written to a virtual device is encrypted by the special VM.
virtualized
system
operators
encrypted data
virtual
devices
user VM ?
remote client
trusted hypervisor

5. vulnerable hypervisor
Issues (1/2)
The hypervisor can be compromised by insiders
Cloud operators can abuse rich interfaces to privileged components
Not all the hypervisors are clearly separated from privileged components
KVM runs the hypervisor inside the host operating system
The entire host operating system has to be trusted
As such, the trusted computing base of these systems includes the hypervisor.
In addition, these systems depend on cryptography.
Therefore, four issues arise.
First, the hypervisor can be easily compromised by insiders.
To enable privileged components to run on top of the hypervisor, the hypervisor provides rich management interfaces.
This means that the hypervisor is tightly coupled with privileged components.
Untrusted cloud operators can abuse such interfaces.
Second, not all the hypervisors are clearly separated from privileged components.
Such separation is necessary for trusting only the hypervisor.
For example, KVM runs the hypervisor inside the host operating system, so it is
difficult to separate only the hypervisor.
As a result, the entire host operating system has to be trusted, but this increases the size of the TCB.
operators
privileged
components
privileged
components
Linux + KVM
virtualized
system
vulnerable hypervisor

6. Issues (2/2)
The entire virtualized system cannot be managed by cloud operators
Untrusted cloud operators cannot manage the hypervisor
Managing the hypervisor and the rest independently is unrealistic
Cryptography management is problematic
It is not easy to securely manage cryptographic keys
Third, the entire virtualized system cannot be managed by cloud operators.
It has to be guaranteed that untrusted cloud operators cannot manage the trusted hypervisor.
However, it is unrealistic to manage the hypervisor and the rest of the virtualized system independently.
Since the hypervisor is tightly coupled with privileged components, the virtualized system has to be updated as a whole.
Finally, cryptography management is problematic.
It is not easy to securely manage cryptographic keys.
In addition, remote management clients need to be modified to provide an encryption mechanism of I/O data.
operators
virtualized
system
modified
client
privileged
components
trusted
admins
trusted hypervisor
remote host

7. VSBypass
Achieve secure out-of-band remote management outside the virtualized system
Run shadow devices outside the virtualized system
Intercept I/O requests using transparent passthrough
Process them in shadow devices
Prevent information leakage and tampering
We propose VSBypass for achieving secure out-of-band remote management outside the virtualized system.
VSBypass runs virtual devices called shadow devices running outside the virtualized system.
It intercepts I/O requests of user VMs using transparent passthrough and securely processes them in shadow devices.
This enables users to perform out-of-band remote management without relying on the virtualized system.
So, I/O data of out-of-band remote management doesn't leak to untrusted cloud operators inside the virtualized system.
Also, cloud operators cannot tamper with the I/O data.
virtualized
system
I/O
shadow
devices
virtual
devices
user VM
remote host
transparent passthrough

8. System Architecture
Run the entire virtualized system in an outer VM (cloud VM) using nested virtualization
Shadow devices run outside the cloud VM
Remote users access shadow devices instead of virtual devices
The cloud VM and shadow devices run on top of the cloud hypervisor
cloud VM
user VM
This is the system architecture of VSBypass.
VSBypass runs he entire virtualized system in an outer VM using nested virtualization.
The outer VM is called the cloud VM.
Shadow devices run outside the cloud VM and process I/O requests of user VMs like virtual devices inside the virtualized system.
Remote users access shadow devices, instead of virtual devices, for secure out-of-band remote management.
The cloud VM and shadow devices run on top of the hypervisor called the cloud hypervisor.
To distinguish two hypervisors, we call the original hypervisor inside the virtualized system the guest hypervisor.
I/O
shadow
devices
guest hypervisor
remote host
cloud hypervisor

9. Threat Model Cloud operators may be untrusted
Have full control over the virtualized system including the hypervisor
Cloud providers are trusted
The TCB is the cloud hypervisor and shadow devices
A few trusted admins maintain the TCB
Admins form an administrative hierarchy
operators
user VM
We assume that cloud operators inside the virtualized system may be untrusted.
It is difficult to trust all of the cloud operators in large-scale clouds.
Unlike previous approaches, cloud operators have full control over the entire virtualized system including the guest hypervisor.
They can compromise any components inside the virtualized system.
In contrast, we assume that cloud providers are trusted.
This assumption is widely accepted.
Cloud providers maintain the TCB for VSBypass, which is the cloud hypervisor and shadow devices.
For this purpose, cloud providers should have a few trusted system administrators.
Many untrusted operators and a few trusted administrators form an administrative hierarchy, which is also adopted in many systems.
trusted
shadow
devices
trusted
admins
guest hypervisor
cloud hypervisor
untrusted

10. Advantages (1/2) Difficult for cloud operators to attack the TCB
The attack surface to the cloud hypervisor is much smaller
Only the hardware interface is provided
Support any virtualized systems
The entire virtualized system is virtualized using the cloud VM
VSBypass does basically not depend on the internals
VSBypass can resolve the four issues in out-of-band remote management.
First, it is more difficult for untrusted cloud operators to attack the TCB.
This is because the entire virtualized system is confined in the cloud VM.
The attack surface from the cloud VM to the cloud hypervisor is much smaller than that from privileged components to the guest hypervisor.
The cloud hypervisor provides only the hardware interface to the cloud VM, which is the clear boundary of virtualization.
Second, VSBypass supports any virtualized systems because the entire virtualized
system is virtualized using the cloud VM.
To perform out-of-band remote management, VSBypass does basically not depend on the internals of the virtualized system.
cloud VM
privileged
components
Linux + KVM
cloud VM
guest hypervisor
boundary of
virtualization
cloud hypervisor

11. Advantages (2/2)
Cloud operators can manage the entire virtualized system
VSBypass does not need to trust the guest hypervisor
It enables clearly separating the responsibility of system management
Cryptography is not necessary
VSBypass does not rely on encryption to prevent information leakage
Users can use the existing remote management software
Third, cloud operators can manage the entire virtualized system.
VSBypass doesn't need to trust the guest hypervisor inside the virtualized system.
They can update not only privileged components but also the guest hypervisor.
As a result, VSBypass enables cloud providers to clearly separate the responsibility of system management at the clear boundary of virtualization.
Finally, cryptography isn't necessary.
VSBypass doesn’t rely on encryption to prevent information leakage.
It doesn’t need to encrypt I/O data of out-of-band remote management.
So, users can use the existing remote management software.
privileged
components
cloud VM
existing
client
shadow
devices
guest hypervisor
boundary of
virtualization
remote host
cloud hypervisor

12. Shadow Devices Use virtual devices of a proxy VM as shadow devices
Assign minimum resources to a proxy VM
Pause a proxy VM after the boot
Bind a proxy VM to a user VM using EPT
Transparently perform secure out-of-band remote management by specifying proxy VM's ID
To provide shadow devices, VSBypass runs a VM called a proxy VM per user VM on top of the cloud hypervisor.
It uses virtual devices of a proxy VM as shadow devices.
A proxy VM is assigned minimum amounts of resources.
After the boot, it is paused except for virtual devices.
VSBypass binds a proxy VM to a user VM using extended page tables.
Users can transparently perform secure out-of-band remote management of a user VM by specifying the ID of the corresponding proxy VM.
VSBypass identifies a user VM using the EPT, which is created for each user VM.
This is because a user VM is an abstraction inside the virtualized system and the cloud hypervisor cannot directly identify a user VM.
user VM
proxy VM
EPT
I/O
shadow
devices
guest hypervisor
cloud VM
remote host
cloud hypervisor

13. Transparent Passthrough
Intercept an I/O request of a user VM in the cloud hypervisor
A VM exit occurs directly to the cloud hypervisor
By executing an instruction for port- and memory-mapped I/O
Not forward the request to the guest hypervisor as usual
The cloud hypervisor handles that request
To achieve transparent passthrough, VSBypass intercepts an I/O request of a user VM in the cloud hypervisor.
In traditional nested virtualization, when a user VM executes an instruction for port-mapped or memory-mapped I/O, a VM exit occurs directly to the cloud hypervisor.
The cloud hypervisor performs a VM entry into the cloud VM and emulates that instruction in the guest hypervisor.
In contrast, VSBypass doesn't forward the intercepted I/O request to the guest hypervisor.
The cloud hypervisor handles that request instead of the guest hypervisor.
cloud VM
I/O
instruction
user VM
proxy VM
shadow
devices
guest hypervisor
VM exit
cloud hypervisor

14. I/O Processing in Shadow Devices
Redirect the intercepted I/O request to a shadow device
Find a proxy VM from the trapped user VM using EPT
Send that request as if the request is issued by the proxy VM
Return the result to the user VM
Intercept a completion event to the proxy VM
Redirect that event to the cloud VM
cloud VM
The cloud hypervisor redirects the intercepted I/O request to the corresponding shadow device.
It first finds a proxy VM from the trapped user VM using EPT.
Then, it sends that I/O request to the shadow device as if that request is issued by the proxy VM.
After request processing, the shadow device returns the result to the user VM.
For this purpose, it sends a completion event to the proxy VM.
At this time, the cloud hypervisor intercepts that event and redirects it to the cloud VM.
user VM
EPT
proxy VM
shadow
devices
guest hypervisor
I/O request
event
cloud hypervisor

15. Sharing VRAM A user VM directly accesses VRAM in its main memory
The cloud hypervisor cannot intercept that access with low overhead
A shadow video card shares the VRAM of a user VM
Process the VRAM as if it were the virtual device of the user VM
Track updates of the VRAM using the log-dirty mechanism
cloud VM
A user VM not only performs memory-mapped I/O but also directly accesses the VRAM in its main memory.
Since such direct access of the VRAM doesn't cause a VM exit by default, the cloud hypervisor cannot intercept that access.
If it intercepts all accesses to the VRAM, that imposes too large overhead.
So, a shadow video card shares the VRAM of a user VM.
It can process the shared VRAM as if it were the virtual video card of the user VM inside the virtualized system.
To detect updated screen areas, the cloud hypervisor tracks updates of the VRAM using the log-dirty mechanism.
VRAM
user VM
share
shadow
video card
guest hypervisor
updated area
cloud hypervisor

16. Virtual Interrupts in Shadow Devices
Redirect virtual interrupts caused in shadow devices to a user VM
Deliver interrupts via the interrupt server in the virtualized system
Virtual interrupts include no sensitive information
Share a ring buffer using an ultracall [Miyama+ ESORICS'17]
Receive interrupts by polling
cloud VM
VSBypass redirects virtual interrupts caused in shadow devices to a user VM.
It is difficult to do that without relying on the guest hypervisor because the virtual interrupt controller is often implemented in the guest hypervisor.
So, VSBypass runs the interrupt server in the virtualized system and delivers interrupts via the server from shadow devices to a user VM.
Although the redirection of virtual interrupts depends on the untrusted virtualized system, confidentiality is maintained because virtual interrupts include no sensitive information.
To minimize communication overhead and avoid modification to the virtualized system, VSBypass shares a ring buffer between shadow devices and the interrupt server using a mechanism called an ultracall.
The interrupt server receives interrupts by polling.
interrupt
server
user VM
shadow
devices
interrupt
guest hypervisor
cloud hypervisor

17. Experiments
We conducted experiments to show the effectiveness of VSBypass
Eavesdrop on I/O of out-of-band remote management
Measure the performance of a virtual serial console and GUI remote access
Comparison with the traditional system without nested virtualization
We conducted experiments to show the e ectiveness of VSBypass.
First, we attempted to eavesdrop on I/O of out-of-band remote management.
Second, we measured the performance of a virtual serial console and GUI remote access.
For comparison, we used the traditional system without using nested virtualization.
As the cloud hypervisor, we ran Xen 4.8 modified for VSBypass.
As a virtualized system in the cloud VM, we ran Xen 4.4.
cloud VM
user VM
vCPU: 3
Memory: 4 GB
vCPU: 3
Memory: 1 GB
Gigabit Ethernet
CPU: Xeon E3-1270
Memory: 8 GB
CPU: Xeon E v2
Memory: 8 GB

18. Eavesdropping We eavesdropped on I/O data in virtual devices
Record input/output characters in the virtual serial device
Record keys in the virtual keyboard and screenshots in the virtual video card
We could not obtain any data in VSBypass
We could capture all data in the traditional system
To show that VSBypass can prevent information leakage, we attempted to eavesdrop on I/O data in virtual devices.
First, we used a virtual serial console via SSH and recorded input and output characters in the virtual serial device.
Next, we performed GUI remote management with VNC and recorded keys in the virtual keyboard and screenshots in the virtual video card.
In the traditional system, we could capture all the data, as shown in the left figure.
But in VSBypass, we couldn't obtain any data like the right figure.
captured screen
(traditional system)
captured screen
(VSBypass)

19. Performance of a Virtual Serial Console
The response time between a character input and its echo- back in an SSH client
VSBypass was 1.3 ms longer due to nested virtualization
The throughput of outputting a text file to an SSH client
VSBypass was almost the same
We measured the response time of a virtual serial console using SSH.
The response time was from sending a character input to receiving its echo-back from the user VM.
The left figure shows the result.
The response time in VSBypass was 1.3 ms longer than that in the traditional system.
This is because it took a longer time to process input data in the user VM due to the overhead of nested virtualization.
Next, we measured the throughput of a virtual serial console.
We outputted a text file in the user VM and measured the completion time.
The right figure shows the result.
Surprisingly, the throughput in VSBypass was almost the same as that in the traditional system.
In this experiment, the shadow serial device outside the cloud VM dominated output processing and it wasn't affected by nested virtualization.

20. Performance of GUI Remote Access
The response time between a key press and a screen update in a VNC client
VSBypass was 13 ms longer due to the screen update
The update time of user VM's screen in a VNC client
VSBypass was 23 ms longer
We measured the response time of a keyboard input in GUI remote access using VNC.
The response time was from pressing a key to receiving screen data updated in the user VM.
The left figure shows the result.
The response time in VSBypass was 13 ms longer than the traditional system.
This is because the screen update took a longer time due to nested virtualization.
Next, we measured only the update time of the screen data.
The update time was from sending a request for screen update to receiving updated screen data.
In this experiment, we modified a larger area of the screen in the user VM.
The right figure shows the result.
The update time in VSBypass was 23 ms longer, but we couldn't notice that degradation in remote management.

21. Related Work Device passthrough CloudVisor [Zhang+ SOSP'11]
Enable direct access to physical devices from VMs
Transparent passthrough in VSBypass is applied to virtual devices
CloudVisor [Zhang+ SOSP'11]
Encrypt the memory/disks of user VMs in the cloud hypervisor
Protect no virtual devices for out-of-band remote management
V-Met [Miyama+ ESORICS'17]
Offload IDSes outside the virtualized system with nested virtualization
Provide deep VM introspection to obtain the internal state of user VMs
Device passthrough enables direct access to physical devices from VMs.
This mechanism is used to improve I/O performance of VMs.
Transparent passthrough in VSBypass is applied to virtual devices to enhance security.
CloudVisor encrypts the memory and disks of user VMs in the thin cloud hypervisor.
However, it doesn't protect virtual devices used for out-of-band remote management.
VSBypass can be integrated with it.
V-Met offloads intrusion detection systems outside the virtualized system using nested virtualization.
It provides deep VM introspection to obtain the internal state of user VMs.
VSBypass uses several techniques proposed for V-Met.

22. Conclusion
We proposed secure out-of-band remote management outside the virtualized system
Run the virtualized system in a VM using nested virtualization
Intercept I/O requests outside it using transparent passthrough
Process them in shadow devices
Future work
Create/destroy proxy VMs with the boot/shutdown of user VMs
Support the migration of user VMs with the state of shadow devices
In conclusion, we proposed VSBypass for enabling secure out-of-band remote management outside the virtualized system.
VSBypass runs the virtualized system in a VM using nested virtualization.
Then, it intercepts I/O requests outside the virtualized system by transparent passthrough and processes them in shadow devices.
Thus, untrusted cloud operators in the virtualized system cannot eavesdrop on or tamper with I/O data of out-of-band remote management.
One of our future work is to dynamically create and destroy a proxy VM when a user VM is booted and shut down, respectively.
Another direction is to support the migration of user VMs.
Since the state of shadow devices isn't transferred to the destination, out-of-band remote management isn't continued correctly.
We are currently developing a new mechanism for restoring the state of shadow devices.