Secure Out-of-band Remote Management Using Encrypted Virtual Serial Consoles in IaaS Clouds Kenichi Kourai Tatsuya Kajiwara Kyushu Institute of Technology

IaaS Clouds  Provide users with virtual machines (VMs)  Users manage their systems through SSH  In-band remote management is usually used  A user connects an SSH client to an SSH server running in his VM  Not usable when a user failed network configurations SSH server SSH client user VM cloud firewall

Virtual Serial Console (VSC)  IaaS provides out-of-band remote management  An SSH server runs in the management VM  Access a user VM via a VSC  A VSC consists of a virtual serial device and a serial driver  A user can access his VM without relying on VM's network virtual serial device serial driver SSH server SSH client management VMuser VM VSC firewall

Untrusted Admins in IaaS  The management VM is not always trustworthy  Honest-but-curious admins take information that they can easily look at  Malicious admins act as inside attackers  Skill-less admins cause a vulnerable management VM to be penetrated by outside attackers management VMuser VM untrusted admins outside attackers

Attacks against Remote Management  Attackers can eavesdrop on inputs and outputs of out-of-band remote management  By modifying an SSH server in the management VM  Attack examples  Extract passwords from inputs to login prompts  Steal displayed secrets virtual serial device serial driver SSH server SSH client management VMuser VM

SCCrypt  Provide an encrypted VSC to prevent information leakage  Receive encrypted inputs from the management VM  Decrypt and send them to a user VM  Receive unencrypted outputs from a user VM  Encrypt and send them to the management VM virtual serial device serial driver SSH server SSH client management VMuser VM encrypted VSC decrypt encrypt encrypted inputs/outputs

Where to Encrypt/Decrypt?  Virtual serial device in the management VM?  The entire management VM is untrusted  Attackers can tamper with the virtual device  Serial driver in a user VM?  Modifying the existing device driver is not desirable  Users cannot use favorite OSes virtual serial device serial driver management VM user VM en-/de- crypt?

Leveraging the Trusted VMM  SCCrypt encrypts/decrypts data in the virtual machine monitor (VMM)  The integrity of the VMM can be guaranteed  At boot time: remote attestation with TPM  At runtime: HyperGuard [Rutkowska+ '08], HyperCheck [Wang+ '10], and HyperSentry [Azab+ '10] with SMM virtual serial device serial driver management VM user VM VMM encrypted VSC SCCrypt trusted authority

How to Identify Inputs/Outputs?  The traditional VMM does not recognize a virtual serial device or a serial driver  Cannot rely on information from the untrusted device in the management VM  Cannot obtain information from the unmodified driver in a user VM untrusted virtual serial device unmodified serial driver management VM user VM VMM SCCrypt ?

Tracking Device States  The VMM identifies inputs/outputs without the cooperation of the device or the driver  Track the state of a virtual serial device  From the interactions between the device and the driver  Extract only inputs/outputs  Based on the knowledge of the standard of serial devices virtual serial device serial driver management VM user VM VMM SCCrypt

Implementation  We have implemented SCCrypt in Xen and the OpenSSH 6.0p1 client  Support HVM guests  Emulate hardware as is  Support PV guests (see the paper)  Define interfaces suited for virtualization

Output Delivery  A serial driver writes data using OUT instruction  The VMM traps this instruction using Intel VT-x  It encrypts the written data using RC4  Only if the specified I/O port address is 3F8  A session key is exchanged for each connection virtual serial device serial driver management VM user VM VMM encrypt trap OUT SSH server SSH client

State Tracking  The VMM encrypts the written data if  The FIFO buffers are enabled  Neither in the divisor latch access nor loopback mode  Devisor latch access mode: set a baud rate  Loopback mode: test a device  It watches all the writes to recognize the context FIFO disabled divisor latch access mode looback mode FIFO enabled

Input Delivery  A serial driver reads data using IN instruction  The VMM traps this instruction  Receive input data from a virtual serial device  It decrypts input data using RC4  If the same conditions as in output delivery are satisfied virtual serial device serial driver management VM user VM VMM decrypt IN SSH server SSH client trap

Pending Outputs at Reconnection  An SSH client cannot decrypt pending outputs correctly at reconnection  Console outputs are stored in a virtual serial device  While an SSH client is not connected  Pending outputs are sent to the client at reconnection  Encrypted with an old session key virtual serial device management VM SSH server SSH client pending data encrypted with an old session key new session key ?

Re-encryption of Outputs  The VMM re-encrypts pending outputs  Restore unencrypted data with an old session key  Cannot 'decrypt' it  Reverse the encryption process in RC4  Encrypt the data with a new session key virtual serial device management VM VMM re-encrypt SSH server SSH client pending data old/new session keys !#$ A#$ AB$AB$ ABC !#$ !#C !BC!BC ABC decryptreverse

Experiments  We confirmed the effectiveness of SCCrypt  Response time in an SSH client  Throughput of console outputs  CPU utilization  We compared results between SCCrypt and vanilla Xen CPU: Intel Xeon E5630 Mem: 6 GB Net: Gigabit Ethernet OS: Linux SSH: OpenSSH 6.0p1 CPU: Intel Core i7 870 Mem: 4 GB Net: Gigabit Ethernet VMM: Xen client server vCPU: 8 Mem: 1 GB OS: Linux user VM vCPU: 8 Mem: 3 GB OS: Linux SSH: OpenSSH 5.9p1 management VM

Response Time (Input)  We measured the response time of inputs  An SSH client sent an input to an SSH server  It received an output caused by its remote echo  Result  2.9% shorter in SCCrypt  The reason was unclear

Throughput (Output)  We measured the throughput of outputs  We wrote large text to a VSC in a user VM  Result  5.6% higher in SCCrypt  The reason was unclear

CPU Utilization (Input)  We measured the CPU utilization under the keyboard auto-repeat at a client  Management VM: 0.8% point lower in SCCrypt  User VM: almost the same management VM user VM

CPU Utilization (Output)  We measured the CPU utilization under periodic write to a VSC from a user VM  Management VM: 1.3% points difference at most  User VM: almost the same management VM user VM

Related Work  FBCrypt [Egawa et al.'12]  Encrypt keyboard/mouse inputs and video outputs in out-of-band remote management using VNC  Strongly depend on VNC  Xoar [Colp et al.'11]  Run a virtual serial device in trusted Console VM  Console VM can be compromised via an SSH server  VMware vSphere  Run virtual devices and a VNC server in the VMM  The VMM can be compromised via the VNC server

Conclusion  SCCrypt for enabling secure out-of-band remote management in untrusted IaaS clouds  Provide encrypted VSCs  Inputs/outputs are securely decrypted/encrypted in the trusted VMM  Future work  Apply SCCrypt to other remote management systems using VSCs  E.g., web-based Ajaxterm