CertiKOS Implementation Progress Liang Gu Yale University

Content Overview Virtual Disk Port to Intel platform Reduce virtual device code Virtual device at user mode Pass through device with IOMMU Demo

Progress after Boston PI meeting Virtualization Virtual devices: PCI, Virtual Disk ( virtio ,virtio block) Support Multiple VM guests Move virtual devices to execute in user mode IOMMU based device pass-through to guest OS Port CertiKOS to work on bare metal PCI and AHCI Port to Intel platform with VT-x Green- done Blue- almost done Red- ongoing

CertiKOS Architecture APP (uncertified) APP (certified) Mgmt Shell Mgmt OS (Linux) Commodity OS Application Master Master Syscall Slave Slave Syscall Process Management Virtual Machine Management Memory V-Interrupt V-Devices Hypercall Context IPC CertiKOS Bottom is the commodity hardware. Maybe we need to mention CertiKOS requires CPU provides hardware virtualization support, i.e. AMD SVM in latest CertiKOS. The medium layer is CertiKOS kernel. Hardware abstraction layer (HAL) provides some device drivers, e.g. PCI adaptor, hard disk drive, ethernet adaptor, etc. It’s almost empty in latest CertiKOS. The layer upon HAL provides basic kernel primitives. SMP management, memory management, virtual memory management, interrupt handling and spinlock are similar to those in other operating systems. The extension is the virtualization part, which is composed of SVM primitives, virtual devices and virtualization abstraction. SVM primitives provide functions to control the underlying AMD SVM CPU. Virtual Devices provide necessary device emulation and passthrough, e.g. CertiKOS emulates interrupt controller, programmable timer, keyboard controller, etc, blocks PCI adaptor and devices, and passes through other devices. Virtualization abstraction provides an abstract interface to other parts of CertiKOS kernel, so that they don’t need to take care which the underlying hardware virtualization is and if the access to devices from the guest should be emulated or just passed through. CertiKOS provides two kinds of runnable objects: process and virtual machine. Process is similar to those in other operating systems except that CertiKOS uses a much simple process model: each physical process core runs at most one process. Context contains the information used in process schedule, trap handling, etc. IPC provides message-passing style communication mechanism between processes. Virtual machine management is to Create a virtualization environment for the guest, which contains virtualizing CPU, memory, interrupt mechanism and a minimal set of devices. Control the execution of the virtual machine, e.g. start, stop, resume, shutdown. (functions in bold have not been done yet) Handle the hypercals. The details will be shown in the following slides. CertiKOS provides two kinds of system calls: Master system calls are for the management shell. Slave system calls are for the certikos-based applications. We can run four kinds of applications upon CertiKOS. Management shell, which launches and controls other applications. We plan to move it to the guest and implement it as a userspace application of the guest OS. In this way the management shell can leverage the libraries in the guest OS to provide a better user interface, and use hypercalls provided by CertiKOS to complete the management work. Virtual machine, or guest, in which runs a commodity operating system and its applications. No modifications are required for both the commodity operating system and the applications, except when they want to use hypercalls provided by CertiKOS. Except the hypercall, CertiKOS is transparent for them. Uncertified CertiKOS-based applications. Certified CertiKOS-based applications, which are formally certified and doing security-concerned jobs. Only objects below the red line are in the trusted computing base. SMP Management Virtual Memory Interrupt Handling Vconsole Virtualization Abstraction SVM Primitives Virtual Devices Memory Management Hardware Abstraction Layer Hardware CPU Cores Memory Hard Disk Network . . .

CertiKOS Architecture APP (uncertified) APP (certified) Virtual Devices Mgmt Shell Mgmt OS (Linux) Commodity OS Application Master Master Syscall Slave Slave Syscall Process Management Virtual Machine Management Memory V-Interrupt V-Devices Hypercall Context IPC CertiKOS Bottom is the commodity hardware. Maybe we need to mention CertiKOS requires CPU provides hardware virtualization support, i.e. AMD SVM in latest CertiKOS. The medium layer is CertiKOS kernel. Hardware abstraction layer (HAL) provides some device drivers, e.g. PCI adaptor, hard disk drive, ethernet adaptor, etc. It’s almost empty in latest CertiKOS. The layer upon HAL provides basic kernel primitives. SMP management, memory management, virtual memory management, interrupt handling and spinlock are similar to those in other operating systems. The extension is the virtualization part, which is composed of SVM primitives, virtual devices and virtualization abstraction. SVM primitives provide functions to control the underlying AMD SVM CPU. Virtual Devices provide necessary device emulation and passthrough, e.g. CertiKOS emulates interrupt controller, programmable timer, keyboard controller, etc, blocks PCI adaptor and devices, and passes through other devices. Virtualization abstraction provides an abstract interface to other parts of CertiKOS kernel, so that they don’t need to take care which the underlying hardware virtualization is and if the access to devices from the guest should be emulated or just passed through. CertiKOS provides two kinds of runnable objects: process and virtual machine. Process is similar to those in other operating systems except that CertiKOS uses a much simple process model: each physical process core runs at most one process. Context contains the information used in process schedule, trap handling, etc. IPC provides message-passing style communication mechanism between processes. Virtual machine management is to Create a virtualization environment for the guest, which contains virtualizing CPU, memory, interrupt mechanism and a minimal set of devices. Control the execution of the virtual machine, e.g. start, stop, resume, shutdown. (functions in bold have not been done yet) Handle the hypercals. The details will be shown in the following slides. CertiKOS provides two kinds of system calls: Master system calls are for the management shell. Slave system calls are for the certikos-based applications. We can run four kinds of applications upon CertiKOS. Management shell, which launches and controls other applications. We plan to move it to the guest and implement it as a userspace application of the guest OS. In this way the management shell can leverage the libraries in the guest OS to provide a better user interface, and use hypercalls provided by CertiKOS to complete the management work. Virtual machine, or guest, in which runs a commodity operating system and its applications. No modifications are required for both the commodity operating system and the applications, except when they want to use hypercalls provided by CertiKOS. Except the hypercall, CertiKOS is transparent for them. Uncertified CertiKOS-based applications. Certified CertiKOS-based applications, which are formally certified and doing security-concerned jobs. Only objects below the red line are in the trusted computing base. SMP Management Virtual Memory Interrupt Handling Vconsole Virtualization Abstraction SVM /VMX Primitives SVM Primitives Virtual Devices Memory Management Hardware Abstraction Layer Hardware CPU Cores Memory Hard Disk Network . . .

CertiKOS Architecture APP (uncertified) APP (certified) Virtual Devices Mgmt Shell Mgmt OS (Linux) Commodity OS Application Master Master Syscall Slave Slave Syscall Process Management Virtual Machine Management Memory V-Interrupt V-Devices Hypercall Context IPC CertiKOS Bottom is the commodity hardware. Maybe we need to mention CertiKOS requires CPU provides hardware virtualization support, i.e. AMD SVM in latest CertiKOS. The medium layer is CertiKOS kernel. Hardware abstraction layer (HAL) provides some device drivers, e.g. PCI adaptor, hard disk drive, ethernet adaptor, etc. It’s almost empty in latest CertiKOS. The layer upon HAL provides basic kernel primitives. SMP management, memory management, virtual memory management, interrupt handling and spinlock are similar to those in other operating systems. The extension is the virtualization part, which is composed of SVM primitives, virtual devices and virtualization abstraction. SVM primitives provide functions to control the underlying AMD SVM CPU. Virtual Devices provide necessary device emulation and passthrough, e.g. CertiKOS emulates interrupt controller, programmable timer, keyboard controller, etc, blocks PCI adaptor and devices, and passes through other devices. Virtualization abstraction provides an abstract interface to other parts of CertiKOS kernel, so that they don’t need to take care which the underlying hardware virtualization is and if the access to devices from the guest should be emulated or just passed through. CertiKOS provides two kinds of runnable objects: process and virtual machine. Process is similar to those in other operating systems except that CertiKOS uses a much simple process model: each physical process core runs at most one process. Context contains the information used in process schedule, trap handling, etc. IPC provides message-passing style communication mechanism between processes. Virtual machine management is to Create a virtualization environment for the guest, which contains virtualizing CPU, memory, interrupt mechanism and a minimal set of devices. Control the execution of the virtual machine, e.g. start, stop, resume, shutdown. (functions in bold have not been done yet) Handle the hypercals. The details will be shown in the following slides. CertiKOS provides two kinds of system calls: Master system calls are for the management shell. Slave system calls are for the certikos-based applications. We can run four kinds of applications upon CertiKOS. Management shell, which launches and controls other applications. We plan to move it to the guest and implement it as a userspace application of the guest OS. In this way the management shell can leverage the libraries in the guest OS to provide a better user interface, and use hypercalls provided by CertiKOS to complete the management work. Virtual machine, or guest, in which runs a commodity operating system and its applications. No modifications are required for both the commodity operating system and the applications, except when they want to use hypercalls provided by CertiKOS. Except the hypercall, CertiKOS is transparent for them. Uncertified CertiKOS-based applications. Certified CertiKOS-based applications, which are formally certified and doing security-concerned jobs. Only objects below the red line are in the trusted computing base. SMP Management Virtual Memory Interrupt Handling Vconsole Virtualization Abstraction SVM Primitives SVM /VMX Primitives IOMMU Memory Management Hardware Abstraction Layer Hardware CPU Cores Memory Hard Disk Network . . .

Content Overview Virtual Disk Port to Intel platform Reduce virtual device code Virtual device at user mode Pass through device with IOMMU Demo

Virtual Disk … Motivation Virtual PCI Virtual disk based on virtio Enable CertiKOS to boot guest OS on bare metal Separate the storage of guest OS from CertiKOS physically Virtual PCI Virtual disk based on virtio a. Mgmt shell Linux Legacy OS, e.g., Linux c. CertiKOS-based APP … d.Mgmt tool in Linux CertiKOS Virtual Disk disk0 disk1

Virtio Virtio Rusty Russell, “virtio: Towards a De-Facto Standard For Virtual I/O Devices” Available in both Linux and Windows A simple and efficient framework to provide virtual devices to guest OS Virtio is an abstraction for a set of common virtual devices Legacy OS, e.g., Linux Add an example to show the virtqueue and virtring Virtqueue Front-end driver Back-end driver CertiKOS … Disk driver disk1

Boot CertiKOS on Bare Metal Multiple settings for booting CertiKOS on bare metal Boot CertiKOS and Guest on the same disk Boot CertiKOS and Guest on different disks Boot CertiKOS on USB and boot the guest on disk

Content Overview Virtual Disk Port to Intel platform Reduce virtual device code Virtual device at user mode Pass through device with IOMMU Demo

Port to Intel platform Motivation Modularized implementation LOCs Another widely supported Hardware-based Virtualization solution Widely available VT-d support Modularized implementation Separate architecture dependent modules Integrated by interfaces in the abstraction layer LOCs Sys/virt/svm 1775 Sys/virt/vmx 2344 VMX uses more sophisticated methods to control the virtualization Access memory region for control data structures by special instructions, instead of direct memory read and write More sophisticated setup Virtual Machine Management Virtualization Abstraction SVM Primitive VMX Primitive Virtual Devices

Content Overview Virtual Disk Port to Intel platform Reduce virtual device code Virtual device at user mode Pass through device with IOMMU Demo

Virtual Device LOCs in previous version at Boston PI meeting Sys/virt/ 4441* Sys/virt/dev/ 2384* With Virtual PIC, KBD, PIT, text mode VGA LOCs in current clean_code branch Sys/virt/ 8237*# Sys/virt/dev/ 3643* Added virtual PCI, Virtio, Virtio-blk Considering more devices, such as USB, Network, … Moving virtual device to execute in user mode Securely pass through device with IOMMU ( * counted by cloc 1.56) ( # with Intel vt-x )

Virtual Device at User Mode For untrusted guest domains, their virtual devices don’t have to be trusted Process model extension Multiple processes on a single core based on round-robin scheduling Message passing via channels among processes a. Idle … d. Legacy Linux V-KBD V-PIC V-PIT … CertiKOS CPU0 CPU1

Virtual Device at User Mode Support multiple VM guests with VM session extension VM Session 1 VM Session 2 a. Idle … … d. Legacy Linux Guest Linux 2 … … … CertiKOS … CPU0 CPU1 CPU2

Pass Through Device Exclusively used devices can be directly exposed to guest VM, without introducing device virtualization code However, malicious DMA operations are capable of attacking memory spaces IOMMU / VT-d allow a guest OS running under a VMM to have direct control of a device Provide fine-grain control of device access to system memory

IOMMU from AMD IOMMU specification Revision 2

IOMMU Main memory IOMMU MMU Device CPU Physical Address IOMMU MMU Device Address Virtual Address Device CPU from http://en.wikipedia.org/wiki/IOMMU

IOMMU IOMMU MMU Device CPU Page Table 1 Page Table 2 IOMMU MMU Device Table … Device Address Virtual Address … … … Device CPU … Interrupt Remapping Table Based on image from http://en.wikipedia.org/wiki/IOMMU

Pass through device with IOMMU Legacy OS, e.g., Linux NPT Device Table CertiKOS … Interrupt Remapping Table IOMMU device

Content Overview Virtual Disk Port to Intel platform Reduce virtual device code Virtual device at user mode Pass through device with IOMMU Demo

CertiKOS Demo Setting For Previous Version … … BSP AP AP KVM a. Mgmt shell Legacy OS, e.g., Linux c. CertiKOS-based APP … master slave slave CertiKOS Dotted line of mgmt Linux means it is under developing. … Qemu BSP AP AP Linux KVM AMD processor with SVM BSP- Boot Strap Processor AP-Application Processor

AMD processor with SVM/ Intel with VT-x CertiKOS Demo Setting a. Mgmt shell Legacy OS, e.g., Linux c. CertiKOS-based APP … master slave slave CertiKOS Dotted line of mgmt Linux means it is under developing. AMD processor with SVM/ Intel with VT-x BSP- Boot Strap Processor AP-Application Processor

Thank you!