THAWAN KOOBURAT MICHAEL SWIFT UNIVERSITY OF WISCONSIN - MADISON 1 The Best of Both Worlds with On-Demand Virtualization

Motivation 2 Virtualization offers many benefits Why virtualization has not yet become ubiquitous?

Concerns with Virtualization 3 Virtualization adds an extra layer that introduces overhead all the time Lack of functionality  GPUs, WLAN and etc. Performance overhead  I/O Devices - Trap and emulate  Memory Virtualization – Extra level of page tables

On-Demand Virtualization 4 Mechanism  Normally execute natively  Virtualize on-demand  Minimal service disruption Benefits  Native performance and full functionality  Pay for virtualization overhead only when used

Uses of On-Demand Virtualization 5 Datacenter  Resource Consolidation  Debugging Desktop Environment  Migration  Checkpointing  Introspection

Main Ideas 6 High-level Design  Pause OS operation and resume inside VM  Set aside VMM partition on disk  Virtualize everything Physical VMM VM Native OS Guest OS

Problem:  Need to transfer OS/Process state from native to virtual execution Solution:  Hibernate/Resume can capture both OS and process state  Snapshot system memory and write image to disk \\ Capturing OS and Process State 7 Native Guest VM Physical Virtualize VMM

Capturing OS and Process States 8 Physical VMM Partition VMM Partition Guest Partition Guest Partition Native Kernel Hibernate Boot Guest Kernel Resume VMM Kernel VMM Kernel Start VM

Discovering Devices inside the VM 9 Problem:  Need to switch from physical to virtual devices Solution:  Use Hotplug support in device  Detach/disable original physical devices before hibernate  Scan for virtual devices after resume \\ 9 VM Physical Virtualize Disk Native NIC Disk Guest NIC

Problem:  Need to preserve OS/Application state which ties to devices  Network connections, File systems  Need to transfer device bindings across different models/types of devices Preserving Device Bindings 10 VM Physical Virtualize Disk Kernel NIC Disk Kernel NIC Apps

Preserving Device Bindings 11 Solution:  Use logical device to retain states and hide changes  Networks: Bonding Driver Detach physical NIC and attach virtual NIC after resume  Block Devices:Device Mapper Update DM’s table to point to a virtual block device Applications Phy VM e1000e e1000 bond0 Kernel Applications Phy VM SATA IDE /dev/mapper/vdisk Kernel Network Disk

The Cost of Preparation 12 Set aside VMM partition on disk  No need to pre-allocate CPU or memory Access hardware through logical devices  Only for devices whose state must be preserved Disable hardware that is not emulated by VMM  Eg. IOMMU

Status 13 Status  Prototype  Linux (~1200 lines of modification)  TuxOnIce (Hibernation patch)  KVM  Support one-way conversion from physical to virtual machine Experience  SCP/SSH remains open  Take about 90 seconds to convert  Hibernating and rebooting take most of the time

Future Work 14 Hibernate to RAM Use kexec for machine reboot Support devirtualize  Transfer VM back to its original machine

Conclusion 15 Benefits  Add another method of using virtualization  Allow more environments to benefit from virtualization without its overhead all the time Contributions  On-demand virtualization is possible  Existing mechanisms in OS can be repurposed

Questions 16

Other Devices 17 CPU  Current:Offline other CPUs, only one core is left running  Future:CPU hotplug Memory  Current:VM must have equal or more RAM than physical machine  Future:Memory hotplug Display  VGA is supported by all graphic cards  Revert to legacy VGA mode before hibernate

Dealing with Platform Devices 18 Platform devices  Device on main board – i.e. timers, interrupt routers  No hotplug support  Kernel only configure them at boot time via ACPI Solution  Boot kernel pass configuration information to resume kernel so that it can reconfigure its devices \\ VM Boot Kernel Resume config \\ VM Guest Kernel config Hibernate Image

Conversion Time 19 Hibernate 10 s Reboot machine 50 s Boot VM 9 s Resume 21 s

Overall Process 20 Hibernate and resume the kernel inside a virtual machine Use hotplugging to detach and scan for new hardware Reconfigure platform devices Update logical device mappings Resume OS subsystem/Applications