Heterogeneous Live Migration of Virtual Machines Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang Parallel Processing Institute, Fudan University

The market of Virtualization is booming Virtual data-center, grid computing, and cloud computing “By 2010, at least 30%of non-desktop IT infrastructure will be virtualized (up from less than 5 percent in 2007).” – What can virtualization do? – Sever consolidation – Mobility Virtual machine live migration –…–…

VM Live Migration Benefits – Load balancing – Online maintenance – Fault tolerance PPI, Fudan University3 Virtual Machine Virtual Machine VMM Hardware VMM

War between different VMM venders The battle in virtualization area is and will be continuing – Vmware, Microsoft, kvm, … – They will likely co-exisit in a long time Linux kernel embeds different vmm support – Xen, lguest, kvm Cloud Computing More resources brings inevitably heterogeneity

Virtual Machine Virtual Machine Motivation Heterogeneity of different VMMs – Break uniformity and mobility – Downgrade the applicability of live migration PPI, Fudan University5 VMM 2 VMM 1 Hardware

Contribution: Vagrant A preliminary study aim at – Bridging the heterogeneity of different VMMs and support live migration Goal – Provide common migration protocol – Use common format of VM abstraction – Provide a pool of memory migration algorithms Preliminary results – Support VM migration across Xen and KVM – Accept able performance

Outline Heterogeneity among VMMs Vagrant Design and Implementation Evaluation Conclusion PPI, Fudan University7

Heterogeneity in Abstracting Resources CPU – Internal virtual CPU format Memory – Memory management information I/O Devices – Virtual I/O Device format PPI, Fudan University8

Heterogeneity in Migration Algorithms Stop-and-copy Pre-copy Push and pull PPI, Fudan University9

Heterogeneity in Migration Protocols Different set of control commands Example – Xen VMM initiates migration by sending a string ”receive” to the target – KVM sends a special integer to start migration PPI, Fudan University10

Outline Heterogeneity among VMMs Vagrant Design and Implementation Evaluation Conclusion PPI, Fudan University11

Architecture of Vagrant Vagrant acts as a middle man between different VMMs PPI, Fudan University12 VMM A VM Management VM VMM B Vagrant Agency State Transformer Algorithm pool Management VM Vagrant Agency State Transformer Algorithm pool

Bridging the Heterogeneity CPU State – Intercept the state from local virtual CPU module – Essential CPU state is transformed into Vagrant format – Requirements : VMMs can expose same CPU features PPI, Fudan University13

Bridging the Heterogeneity (2/2) I/O Devices – Intercept the state from local device module H/W based virtualization relies on device emulation – Eg. QEMU used by Xen, KVM and VirtualBox – Requirements: VMMs can support all I/O devices existing in the migrated VM PPI, Fudan University14

Migration Protocol Initiate and Control the migration process – Common migration protocol and commands – Negotiation before the migration Ensure the same capability of pre- and post- migration environment – CPU features, supported I/O devices PPI, Fudan University15 VMM AVMM B Hey,I want a RTL8169 NIC, Can you support?

Migration Algorithms Each VMM has its own migration algorithm PPI, Fudan University16

Stop and Copy PPI, Fudan University17 Virtual Machine Virtual Machine VMM Hardware VMM Simply stop the VM and send VM state May cause very long downtime, but total time is short

Pre-copy Based Algorithm Sending memory pages while the VM is still running PPI, Fudan University18 Virtual Machine Virtual Machine VMM Hardware VMM Sending memory pages VM is still running Stop the VM Send whole VM state to destination Pre-copy Reduces downtime

Algorithm Pool Algorithm pool – At least one common algorithm is needed – Choose the algorithm by negotiation Ideally, support a set of different algorithms – Choose appropriate algorithm in different scenarios – eg, the user wants to finish the migration as soon as possible, stop and copy is better PPI, Fudan University19

Implementation A prototype based on – X86 with VT-x extensions – Xen and KVM-48 – CPU Both expose most CPU features – I/O Device Both rely on QEMU VMM support for some devices, such as PIC/APIC/IOAPIC PPI, Fudan University20

Outline Heterogeneity among VMMs Vagrant Design and Implementation Evaluation – Migration time – Application performance Conclusion PPI, Fudan University21

Experimental Setup(1/2) Machine setup: – Migration between pair of PCs with 2.33 GHz Intel Core Duo CPU, 2GB RAM RTL8169 NIC 250 GB 7200 RPM SATA disk – NFS server 3.0 GHz Pentium IV with 1GB RAM Intel Pro 100/1000 Ethernet NIC 250 GB 7200 RPM SATA disk – Machines were connected via switched Gigabit Ethernet PPI, Fudan University22

Experimental Setup (2/2) VM configuration – 128 MB of RAM – 5GB disk image with installation of Fedora Core 6 Migration time under different workload – Idle, kernel-compile, memtest86, apache benchmark (ab) Application performance – Migrating a running Web Server Continuously serves a 256 KB file to a remote client PPI, Fudan University23

Questions for Evaluation Whether Vagrant has an acceptable total migration and minimal downtime? Whether applications during the migration can still provide acceptable performance? Is Vagrant’s performance comparable to other migration framework, eg. Xen and KVM? PPI, Fudan University24

Total Migration Time PPI, Fudan University25 Vagrant adopts batched memory transfer while KVM uses one-by-one manner

Migration Downtime PPI, Fudan University26 Downtime in Vagrant in comparable to homogeneous live migration

Migrating a Running Web Server PPI, Fudan University27 Memory Pre-copy, about 10% downgrade Less than 1 second

Related Work Homogenous live migration over LAN – Vmotion, Xen (Clark et al. ’05) and KVM Live wide-area migration – Bradford et al. (vee’07) – “VM Turntable” (FGCS’06) Tons of virtualization related work PPI, Fudan University28

Conclusion Heterogeneous live VM migration framework – General abstraction of VMs and migration protocol – No changes to hosted OS and application – Minor changes to VMM abstraction – Comparable downtime and end-to-end time PPI, Fudan University29

Questions? Thank you! PPI, Fudan University

Heterogeneous Live Migration of Virtual Machines Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang Parallel Processing Institute, Fudan University

Backup Slides Control commands in Vagrant CPU state in Vagrant I/O devices configured for the VM Memory migrating algorithm PPI, Fudan University32

Control Commands in Vagrant Common set of control commands – Each has a 3 seconds timeout PPI, Fudan University33

CPU State in Vagrant The necessary information to be transferred includes – GPR, control regs, segment regs, debug regs, msr, fpu regs, sysenter registers – Time stamp counter – Pending events PPI, Fudan University34

I/O devices configured for the VM The migrated VM is configured with – Keyboard and mouse – Rtc – PIC/APIC/IOAPIC – PIT – IDE disk – NIC – Serial port – VGA PPI, Fudan University35

Memory migrating algorithm Algorithm implemented by Vagrant – Pre-copy in batched manner Batch size is PPI, Fudan University36