1. Heterogeneous Live Migration of Virtual Machines Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang Parallel Processing Institute, Fudan University http://ppi.fudan.edu.cn

2. The market of Virtualization is booming Virtual data-center, grid computing, and cloud computing “By 2010, at least 30%of non-desktop IT infrastructure will be virtualized (up from less than 5 percent in 2007).” – www.cio.com/article/170653 What can virtualization do? – Sever consolidation – Mobility Virtual machine live migration –…–…

3. VM Live Migration Benefits – Load balancing – Online maintenance – Fault tolerance 2015-5-15PPI, Fudan University3 Virtual Machine Virtual Machine VMM Hardware VMM

4. War between different VMM venders The battle in virtualization area is and will be continuing – Vmware, Microsoft, kvm, … – They will likely co-exisit in a long time Linux kernel 2.6.23 embeds different vmm support – Xen, lguest, kvm Cloud Computing More resources brings inevitably heterogeneity

5. Virtual Machine Virtual Machine Motivation Heterogeneity of different VMMs – Break uniformity and mobility – Downgrade the applicability of live migration 2015-5-15PPI, Fudan University5 VMM 2 VMM 1 Hardware

6. Contribution: Vagrant A preliminary study aim at – Bridging the heterogeneity of different VMMs and support live migration Goal – Provide common migration protocol – Use common format of VM abstraction – Provide a pool of memory migration algorithms Preliminary results – Support VM migration across Xen and KVM – Accept able performance

7. Outline Heterogeneity among VMMs Vagrant Design and Implementation Evaluation Conclusion 2015-5-15PPI, Fudan University7

8. Heterogeneity in Abstracting Resources CPU – Internal virtual CPU format Memory – Memory management information I/O Devices – Virtual I/O Device format 2015-5-15PPI, Fudan University8

9. Heterogeneity in Migration Algorithms Stop-and-copy Pre-copy Push and pull 2015-5-15PPI, Fudan University9

10. Heterogeneity in Migration Protocols Different set of control commands Example – Xen VMM initiates migration by sending a string ”receive” to the target – KVM sends a special integer to start migration 2015-5-15PPI, Fudan University10

11. Outline Heterogeneity among VMMs Vagrant Design and Implementation Evaluation Conclusion 2015-5-15PPI, Fudan University11

12. Architecture of Vagrant Vagrant acts as a middle man between different VMMs 2015-5-15PPI, Fudan University12 VMM A VM Management VM VMM B Vagrant Agency State Transformer Algorithm pool Management VM Vagrant Agency State Transformer Algorithm pool

13. Bridging the Heterogeneity CPU State – Intercept the state from local virtual CPU module – Essential CPU state is transformed into Vagrant format – Requirements : VMMs can expose same CPU features 2015-5-15PPI, Fudan University13

14. Bridging the Heterogeneity (2/2) I/O Devices – Intercept the state from local device module H/W based virtualization relies on device emulation – Eg. QEMU used by Xen, KVM and VirtualBox – Requirements: VMMs can support all I/O devices existing in the migrated VM 2015-5-15PPI, Fudan University14

15. Migration Protocol Initiate and Control the migration process – Common migration protocol and commands – Negotiation before the migration Ensure the same capability of pre- and post- migration environment – CPU features, supported I/O devices 2015-5-15PPI, Fudan University15 VMM AVMM B Hey,I want a RTL8169 NIC, Can you support?

16. Migration Algorithms Each VMM has its own migration algorithm 2015-5-15PPI, Fudan University16

17. Stop and Copy 2015-5-15PPI, Fudan University17 Virtual Machine Virtual Machine VMM Hardware VMM Simply stop the VM and send VM state May cause very long downtime, but total time is short

18. Pre-copy Based Algorithm Sending memory pages while the VM is still running 2015-5-15PPI, Fudan University18 Virtual Machine Virtual Machine VMM Hardware VMM Sending memory pages VM is still running Stop the VM Send whole VM state to destination Pre-copy Reduces downtime

19. Algorithm Pool Algorithm pool – At least one common algorithm is needed – Choose the algorithm by negotiation Ideally, support a set of different algorithms – Choose appropriate algorithm in different scenarios – eg, the user wants to finish the migration as soon as possible, stop and copy is better 2015-5-15PPI, Fudan University19

20. Implementation A prototype based on – X86 with VT-x extensions – Xen 3.1.0 and KVM-48 – CPU Both expose most CPU features – I/O Device Both rely on QEMU VMM support for some devices, such as PIC/APIC/IOAPIC 2015-5-15PPI, Fudan University20

21. Outline Heterogeneity among VMMs Vagrant Design and Implementation Evaluation – Migration time – Application performance Conclusion 2015-5-15PPI, Fudan University21

22. Experimental Setup(1/2) Machine setup: – Migration between pair of PCs with 2.33 GHz Intel Core Duo CPU, 2GB RAM RTL8169 NIC 250 GB 7200 RPM SATA disk – NFS server 3.0 GHz Pentium IV with 1GB RAM Intel Pro 100/1000 Ethernet NIC 250 GB 7200 RPM SATA disk – Machines were connected via switched Gigabit Ethernet 2015-5-15PPI, Fudan University22

23. Experimental Setup (2/2) VM configuration – 128 MB of RAM – 5GB disk image with installation of Fedora Core 6 Migration time under different workload – Idle, kernel-compile, memtest86, apache benchmark (ab) Application performance – Migrating a running Web Server Continuously serves a 256 KB file to a remote client 2015-5-15PPI, Fudan University23

24. Questions for Evaluation Whether Vagrant has an acceptable total migration and minimal downtime? Whether applications during the migration can still provide acceptable performance? Is Vagrant’s performance comparable to other migration framework, eg. Xen and KVM? 2015-5-15PPI, Fudan University24

25. Total Migration Time 2015-5-15PPI, Fudan University25 Vagrant adopts batched memory transfer while KVM uses one-by-one manner

26. Migration Downtime 2015-5-15PPI, Fudan University26 Downtime in Vagrant in comparable to homogeneous live migration

27. Migrating a Running Web Server 2015-5-15PPI, Fudan University27 Memory Pre-copy, about 10% downgrade Less than 1 second

28. Related Work Homogenous live migration over LAN – Vmotion, Xen (Clark et al. ’05) and KVM Live wide-area migration – Bradford et al. (vee’07) – “VM Turntable” (FGCS’06) Tons of virtualization related work 2015-5-15PPI, Fudan University28

29. Conclusion Heterogeneous live VM migration framework – General abstraction of VMs and migration protocol – No changes to hosted OS and application – Minor changes to VMM abstraction – Comparable downtime and end-to-end time 2015-5-15PPI, Fudan University29

30. Questions? Thank you! 302015-5-15PPI, Fudan University

31. Heterogeneous Live Migration of Virtual Machines Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang Parallel Processing Institute, Fudan University http://ppi.fudan.edu.cn

32. Backup Slides Control commands in Vagrant CPU state in Vagrant I/O devices configured for the VM Memory migrating algorithm 2015-5-15PPI, Fudan University32

33. Control Commands in Vagrant Common set of control commands – Each has a 3 seconds timeout 2015-5-15PPI, Fudan University33

34. CPU State in Vagrant The necessary information to be transferred includes – GPR, control regs, segment regs, debug regs, msr, fpu regs, sysenter registers – Time stamp counter – Pending events 2015-5-15PPI, Fudan University34

35. I/O devices configured for the VM The migrated VM is configured with – Keyboard and mouse – Rtc – PIC/APIC/IOAPIC – PIT – IDE disk – NIC – Serial port – VGA 2015-5-15PPI, Fudan University35

36. Memory migrating algorithm Algorithm implemented by Vagrant – Pre-copy in batched manner Batch size is 1024 2015-5-15PPI, Fudan University36