1. Xen and the art of.. Open Source Virtualisation & Consolidation Kris Buytaert CTO, X-Tend kris.buytaert@x-tend.be

2. Whoami Linux and Open Source Consultant „Infrastructure Architect“ Linux since 0.98 IANAKH Senior Consultant/CTO @ x-tend.be

3. WARNING ! Or Credits ;) Lots of stuf in this presentation is taken from Ian’s presentation from OLS 2005

4. What is Xen ? Xen is a virtual machine monitor –for x86 –execution of multiple guest operating systems –unprecedented levels of performance and resource isolation. Xen is Open Source software Fully functional ports of Linux 2.4 and 2.6 running over Xen

5. Virtualisation vs Virtualisation Single OS image: Ensim, Vservers, Solaris Zones –Group user processes into resource containers –Hard to get strong isolation –Typically used for Hosting/Virtual Hosting/ISP Chroot’s on steroids

6. Virtualisation vs Virtualisation Full virtualization: VMware, VirtualPC, Qemu –Run multiple unmodified guest OSes –Hard to efficiently virtualize x86 –No Access to the hardware –No Realtime guarantees –Problematic for certain privileged instructions

7. Virtualisation vs Virtualisation Para-virtualization: UML, Xen, –Run multiple guest OSes ported to special arch –Arch Xen/x86 is very close to normal x86 –Some Access to underlying hardware –Better Performance –Needs modification to the OS

8. Qemu QEMU is a generic and open source processor emulator with 2 modes Full System Emulation (a full PC) User Mode Emulation (Linux Only) Support for multiple CPU types.

9. Qemu “Slow” (additional acceleration module available) Uses sparse loopback devices Supports multiple OS's License issues => Great for Quick install tests, boots from CD as it emulates the whole machine.

10. Xen Secure isolation between Vms Resource controle and QOS Only guest kernel needs to be ported Execution performance is close to real speed Hardware support = Linux Hardware Support Live Relocation of Vms between nodes

11. Xen “ParaVirtualization” Provides some exposure to the actual hardware –Performance increase –OS Needs to be modified –Multiplexes resources at OS granularity (vs Process level granularity) 100 virtual OS's per machine

12. Xen X86 supports 4 levels of privileges –0 for OS, and 3 for applications –Xen downgrades the privilege of Oses Xen exposes a set of simple device abstractions

13. Xen Domain0 hosts the application-level management software –Creation and deletion of virtual network interfaces and block devices

14. Porting an OS to Xen X86 Like Priviledged instructions –are replaced with Xen hypercalls –for Linux 2.6 only arch-dep files were modified Network Device Driver Block Device Driver Source code availaiblity ! <2% of code-base

15. Xen & Networking Virtual firewall-router attached to all domains Round-robin packet scheduler To send a packet, enqueue a buffer descriptor into the transmit ring Bridging

16. Xen & Disk Access Only Domain0 has direct access to disks Other domains need to use virtual block devices –Use the I/O ring –Reorder requests prior to enqueuing them on the ring –If permitted, Xen will also reorder requests to improve performance

17. Xen and Memory Reserved at domain creation Statically Partitioned among domains Balloon driver enables memory reallocation

18. Xen 2.0 Arch Event Channel Virtual MMUVirtual CPU Control IF Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE) Native Device Driver GuestOS (XenLinux) Device Manager & Control s/w VM0 Native Device Driver GuestOS (XenLinux) Unmodified User Software VM1 Front-End Device Drivers GuestOS (XenLinux) Unmodified User Software VM2 Front-End Device Drivers GuestOS (XenBSD) Unmodified User Software VM3 Safe HW IF Xen Virtual Machine Monitor Back-End

19. Xen 3.0 Arch Event Channel Virtual MMUVirtual CPU Control IF Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE) Native Device Driver GuestOS (XenLinux) Device Manager & Control s/w Native Device Driver GuestOS (XenLinux) Unmodified User Software Front-End Device Drivers GuestOS (XenLinux) Unmodified User Software Front-End Device Drivers Unmodified GuestOS (WinXP)) Unmodified User Software Safe HW IF Xen Virtual Machine Monitor Back-End VT-x AGP ACPI PCI SMP

20. Xen, Live VM Migration Why ? –Manage a pool of Vms on a Cluster –Hardware upgrades –Loadbalancing VM's in a cluster Why Difficult –Migrate State of VM's –Sessions/Response time for databases & webservers Requires Shared Storage

21. From: Wim Coekaerts Cc: xen-devel@lists.sourceforge.net Subject: Re: [Xen-devel] databases and xen? testign it has already been fine. I ran 4 databases each in one domain (oracle10g) and it s been amazingly stable. I have not however done performance testing. soon... Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel Early Adopters

22. ● From: Moshe Bar ● Subject: Re: [Xen-devel] databases and xen? ● Date: Tue, 18 Jan 2005 04:02:01 +0200 ● I have run Mysql and Oracle 9i without any problems on 2.0.0 and 2.0.1 ●, but I didn't have LVM (don't think that would create problems) ● Moshe Early Adopters

23. Getting Started FC4 SLES Or download binaries from http://getxen.org http://getxen.org Don’t use the “RedHat” rpm’s (now) “stable” 2.07 vs “experimental “3.0x”

24. Getting Started (any) Download binary install tarball Check prerequisites Python twisted Bridging utils./install modify grub.conf Reboot

25. Getting Started (FC4) Yum install xen kernel-xen0 kernel-xenU http://people.redhat.com/riel/xen_for_fc4/ modify grub.conf Reboot

26. Domain(0/U) What ? Xen boots Bye bye “x86” HOSTA:/etc/xen/scripts # xm list Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 41.2 Domain0 = management DomainU = virtual machines

27. Typical Xen DomainU installations „Copy“ an existing image Yum –installroot=/path/ -y groupinstall Base Debootstrap Urpmi –root=/path basesystem urpmi ssh-server Yast rpmstrap

28. Xen “disks” LVM Sparse loopback files Actual /dev/hdxy (g)NBD (NFS)

29. Xen configuration /etc/xen/hostname : kernel = "/boot/vmlinuz-2.6.11-1.1366_FC4xenU" memory = 128 name = "dokeos.x-tend.be" nics = 1 extra ="selinux=0 3" vif = ['ip = "10.0.11.13", bridge=xen-br0'] disk = ['phy:vm_volumes/root.dokeos,sda1,w','phy:vm_volumes/var.dokeos,sda3,w','phy:vm_volumes/www.dokeos,sda4,w','phy:vm_volumes/swap.dokeos,sda2,w' ] root = "/dev/sda1 ro"

30. Managing Xen Instances Xm create -c domainname Xm list Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 891 0 r---- 62.3 dokeos.x-tend.be 1 127 1 -b--- 24.6 9601 newhope.x-tend.be 2 127 1 -b--- 177.2 9602 Xm console $id Xm shutdown $id Xm destroy $id

31. Xen Networking /etc/xen/scripts/ Brctl show Echo “1” > /proc/sys/net/ipv4/ip_forward

32. Xen Uses Solving Kernel issues Consolidation Testing Large Scale Deployments Building HA Clusters

33. Solving Netfilter Conflicts Building a truly transparant proxy –Integration of LVS and Tproxy fails –Required multiple machines to work –CONNTRACK module conflicts Used Xen to build this on 1 machine

34. Solving Netfilter Conflicts Building a truly transparant proxy –Integration of LVS and Tproxy fails –Required multiple machines to work –CONNTRACK module conflicts Used Xen to build this on 1 machine

35. Lowering the # of machines Telco Environment with maximum 6+16x(2x3+6))=198 machines (actually 6+2x(2x3+6)= 30) –Consolidated already 1 application now 6 + 16x(2x2+6) = 166 machines (6+2x(2x2+6) = 24) we moved already 2 redundant applications to 1 of the 6 shared machines more are following

36. Cluster Consolidation High Availablilty Failover many to 1 ? –Failover all physical machines to multiple virtual machines on 1 physical machine.

37. Xen Future 3.0 VT Vanderpool Technlogy ==> No more ports required –Also non free OS’s Embrace and enhance management tools Infiniband support etc

38. Conclusions Xen is a complete and robust GPL VMM Outstanding performance and scalability Excellent resource control and protection Vibrant development community Strong vendor support http://xen.sf.net

39. Stage 0: pre-migration Stage 1: reservation Stage 2: iterative pre-copy Stage 3: stop-and-copy Stage 4: commitment Relocation Strategy VM active on host A Destination host selected (Block devices mirrored) Initialize container on target host Copy dirty pages in successive rounds Suspend VM on host A Redirect network traffic Synch remaining state Activate on host B VM state on host A released

40. Pre-Copy Migration: Round 1

45. Pre-Copy Migration: Round 2

50. Pre-Copy Migration: Final