Xen and the art of.. Open Source Virtualisation & Consolidation Kris Buytaert CTO, X-Tend

Whoami Linux and Open Source Consultant „Infrastructure Architect“ Linux since 0.98 IANAKH Senior x-tend.be

WARNING ! Or Credits ;) Lots of stuf in this presentation is taken from Ian’s presentation from OLS 2005

What is Xen ? Xen is a virtual machine monitor –for x86 –execution of multiple guest operating systems –unprecedented levels of performance and resource isolation. Xen is Open Source software Fully functional ports of Linux 2.4 and 2.6 running over Xen

Virtualisation vs Virtualisation Single OS image: Ensim, Vservers, Solaris Zones –Group user processes into resource containers –Hard to get strong isolation –Typically used for Hosting/Virtual Hosting/ISP Chroot’s on steroids

Virtualisation vs Virtualisation Full virtualization: VMware, VirtualPC, Qemu –Run multiple unmodified guest OSes –Hard to efficiently virtualize x86 –No Access to the hardware –No Realtime guarantees –Problematic for certain privileged instructions

Virtualisation vs Virtualisation Para-virtualization: UML, Xen, –Run multiple guest OSes ported to special arch –Arch Xen/x86 is very close to normal x86 –Some Access to underlying hardware –Better Performance –Needs modification to the OS

Qemu QEMU is a generic and open source processor emulator with 2 modes Full System Emulation (a full PC) User Mode Emulation (Linux Only) Support for multiple CPU types.

Qemu “Slow” (additional acceleration module available) Uses sparse loopback devices Supports multiple OS's License issues => Great for Quick install tests, boots from CD as it emulates the whole machine.

Xen Secure isolation between Vms Resource controle and QOS Only guest kernel needs to be ported Execution performance is close to real speed Hardware support = Linux Hardware Support Live Relocation of Vms between nodes

Xen “ParaVirtualization” Provides some exposure to the actual hardware –Performance increase –OS Needs to be modified –Multiplexes resources at OS granularity (vs Process level granularity) 100 virtual OS's per machine

Xen X86 supports 4 levels of privileges –0 for OS, and 3 for applications –Xen downgrades the privilege of Oses Xen exposes a set of simple device abstractions

Xen Domain0 hosts the application-level management software –Creation and deletion of virtual network interfaces and block devices

Porting an OS to Xen X86 Like Priviledged instructions –are replaced with Xen hypercalls –for Linux 2.6 only arch-dep files were modified Network Device Driver Block Device Driver Source code availaiblity ! <2% of code-base

Xen & Networking Virtual firewall-router attached to all domains Round-robin packet scheduler To send a packet, enqueue a buffer descriptor into the transmit ring Bridging

Xen & Disk Access Only Domain0 has direct access to disks Other domains need to use virtual block devices –Use the I/O ring –Reorder requests prior to enqueuing them on the ring –If permitted, Xen will also reorder requests to improve performance

Xen and Memory Reserved at domain creation Statically Partitioned among domains Balloon driver enables memory reallocation

Xen 2.0 Arch Event Channel Virtual MMUVirtual CPU Control IF Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE) Native Device Driver GuestOS (XenLinux) Device Manager & Control s/w VM0 Native Device Driver GuestOS (XenLinux) Unmodified User Software VM1 Front-End Device Drivers GuestOS (XenLinux) Unmodified User Software VM2 Front-End Device Drivers GuestOS (XenBSD) Unmodified User Software VM3 Safe HW IF Xen Virtual Machine Monitor Back-End

Xen 3.0 Arch Event Channel Virtual MMUVirtual CPU Control IF Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE) Native Device Driver GuestOS (XenLinux) Device Manager & Control s/w Native Device Driver GuestOS (XenLinux) Unmodified User Software Front-End Device Drivers GuestOS (XenLinux) Unmodified User Software Front-End Device Drivers Unmodified GuestOS (WinXP)) Unmodified User Software Safe HW IF Xen Virtual Machine Monitor Back-End VT-x AGP ACPI PCI SMP

Xen, Live VM Migration Why ? –Manage a pool of Vms on a Cluster –Hardware upgrades –Loadbalancing VM's in a cluster Why Difficult –Migrate State of VM's –Sessions/Response time for databases & webservers Requires Shared Storage

From: Wim Coekaerts Cc: Subject: Re: [Xen-devel] databases and xen? testign it has already been fine. I ran 4 databases each in one domain (oracle10g) and it s been amazingly stable. I have not however done performance testing. soon... Early Adopters

● From: Moshe Bar ● Subject: Re: [Xen-devel] databases and xen? ● Date: Tue, 18 Jan :02: ● I have run Mysql and Oracle 9i without any problems on and ●, but I didn't have LVM (don't think that would create problems) ● Moshe Early Adopters

Getting Started FC4 SLES Or download binaries from Don’t use the “RedHat” rpm’s (now) “stable” 2.07 vs “experimental “3.0x”

Getting Started (any) Download binary install tarball Check prerequisites Python twisted Bridging utils./install modify grub.conf Reboot

Getting Started (FC4) Yum install xen kernel-xen0 kernel-xenU modify grub.conf Reboot

Domain(0/U) What ? Xen boots Bye bye “x86” HOSTA:/etc/xen/scripts # xm list Name Id Mem(MB) CPU State Time(s) Console Domain r Domain0 = management DomainU = virtual machines

Typical Xen DomainU installations „Copy“ an existing image Yum –installroot=/path/ -y groupinstall Base Debootstrap Urpmi –root=/path basesystem urpmi ssh-server Yast rpmstrap

Xen “disks” LVM Sparse loopback files Actual /dev/hdxy (g)NBD (NFS)

Xen configuration /etc/xen/hostname : kernel = "/boot/vmlinuz _FC4xenU" memory = 128 name = "dokeos.x-tend.be" nics = 1 extra ="selinux=0 3" vif = ['ip = " ", bridge=xen-br0'] disk = ['phy:vm_volumes/root.dokeos,sda1,w','phy:vm_volumes/var.dokeos,sda3,w','phy:vm_volumes/ ] root = "/dev/sda1 ro"

Managing Xen Instances Xm create -c domainname Xm list Name Id Mem(MB) CPU State Time(s) Console Domain r dokeos.x-tend.be b newhope.x-tend.be b Xm console $id Xm shutdown $id Xm destroy $id

Xen Networking /etc/xen/scripts/ Brctl show Echo “1” > /proc/sys/net/ipv4/ip_forward

Xen Uses Solving Kernel issues Consolidation Testing Large Scale Deployments Building HA Clusters

Solving Netfilter Conflicts Building a truly transparant proxy –Integration of LVS and Tproxy fails –Required multiple machines to work –CONNTRACK module conflicts Used Xen to build this on 1 machine

Solving Netfilter Conflicts Building a truly transparant proxy –Integration of LVS and Tproxy fails –Required multiple machines to work –CONNTRACK module conflicts Used Xen to build this on 1 machine

Lowering the # of machines Telco Environment with maximum 6+16x(2x3+6))=198 machines (actually 6+2x(2x3+6)= 30) –Consolidated already 1 application now x(2x2+6) = 166 machines (6+2x(2x2+6) = 24) we moved already 2 redundant applications to 1 of the 6 shared machines more are following

Cluster Consolidation High Availablilty Failover many to 1 ? –Failover all physical machines to multiple virtual machines on 1 physical machine.

Xen Future 3.0 VT Vanderpool Technlogy ==> No more ports required –Also non free OS’s Embrace and enhance management tools Infiniband support etc

Conclusions Xen is a complete and robust GPL VMM Outstanding performance and scalability Excellent resource control and protection Vibrant development community Strong vendor support

Stage 0: pre-migration Stage 1: reservation Stage 2: iterative pre-copy Stage 3: stop-and-copy Stage 4: commitment Relocation Strategy VM active on host A Destination host selected (Block devices mirrored) Initialize container on target host Copy dirty pages in successive rounds Suspend VM on host A Redirect network traffic Synch remaining state Activate on host B VM state on host A released

Pre-Copy Migration: Round 1

Pre-Copy Migration: Round 2

Pre-Copy Migration: Final