Xen and the Art of Virtualization MHVLUG Meeting Aug 6th 2006 Sean Dague sean * dague * net

Brief History of Virtualization First introduced virtualization for development environments 5 Years later, it became clear that this was useful for production environments S/390 / zSeries / System z Many iterations of hardware / software coevolution to make virtualization faster and more efficient Big Unix Virtualization coevolved with the hardware In all these cases a single vendor was coevolving hardware and software to make virtualization easy to use, and efficient.

Problems in x86 Land The Intel x86 architecture was never designed to be virtualizable http://www.usenix.org/publications/library/proceedings/sec20 00/full_papers/robin/robin_html/index.html x86 CPUs have 4 modes of operation, known as the 4 rings. Ring 0 is the most priviledged and ring 3 the least. There are 17 sensitive instructions in x86 that can be executed in ring 3. Results end up unexpected. This prevents running arbitrary guest OS in a less priveiledged ring

x86 Virtualization Flavors Containers: OpenVZ, Vservers Add OS hooks to enforce application namespaces All applications running under a single kernel instance Full virtualization: VMware, VirtualPC, QEMU Run multiple unmodified guest OSes Requires binary rewriting and emulation tricks (i386 is not virtualization friendly) Para-virtualization: Xen, Denali Create idealized architecture, port OS to that architecture Requires Source Code for Operating System

Xen Remember, there are 17 instructions that are sensitive You can either spend a lot of time emulating .... or just not do them “It hurts if I touch it.” “Then don't touch it!” Xen takes the para-virtualization approach Guest operating systems are modified to make explicit hypercalls to the hypervisor instead of using troublesome instructions Approach requires access to OS source code Linux 2.4, 2.6, NetBSD, FreeBSD, Plan9 all ported to Xen A Windows port has shown to be possible, though MS source license is required A Xen Running environment Xen runs in Ring 0 – where OS used to run Modified OS runs in Ring 1 or 2 – hasn't be used since OS/2 days Userspace remains in Ring 3 – userspace uneffected

Xen in Pictures SMP VT-x Xen Virtual Machine Monitor 32/64bit VM0 VM1 Device Manager & Control s/w Unmodified User Software Unmodified User Software Unmodified User Software GuestOS (XenLinux) GuestOS (XenLinux) GuestOS (XenLinux) Unmodified GuestOS (WinXP)) Back-End Back-End SMP Native Device Driver Native Device Driver Front-End Device Drivers Front-End Device Drivers VT-x Control IF Safe HW IF Event Channel Virtual CPU Virtual MMU Xen Virtual Machine Monitor 32/64bit Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE)

Xen Terminology Domains – running guest Operating Systems There is no concept of an offline domain. Domain lifecycle is much more like process lifecycle in a UNIX environment Dom0 – The first domain, has administrative priveldges and controls Xen contains no device drivers A Dom0 must be started by Xen on boot Dom0 is a Linux 2.6 kernel (no other OS has been built for Dom0) As of Xen 3.0.2, only 1 Dom0 is allowed Can be thought of as “Host OS”, though not strictly true Root in Dom0 is more powerful than Root in general, as Dom0 can directly map memory of any DomU

Xen Terminology DomU – Additional domains. These do not have administrative privs. Can be thought of as “Guest OS” Where normal workloads would run in Xen Xend / Xm – Xen Management interface A python daemon and command line through which all xen management occurs Python daemon is needed because significant state is required to support DomU reboots, and Migration man xm for full details XenStore / Xenbus – A “registry” and fake device bus used in virtual device bringup

Xen Terminology Frontend / Backend Drivers – Para Virtual Device drivers All virtual device drivers have a front end in DomU Back end in Dom0 Communicate across event channels Current Devices: vbd – virtual block device vif – virtual interface vtpm – virtual tpm Live Migration – Migration of a DomU from 1 Dom0 to another, without DomU service interuption Highly cool Xen demo, uses shadow page tables to allow copying of domain state while running Final cutover is ~30 ms downtime

Using Xen

Xen Requirements CPU => PII or better Realistically > 2 Ghz if you have lots of images x86_64 supported Memory => > 1 GB recommended Need memory for Xen, Dom0, and DomUs Memory can not be overcommitted Devices => NIC & HD Often good to have 2 NICs, one for management of Dom0, one for DomU data traffic Linux Install for Dom0 Nearly any distro can be made to work

Getting Xen Yourself Xen is included in many distros already FC5, SUSE 10.1, SLES10, Gentoo Download from - http://getxen.org Binary builds, rpms, live cd Get the latest and greatest bits get mercurial (http://www.selenic.com/mercurial/) hg clone http://xenbits.xensource.com/xen-unstable.hg cd xen-unstable.hg make world make install

Xen grub.conf # Grub conf file # grub is required due to multiboot support for Xen title Xen 3.0 / XenLinux 2.6 kernel /boot/xen-3.0.gz dom0_mem=262144 module /boot/vmlinuz-2.6-xen root=/dev/sda4 ro console=tty0 module /boot/initrd-2.6-xen.img

A Xen Guest Config kernel = "/boot/vmlinuz-2.6.16.13-xen" memory = 64 name = "ExampleDomain" vif = [ 'mac=00:16:3e:00:00:11', 'bridge=xenbr0' ] disk = [ 'phy:sda7,hda1,w' ] root = "/dev/hda1 ro" #on_poweroff = 'destroy' #on_reboot = 'restart' #on_crash = 'restart'

Guest Disk Options Objects are exported to Xen DomUs as raw partitions Exported Devices can be one of: raw partition loopback mount device logical volume blktap files Guests can have swap (which is important for memory constrained systems)

Guest Root Partition Look very similar to bare linux root partition Can get samples from http://www.oszoo.org/download.html Differences Devices will be different virtual consoles don't exist in the same way

Guest Network Options Guest NICs appears as vifX.Y in Dom0 Dom0 can use any standard Linux networking mechanisms to get guests on the network bridging – bind all vifs + physical network to a single bridge device routing – setup routes per standard mechanisms nat – use iptables to nat out guests

Common Xen Usage Virtual Colocation users can all have root access to their own systems Virtual Build Farm Single machine has many running OS versions at the same time Server Consolodation Moving many workloads to a single machine to increase power

Full Virtualization with Xen Intel and AMD have seen the value of virtualization Released processors that support virtualization Intel – Vtx (Core2Duo) AMD – SVM (AM2) Full Virtualization allows running of Legacy Oses which can not be modified Old RHEL, SLES MS Windows ;) Hey, I'm a Linux guy

Getting Help Mailing List xen-users@lists.xensource.com IRC #xen on irc.oftc.net ##xen on irc.freenode.net