1. Xen and the Art of Virtualization
MHVLUG Meeting
Aug 6th 2006
Sean Dague
sean * dague * net

2. Brief History of Virtualization
First introduced virtualization for development environments
5 Years later, it became clear that this was useful for production environments
S/390 / zSeries / System z
Many iterations of hardware / software coevolution to make virtualization faster and more efficient
Big Unix
Virtualization coevolved with the hardware
In all these cases a single vendor was coevolving hardware and software to make virtualization easy to use, and efficient.

3. Problems in x86 Land
The Intel x86 architecture was never designed to be virtualizable
00/full_papers/robin/robin_html/index.html
x86 CPUs have 4 modes of operation, known as the 4 rings. Ring 0 is the most priviledged and ring 3 the least.
There are 17 sensitive instructions in x86 that can be executed in ring 3. Results end up unexpected.
This prevents running arbitrary guest OS in a less priveiledged ring

4. x86 Virtualization Flavors
Containers: OpenVZ, Vservers
Add OS hooks to enforce application namespaces
All applications running under a single kernel instance
Full virtualization: VMware, VirtualPC, QEMU
Run multiple unmodified guest OSes
Requires binary rewriting and emulation tricks (i386 is not virtualization friendly)
Para-virtualization: Xen, Denali
Create idealized architecture, port OS to that architecture
Requires Source Code for Operating System

5. Xen Remember, there are 17 instructions that are sensitive
You can either spend a lot of time emulating
.... or just not do them
“It hurts if I touch it.” “Then don't touch it!”
Xen takes the para-virtualization approach
Guest operating systems are modified to make explicit hypercalls to the hypervisor instead of using troublesome instructions
Approach requires access to OS source code
Linux 2.4, 2.6, NetBSD, FreeBSD, Plan9 all ported to Xen
A Windows port has shown to be possible, though MS source license is required
A Xen Running environment
Xen runs in Ring 0 – where OS used to run
Modified OS runs in Ring 1 or 2 – hasn't be used since OS/2 days
Userspace remains in Ring 3 – userspace uneffected

6. Xen in Pictures SMP VT-x Xen Virtual Machine Monitor 32/64bit VM0 VM1
Device
Manager &
Control s/w
Unmodified
User
Software
Unmodified
User
Software
Unmodified
User
Software
GuestOS
(XenLinux)
GuestOS
(XenLinux)
GuestOS
(XenLinux)
Unmodified
GuestOS
(WinXP))
Back-End
Back-End
SMP
Native
Device
Driver
Native
Device
Driver
Front-End
Device Drivers
Front-End
Device Drivers
VT-x
Control IF
Safe HW IF
Event Channel
Virtual CPU
Virtual MMU
Xen Virtual Machine Monitor
32/64bit
Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE)

7. Xen Terminology Domains – running guest Operating Systems
There is no concept of an offline domain. Domain lifecycle is much more like process lifecycle in a UNIX environment
Dom0 – The first domain, has administrative priveldges and controls
Xen contains no device drivers
A Dom0 must be started by Xen on boot
Dom0 is a Linux 2.6 kernel (no other OS has been built for Dom0)
As of Xen 3.0.2, only 1 Dom0 is allowed
Can be thought of as “Host OS”, though not strictly true
Root in Dom0 is more powerful than Root in general, as Dom0 can directly map memory of any DomU

8. Xen Terminology
DomU – Additional domains. These do not have administrative privs.
Can be thought of as “Guest OS”
Where normal workloads would run in Xen
Xend / Xm – Xen Management interface
A python daemon and command line through which all xen management occurs
Python daemon is needed because significant state is required to support DomU reboots, and Migration
man xm for full details
XenStore / Xenbus – A “registry” and fake device bus used in virtual device bringup

9. Xen Terminology
Frontend / Backend Drivers – Para Virtual Device drivers
All virtual device drivers have a front end in DomU
Back end in Dom0
Communicate across event channels
Current Devices:
vbd – virtual block device
vif – virtual interface
vtpm – virtual tpm
Live Migration – Migration of a DomU from 1 Dom0 to another, without DomU service interuption
Highly cool Xen demo, uses shadow page tables to allow copying of domain state while running
Final cutover is ~30 ms downtime

10. Using Xen

11. Xen Requirements CPU => PII or better
Realistically > 2 Ghz if you have lots of images
x86_64 supported
Memory => > 1 GB recommended
Need memory for Xen, Dom0, and DomUs
Memory can not be overcommitted
Devices => NIC & HD
Often good to have 2 NICs, one for management of Dom0, one for DomU data traffic
Linux Install for Dom0
Nearly any distro can be made to work

12. Getting Xen Yourself Xen is included in many distros already
FC5, SUSE 10.1, SLES10, Gentoo
Download from -
Binary builds, rpms, live cd
Get the latest and greatest bits
get mercurial (
hg clone
cd xen-unstable.hg
make world
make install

13. Xen grub.conf # Grub conf file
# grub is required due to multiboot support for Xen
title Xen 3.0 / XenLinux 2.6
kernel /boot/xen-3.0.gz dom0_mem=262144
module /boot/vmlinuz-2.6-xen root=/dev/sda4 ro console=tty0
module /boot/initrd-2.6-xen.img

14. A Xen Guest Config kernel = "/boot/vmlinuz-2.6.16.13-xen" memory = 64
name = "ExampleDomain"
vif = [ 'mac=00:16:3e:00:00:11', 'bridge=xenbr0' ]
disk = [ 'phy:sda7,hda1,w' ]
root = "/dev/hda1 ro"
#on_poweroff = 'destroy'
#on_reboot = 'restart'
#on_crash = 'restart'

15. Guest Disk Options Objects are exported to Xen DomUs as raw partitions
Exported Devices can be one of:
raw partition
loopback mount device
logical volume
blktap files
Guests can have swap (which is important for memory constrained systems)

16. Guest Root Partition Look very similar to bare linux root partition
Can get samples from
Differences
Devices will be different
virtual consoles don't exist in the same way

17. Guest Network Options Guest NICs appears as vifX.Y in Dom0
Dom0 can use any standard Linux networking mechanisms to get guests on the network
bridging – bind all vifs + physical network to a single bridge device
routing – setup routes per standard mechanisms
nat – use iptables to nat out guests

18. Common Xen Usage Virtual Colocation
users can all have root access to their own systems
Virtual Build Farm
Single machine has many running OS versions at the same time
Server Consolodation
Moving many workloads to a single machine to increase power

19. Full Virtualization with Xen
Intel and AMD have seen the value of virtualization
Released processors that support virtualization
Intel – Vtx (Core2Duo)
AMD – SVM (AM2)
Full Virtualization allows running of Legacy Oses which can not be modified
Old RHEL, SLES
MS Windows ;)
Hey, I'm a Linux guy

20. Getting Help Mailing List xen-users@lists.xensource.com IRC
#xen on irc.oftc.net
##xen on irc.freenode.net