1. Xen and the Art of Virtualization University of Cambridge Presenter: Ashish Gupta

2. Features An open infrastructure for global distributed computing Run multiple services on a single Xenoserver  Envisage running up to 100 per server Secure and accountable execution  Strong isolation, logging and auditing Flexible: low-level execution environment Economical: execute on commodity hardware  (x86)

3. Virtualization techniques Single OS image (Ensim, VServers)  Group user processes into resource container.  Implement new schedulers in the OS to ensure isolation  Hard to retrofit isolation to conventional Oses Full virtualization (VMware, Connectix, Bochs)  Run full OSes as unmodified guests  The VMM enforces resource isolation  But it’s hard to efficiently virtualize uncooperative architectures

4. Paravirtualization Goals Low Virtualization Overhead Performance Isolation Also (Flexibility) Support full-featured multi-user multi- application OSes

5. System Performance

6. Para-virtualization – Principles ? Para-virtualization vs. full-virtualization  Expose guest OS to “real resources” (time, MMU etc.) Better support time sensitive tasks Allows guest OS optimizations Correctness issues The Downside

7. Para-virtualization Mechanisms

8. Three broad aspects Memory Management CPU Device I/O

9. Memory Management The VMWare approach – shadow page tables

10. Modifications Paravirtualization obviates the need for shadow page tables Guest OSes allocate and manage their own page tables HOW ?

11. Mechanism Updates to page tables must be passed to Xen for  validation  Updates may be queued and processed in batches Validation rules (applied to each PTE):  1. only map a page if owned by the requesting guest OS  2. only map a page containing PTEs for read-only access Xen tracks page ownership and current use

12. Memory Management The Xen approach

13. Memory benchmarks

14. CPU Efficient because - Four privilege levels  OS – Ring 1, Applications – Ring 3 Privileged instructions required to be validated and executed by Xen Exceptions Guest OS registers handlers with Xen Para-virtualization  Unchanged handlers “fast handlers” for most exceptions, Xen isn’t involved Page faults – CR2 register read by Xen, so must enter Xen

15. Xen uses the 4-ring model

16. VM ↔ VMM Guest OS  Xen : Hypercalls  Like system calls Xen  Guest OS : Events  Like UNIX signals

17. I/O Virtualization Need to minimize cost of transferring bulk data via Xen  Copying costs time  Copying pollutes caches  Copying requires intermediate memory Device classes  Net  Disk  Graphics

18. I/O Virtualization Use rings of buffer descriptors  Descriptors are small: cheap to copy and validate  Descriptors refer to bulk data  No need to map or copy the data into Xen’s address space  Exception: checking network packet headers prior to TX Use zero-copy DMA to transfer bulk data between hardware and guest OS  Net TX: DMA packet payload separately from validated packet header  Net RX: Page-flip receive buffers into guest address space

20. TCP Benchmarks

21. Effect of I/O and OS interaction SPEC INT2000 score CPU Intensive Little I/O and OS interaction SPEC WEB99 180Mb/s TCP traffic Disk read-write on 2GB dataset

22. Scalability

23. Performance Isolation 4 domains  2 PostgreSQL, SPECWEB99 workloads  2 anti-social workloads Disk bandwidth hog: huge number of small file creations Fork Bomb The Bad guys could not kill the Good guys In Native Linux: Rendered the machine completely unusable !

25. Denali Isolation Kernel University of Washington

26. Motivation Functionality pushed into the network:  Google, IMDB, Hotmail, Amazon, EBay, online banking, …lots! Major players use dedicated hardware. Lesser services find that cumbersome, expensive and limiting:  Hardware, rack space, bandwidth Big deployment barrier for little services.

27. Virtual hosting Third-party hardware, with small services multiplexed on machines. Need the ability to run untrusted code. Likewise for CDNs for dynamic content.

28. Goals:  strong security  resource control. Don’t need: resource sharing. Conventional OSs do not isolate enough Spectrum of Ideas !  #1: OSs with Perf isolation  #2: OSs and sandboxing  #3: Exo- / Micro- kernels  #4: Conventional VMs  #5: Isolation Kernel

29. Isolation Kernel Focus here is on  Performance with Scaling and  Isolation/Security Reconsider the exposed Virtual Architecture Downside  (Linux port ?)

30. Scaling Arguments

31. Denali Mechanism

32. Overall Architecture

33. ISA Biggest challenge for x86 virtualization:  Ambiguous instruction semantics  No support for ambiguous instructions Two virtual Instructions  Idle-with-timeout  Terminate execution

34. Memory Architecture Simple DOS-like architecture: No virtual MMU Why ?  TLB Problems on x86 : Hardware mapped: Inflexible  Avoids TLB Flushes Optional Virtual MMU ?

35. I/O and Interrupt Model Simpler interfaces to NIC, Disk, keyboard, console and Timer  Avoid the “chatty” interfaces Interrupt Model  Physical Interrupts  Virtual interrupts Interrupt Dispatch Model  Delays and batches interrupts for non-running VMs  Timing related interrupts ?? Real time apps, games etc ?

36. Implementation Round robin scheduling  Idle-with-timeout compensated with a higher priority for next quantum. Can use existing compilers (gcc) to generate code VMs are paged in on demand. VMM always in core

37. Memory Virtualized 16MB of physical address space per VM (since no virtual MMU). Recently they added a virtual MIPS-style virtual MMU, so guest OS can virtualize its apps’ space. Overhead? - Pre-allocated, strided swap space. No sharing, so each VM’s space is contiguous.

38. Networked IO Ethernet driver moved from guest OS to Denali. Rest of TCP/IP stack stays. This suffices for early-demuxing received packets into the appropriate VM. Virtual packet send/recv is 1 PIO each

39. Guest OS Guest OS: currently only a library, with no simulated protection boundary there. Supports a POSIX subset. Different from a traditional VM : OS more like a process: single user, single task OS ?  Flexibility ?

40. Evaluation Network Latency

41. TCP, HTTP throughput TCP: BSD-Linux607Mb/s Denali-Linux569Mb/

42. Fair comparison? Denali with library kernel compared against BSD: both have one protection boundary Denali-Linux will have one real and one simulated protection boundary: different ?

43. Batching Reduction in context switching frequency

44. Idle-with-timeout

45. Scalability In-core regime – constant performance disk bound regime - problems

46. Scalability and block size Internal fragmentation!

47. Evaluation summary Good performance and scalability due to architectural modifications various techniques Is the lib OS representative of a real OS?