1. Virtualization-optimized architectures
Yu-Shen Ng
Group Product Manager
ESX Server
VMware

2. Agenda: Virtualization-optimized architectures
CPU-optimizations
Memory-optimizations
OS-optimizations

3. Background context: the full virtualization software stack
Third-party Solutions
Distributed
Services
DRS
VMotion
DAS
Provisioning
Backup
VirtualCenter
Management
and
Virtualization VM VM VM VM
Third-
Party
Agents
SDK / VirtualCenter Agent
VMX
VMX
VMX
VMX
Virtual Machine
Monitor
I/O Stack
VMM
VMM
VMM
VMM
Service
Console
Device Drivers
Distributed Virtual Machine
File System
Virtual NIC
and Switch
Resource Management
CPU Scheduling
Memory Scheduling
Storage Bandwidth
Network Bandwidth
Enterprise
Class
Virtualization
Functionality
Storage Stack
Network Stack
Device Drivers
2 minutes
This is what the core ESX Server architecture looks like today.
The VMM provides the abstraction of a virtual machine to a guest operating system. It allows a guest to think it is running on real hardware.
The VMkernel manages the hardware and drives storage and network I/O.
The Service Console is a privileged VM running Linux. It hosts these VMX processes that assist the VMM in
virtualizing and driving non-performance critical I/O devices. Each VMM is associated with a VMX process.
ESX Server enhances core virtualization with enterprise class functionality.
The resource management subsystem provides efficient allocation of the hardware resources to the running
VMs. It enhances consolidation ratios by allowing many VMs to run with tight resource guarantees
on a single server box.
The storage subsystem contains a file system for efficient storage of large virtual disk files. It is a distributed
filesystem that allows multiple ESX Servers to mount the same filesystem volumes. It supports clustering and
backup functionality. The storage subsystems also supports SAN multipathing for high-availability.
The network subsystem contains a virtual switch that supports NIC teaming for high-availability and VLANs for
enterprise class network management.
Hardware vendors run their system management agents, such as IBM Director and HP Insight Manager in the
Service Console. We provide the necessary interfaces for these agents to access the underlying hardware.
Finally, to support management of a cluster of ESX Server computers, the Service Console hosts an SDK and a
VirtualCenter Agent that allows VirtualCenter and third-party solutions to provide distributed virtualization services.
The SDK is a system management API.
The rest of the talk will consider how this system organization will evolve in the future.
ESX Server
VMkernel Hardware Interface
Hardware

4. Virtualization Software Technology
VMM
VMM
VMM
Enhanced
Functionality
Base Functionality (e.g. scheduling)
Hypervisor
Virtual Machine Monitor (VMM)
SW component that implements virtual machine hardware abstraction
Responsible for running the guest OS
Hypervisor
Software responsible for hosting and managing virtual machines
Run directly on the hardware
Functionality varies greatly with architecture and implementation

5. Agenda: Virtualization-optimized architectures
CPU-optimizations
Memory-optimizations
OS-optimizations

6. Background: virtualizing the whole system
Three components to classical virtualization techniques
Many virtualization technologies focus on handling privileged instructions
Privileged instruction virtualization
Handling privileged instructions by de-privileging or ring compression
Memory virtualization
Memory partitioning and allocation of physical memory
Device and I/O virtualization
Routing I/O requests between virtual devices and physical hardware

7. CPU Virtualization
Three components to classical virtualization techniques
Many virtualization technologies focus on handling privileged instructions
Privileged instruction virtualization
Handling privileged instructions by de-privileging or ring compression
Memory virtualization
Memory partitioning and allocation of physical memory
Device and I/O virtualization
Routing I/O requests between virtual devices and physical hardware

8. In traditional OS’s (e.g. Windows)
What are privileged instructions and how are they traditionally handled?
In traditional OS’s (e.g. Windows)
OS runs in privileged mode
OS exclusively “owns” the CPU hardware & can use privileged instructions to access the CPU hardware
Application code has less privilege
In mainframes/traditional VMM’s
VMM needs highest privilege level for isolation and performance
Use either “ring compression” or “de-privileging” technique
Run privileged guest OS code at user-level
Privileged instructions trap, and are emulated by VMM
This way, the guest OS does NOT directly access the underlying hardware
Apps
Ring 3
Guest OS
Ring 0
Apps
Ring 3
Guest OS
VMM
Ring 0

9. Handling Privileged Instructions for x86
De-privileging not possible with x86!
Some privileged instructions have different semantics at user-level: “non-virtualizable instructions”
VMware uses direct execution and binary translation (BT)
BT for handling privileged code
Direct execution of user-level code for performance
Any unmodified x86 OS can run in virtual machine
Virtual machine monitor lives in the guest address space

10. Protecting the VMM (since it lives in guest’s address for BT perf.)
Need to protect VMM and ensure isolation
Protect virtual machines from each other
Protect VMM from virtual machines
VMware traditionally relies on memory segmentation hardware to protect the VMM
VMM lives at top of guest address space
Segment limit checks catch writes to VMM area
VMM
4GB
Summary: since the VMM is in the same address space as guests (for BT performance benefits), segment limit checks protect the VMM

11. CPU assists: Intel VT-x / AMD-V
CPU vendors are embracing virtualization
Intel Virtualization Technology (VT-x)
AMD-V
Key feature is new CPU execution mode (root mode)
VMM executes in root mode
Allows x86 virtualization without binary translation or paravirtualization
Non-root mode
Apps
Apps
Ring 3
Guest OS
Guest OS
Ring 0
Root mode VM
exit VM
enter
Virtual Machine Monitor (VMM)

12. 1st Generation CPU Assist
Initial VT-x/AMD-V hardware targets privileged instructions
HW is an enabling technology that makes it easier to write a functional VMM
Alternative to using binary translation
Initial hardware does not guarantee highest performance virtualization
VMware binary translation outperforms VT-x/AMD-V
Current VT-x/AMD-V
Privileged instructions
Yes
Memory virtualization No
Device and I/O virtualization

13. Challenges of Virtualizing x86-64
Older AMD64 and Intel EM64T architectures did not include segmentation in 64-bit mode
How do we protect the VMM?
64-bit guest support requires additional hardware assistance
Segment limit checks available in 64-bit mode on newer AMD processors
VT-x can be used to protect the VMM on EM64T
Requires trap-and-emulate approach instead of BT

14. Agenda: Virtualization-optimized architectures
CPU-optimizations
Memory-optimizations
OS-optimizations

15. Memory Virtualization
One of the most challenging technical problems in virtualizing the x86 architecture
Privileged instruction virtualization
Handling privileged instructions by de-privileging or ring compression
Memory virtualization
Memory partitioning and allocation of physical memory
Device and I/O virtualization
Routing I/O requests between virtual devices and physical hardware

16. Review of the “Virtual Memory” concept
Process 1
Process 2 GB GB
Virtual Memory VA
Physical Memory PA
Modern operating systems provide virtual memory support
Applications see a contiguous address space that is not necessarily tied to underlying physical memory in the system
OS keeps mappings of virtual page numbers to physical page numbers
Mappings are stored in page tables
CPU includes memory management unit (MMU) and translation lookaside buffer (TLB) for virtual memory support

17. Virtualizing Virtual Memory
VM 1
VM 2
Process 1
Process 2
Process 1
Process 2
Virtual Memory VA
Physical Memory PA
Machine Memory MA
In order to run multiple virtual machines on a single system, another level of memory virtualization must be done
Guest OS still controls mapping of virtual address to physical address: VA -> PA
In virtualized world, guest OS cannot have direct access to machine memory
Each guest’s physical memory is no longer the actual machine memory in system
VMM maps guest physical memory to the actual machine memory: PA -> MA

18. Virtualizing Virtual Memory: Shadow Page Tables
VM 1
VM 2
Process 1
Process 2
Process 1
Process 2
Virtual Memory VA
Physical Memory PA
Machine Memory MA
VMM uses “shadow page tables” to accelerate the mappings
Directly map VA -> MA
Can avoid the two levels of translation on every access
Leverage TLB hardware for this VA -> MA mapping
When guest OS changes VA -> PA, the VMM updates the shadow page tables

19. Future Hardware Assist at the Memory level
Both AMD and Intel have announced roadmap of additional hardware support
Memory virtualization (Nested paging, Extended Page Tables)
Device and I/O virtualization (VT-d, IOMMU)
HW Solution
Privileged instructions
VT-X / AMD-V
Memory virtualization
EPT / NPT
Device and I/O virtualization
Intelligent devices, IOMMU / VT-d

20. Nested Paging / Extended Page Tables
VM 1
VM 2
Process 1
Process 2
Process 1
Process 2
Virtual Memory VA
Physical Memory PA
Machine Memory MA
Hardware support for memory virtualization is on the way
AMD: Nested Paging / Nested Page Tables (NPT)
Intel: Extended Page Tables (EPT)
Conceptually, NPT and EPT are identical
Two sets of page tables exist: VA -> PA and PA -> MA
Processor HW does page walk for both VA -> PA and PA -> MA

21. Benefits of NPT/EPT Performance Reducing memory consumption
Compute-intensive workloads already run well with binary translation/direct execution
NPT/EPT will provide noticeable performance improvement for workloads with MMU overheads
Hardware addresses the performance overheads due to virtualizing the page tables
With NPT/EPT, even more workloads become candidates for virtualization
Reducing memory consumption
Shadow page tables consume additional system memory
Use of NPT/EPT will reduce “overhead memory”
Today, VMware uses HW assist in very limited cases
NPT/EPT provide motivation to use HW assist much more broadly
NPT/EPT require usage of AMD-V/VT-x

22. Flexible VMM Architecture
Flexible “multi-mode” VMM architecture
Separate VMM per virtual machine
Select mode that achieves best workload-specific performance based on CPU support
Today
32-bit: BT
64-bit: BT or VT-x
Tomorrow
32-bit: BT or AMD-V/NPT or VT-x/EPT
64-bit: BT or VT-x or AMD-V /NPT or VT-x/EPT
Same VMM architecture for ESX Server, Player, Server, Workstation and ACE VM VM VM VM
. . .
. . .
BT/VT VMM64 BT
VMM32
BT/VT
VMM64 BT
VMM32

23. Agenda: Virtualization-optimized architectures
CPU-optimizations
Memory-optimizations
OS-optimizations

24. CPU Virtualization Alternatives: OS Assist
Three alternatives for virtualizing CPU (handling those troublesome “privileged instructions”)
Binary translation
Hardware assist (first generation)
OS assist or paravirtualization
Binary Translation
Current HW Assist
Paravirtualization
Compatibility
Excellent
Performance
Good
Average
VMM sophistication
High

25. Paravirtualization
Paravirtualization can also address CPU virtualization
Modify the guest OS to remove the troublesome “privileged instructions”
Export a simpler architecture to OS
Binary Translation
Current HW Assist
Paravirtualization
Compatibility
Excellent
Performance
Good
Average
VMM sophistication
High

26. Paravirtualization’s compatibility rating: poor
Paravirtualization can also address CPU virtualization
Modify the guest OS to remove non-virtualizable instructions
Export a simpler architecture to OS
Cannot support unmodified guest OSes (e.g., Windows 2000/XP)
Binary Translation
Current HW Assist
Paravirtualization
Compatibility
Excellent
Poor
Performance
Good
Average
VMM sophistication
High

27. Paravirtualization’s performance rating: excellent
Paravirtualization can also address CPU virtualization
Modify the guest OS to remove non-virtualizable instructions
Export a simpler architecture to OS
Cannot support unmodified guest OSes (e.g., Windows 2000/XP)
Higher performance possible
Paravirtualization not limited to CPU virtualization
Binary Translation
Current HW Assist
Paravirtualization
Compatibility
Excellent
Poor
Performance
Good
Average
VMM sophistication
High

28. Paravirtualization’s VMM sophistication: average
Paravirtualization can also address CPU virtualization
Modify the guest OS to remove non-virtualizable instructions
Export a simpler architecture to OS
Cannot support unmodified guest OSes (e.g., Windows 2000/XP)
Higher performance possible
Paravirtualization not limited to CPU virtualization
Relatively easy to add paravirtualization support; very difficult to add binary translation
Binary Translation
Current HW Assist
Paravirtualization
Compatibility
Excellent
Poor
Performance
Good
Average
VMM sophistication
High

29. Paravirtualization Challenges
XenLinux paravirtualization approach unsuitable for enterprise use
Relies on separate kernel for native and in virtual machine
Guest OS and hypervisor tightly coupled (data structure dependencies)
Tight coupling inhibits compatibility
Changes to the guest OS are invasive
VMware’s proposal: Virtual Machine Interface API (VMI)
Proof-of-concept that high-performance paravirtualization possible with a maintainable interface
VMI provides maintainability & stability
API supports low-level and higher-level interfaces
Allows same kernel to run natively and in a paravirtualized virtual machine: “transparent paravirtualization”
Allows for replacement of hypervisors without a guest recompile
Preserve key virtualization functionality: page sharing, VMotion, etc.

30. Improved Paravirtualization
Great progress is happening in the Linux kernel community
paravirt-ops infrastructure in Linux kernel
VMI-backed (to paravirt-ops) is on track for
Ubuntu has will be shipping a VMI-enabled kernel in their Feisty release.
Paravirt ops
Developers from VMware, IBM LTC, XenSource, Red Hat
Improves compatibility: transparent paravirtualization & hypervisor interoperability`
Binary Translation
Current HW Assist
Paravirtualization
Compatibility
Excellent
Good
Performance
Average
VMM sophistication
High

31. The future impact of NPT/EPT
Role of paravirtualization changes when NPT/EPT is available
NPT/EPT addresses MMU virtualization overheads
Paravirtualization will become more I/O-focused
Binary Translation
Hardware Assist
Paravirtualization
Compatibility
Excellent
Good
Performance
VMM sophistication
High
Average

32. Summary & Review CPU-optimizations Memory-optimizations
OS-optimizations

33. Summary Hardware assist technology will continue to mature
Continue to broaden the set of workloads that can be virtualized
NPT/EPT HW will bring a substantial performance boost
VMware provides flexible architecture to support emerging virtualization technologies
Multi-mode VMM utilizes binary translation, hardware assist and paravirtualization
Select best operating mode based on:
HW available in the processor
The workload being executed