1. Virtualization Trends, Challenges and Solutions
4/22/ :31 AM
Virtualization Trends, Challenges and Solutions
Naresh Sehgal, Ph.D., MBA
Lead SW Architect
Enterprise Platforms and Services Division
Intel Corp, Bangalore
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Robert X. Cringely on Computers…
"If the automobile had followed the same development cycle as the computer…
A Rolls-Royce would today cost $100,
get a million miles per gallon, and
explode once a year, killing everyone inside.” 

3. Hardware Virtual Machines (VMs)
4/22/ :31 AM
Hardware Virtual Machines (VMs)
...
App
App
App
VM0
VM1
...
...
App
App
App
App
App
App
Operating System
A new
layer of
software...
Guest OS0
...
Guest OS1
GFX
Physical Host Hardware
VM Monitor (VMM)
Processors
Memory
Graphics
Physical Host Hardware
Network
Storage
Keyboard / Mouse
Without VMs: Single OS owns all hardware resources
With VMs: Multiple OSes share hardware resources
Virtualization enables multiple operating systems to run on the same platform
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Would you believe ~45 - 50 years?
4/22/ :31 AM
How long has virtualization been around?
Recent development: ~5 years
A while: 10 years
Older than Microsoft: 30 years
A lot longer…..>40 years
Would you believe ~ years?
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Intel introduces Intel Virtualization Technology
Strachey: Time Sharing in Large Fast Computers
4/22/ :31 AM
Virtualization
Open source Xen
is released
MIT: Project MAC
Connectix is founded
Intel introduces Intel Virtualization Technology
VMWare
is founded
Goldberg: Survey of Virtual Machines Research
Microsoft acquires Connectix
1950
1960
1970
1980
1990
2000
Today
IBM: M44/44X Project
1950’s
IBM & MIT collaborate on the Compatible Time Sharing System (CTSS)
Christopher Strachey publishes a paper titled Time Sharing in Large Fast Computers in the Int’l Conference on Information Processing
1960’s
IBM works on the M44/44X the IBM Watson Research Center evaluating time sharing system concepts based on virtual machines
MIT’s Project MAC begins with a focus on the design and implementation of a better time sharing system
1970’s
Robert P Goldberg authors a paper titled Survey of Virtual Machines Research that describes the shortcomings of typical 3rd generation architectures and multi-programming operating systems
1988
Connectix is founded
1998
VMware is founded
1999
VMware delivers VMware Workstation
2001
VMware delivers VMware GSX Server & VMware ESX Server
2003
Microsoft acquires Connectix to offer virtualization solutions
VMware offers VMware VirtualCenter with VMmotion
University of Cambridge describes Xen in a paper and provides first public release
2004
Intel introduces Intel Virtualization Technology on client & server platforms
2005
IBM & MIT: Compatible Time
Sharing System
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Virtualization Challenges
4/22/ :31 AM
Virtualization Challenges
Complexity
CPU virtualization requires binary translation or paravirtualization
Must emulate I/O devices in software
Functionality
Paravirtualization may limit supported guest OSes
Guest OSes “see” only simulated platform and I/O devices
Reliability and Protection
I/O device drivers run as part of host OS or hypervisor
No protection from errant DMA that corrupts memory
Performance
Overheads of address translation in software
Extra memory required (e.g., translated code, shadow tables)
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Processor Virtualization
Guest OSes run at intended rings 3
Para- Virtualization
VMn
Modified OS
Apps
Without VT
With VT
VM0
Legacy OS
Apps
VM0
WinXP
Apps
VMn
Linux
Apps
Ring 3
Ring 1
Ring 0
Ring 3
Ring 0
VMX Root Mode
VM Entry VM
Exit
Ring Compression
H/W VM Control Structure (VMCS)
VMM
VMM
Memory and I/O Virtualization
VMCS Configuration
Binary Translation Cache
Binary Translator 2
Binary Translation
CPU0
CPUn
Processors with VT-x (or VT-i)
VT-x CPU0
Processors (Standard IA-32 or IPF) 1
Ring Deprivileging
CPUn

8. ..… Intel® Virtualization Technology (VT)
1st VT base SW Solutions OS
App OS
App OS
App OS
App
..…
Virtual Machine Monitor
Processors with
Intel® Virtualization Technology
and others …
Intel® VT
First to market with native virtualization support
Broadest HW and SW ecosystem support
CoreTM 2 Microarchitecture based systems
Significant increase in performance and improved VT performance overall segments
Mobile - Intel® Core™2 Duo Mobile Processor for Intel® Centrino® Duo Mobile Technology
Desktop - Intel® Core™2 Duo Desktop Processor E6000 sequence -
Server Dual-Core Intel® Xeon® Processor 5100 series
Get More Done On Every Server
Get More Capabilities On Client

9. Enables rapid deployment
Today’s Uses – Servers Virtualization addresses today’s IT concerns
Server Consolidation
Test and Development
HWn …
HW0
VM1
VMn OS
App … HW
VM1
VMn
VMM OS
App
VM1 OS
App OS
App
VMM HW
10:1 in many cases
Enables rapid deployment
Virtualization increases server utilization, simplifies legacy software migration

10. Emerging Server Usage Models True “Lights Out” Datacenter
Dynamic Load Balancing
Disaster Recovery
VM1a
VM2a OS
App 2
VM1b
VM2b OS
App 4
VM1 HW
VMM OS
App …
VMn OS
App
VM1 …
VMn OS
App OS
App 1 OS
App 3 OS
App
VMM
VMM
VMM HW HW HW
CPU Usage
62%
CPU Usage
90%
CPU Usage
30%
CPU Usage
63%
Upholding high-levels of business continuity
Balancing utilization with head room
Intel® Virtualization Technology will play an integral role on the next generation of VMMs

11. Intel Platform Software
Emerging Business Usage Models
Built-in Management
Proactive Security
Professional
Business
Platform
Energy Efficient Performance
Intel Platform Software

12. Supported by over 45 OEMs, ISVs, & IT Outsourcers
vProTM Key Features
Remotely Manageability - Repair down systems
Securely update systems
Audit powered-down PCs
Prevents malicious packets from entering the OS
HP OpenView
Supported by over 45 OEMs, ISVs, & IT Outsourcers
More details in the IDF vProTM tracks

13. Intel® Virtualization and Intel® vPro™ technology
Uses Intel® VT for creating a separate independent hardware-based environment inside of the PC
Service Partition –
Allowing IT administrators to create a dedicated and tamper resistant service environment or partition where tasks can run independently and isolated from the main operating system as well as from the end user
User partition -
OS and application .
VM0
VM1
User Partition
Service Partition
Stack owned and managed by IT dept… protected from users
“Firewall”
Application
“Management” Application
App0
App1
Appn
Service OS (WinCE or Linux)
User OS (Win2K, XP)
Lightweight VMM (LWVMM)
Intel® architecture Platform VT
AMT
Help desk or console access even when user partition is “down”
Intel, the Intel logo, and Intel architecture are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

14. Intel® Virtualization Technology Evolution
4/22/ :31 AM
Intel® Virtualization Technology Evolution
Standards for IO-device sharing:
Multi-context I/O devices
Endpoint device translation caching
Under definition in the PCI-SIG* IOV
Vector 3: I/O Focus
PCI-SIG
Hardware support for IO-device virtualization:
Device DMA remapping
Direct assignment of I/O devices to VMs
Device-independent control over DMA
VT-d
Vector 2: Platform Focus
Establish foundation for virtualization in the IA-32 and Itanium architectures…
VT-x
VT-i
… followed by on-going evolution of support:
Micro-architectural (e.g., lower VM switch times)
Architectural (e.g., extended page tables, EPT)
Vector 1: Processor Focus
VMM Software
Evolution
Software-only VMMs
Binary translation
Paravirtualization
Simpler and more Secure VMM through foundation of virtualizable ISAs
Increasingly better CPU and I/O virtualization Performance and Functionality as I/O devices and VMMs exploit infrastructure provided by VT-x, VT-i, VT-d
Past No Hardware Support
Today
VMM software evolution over time with hardware support
*Other names and brands may be claimed as the property of others
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Options for I/O Virtualization
4/22/ :31 AM
Hypervisor
Shared Devices
I/O Services
Device Drivers
VM0
Guest OS
and Apps
VMn
Hypervisor Model
Shared Devices
I/O Services
Hypervisor
Device Drivers
Service VMs
VMn
VM0
Guest OS
and Apps
Guest VMs
Service VM Model
Assigned Devices
Hypervisor
VM0
Guest OS
and Apps
Device Drivers
VMn
Pass-through Model
Pro: High Performance
Pro: I/O Device Sharing
Pro: VM Migration
Con: Large Hypervisor
Pro: Higher Security
Pro: I/O Device Sharing
Pro: VM Migration
Con: Lower Performance
Pro: Higher Performance
Pro: Rich Device Features
Con: Limited Sharing
Con: VM Migration Limits
VT Goal: Support all 3 Models
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. VT-d Overview VT-d provides infrastructure for I/O virtualization
4/22/ :31 AM
VT-d Overview
VT-d provides infrastructure for I/O virtualization
Defines architecture for DMA and interrupt remapping
Common architecture across IA platforms
Will be supported broadly across Intel® chipsets
CPU
DRAM
South
Bridge
System Bus
PCI Express
PCI, LPC,
Legacy devices, …
Integrated
Devices
North Bridge
VT-d
PCIe* Root Ports
*Other names and brands may be claimed as the property of others
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. How VTd works? Each VM thinks it is 0 address based
600
1000
100
200
250
350
700
Each VM thinks it is 0 address based
GPA (Guest Physical Address)
But mapped to a different address in the system memory
HPA (Host Physical Address)
VTd does the address mapping between GPA and HPA
Catches any DMA attempt to cross VM memory boundary
VM2
VM0
VM1
100
300 50 10
260

18. DMA Remapping: Hardware Overview
DMA Requests
Memory-resident Partitioning &
Translation Structures
Device
Assignment
Structures
Address Translation
Device D1
Device D2
Bus 255
Bus 0
Bus N
Dev 31, Func 7
Dev P, Func 1
Dev 0, Func 0
Dev P, Func 2
4KB
Page
Frame
4KB Page
Tables
Device ID
Virtual Address
Length …
Fault Generation
DMA Remapping
Engine
Translation Cache
Context Cache
Memory Access with Host Physical Address

19. VT-d Applied to Hypervisor Model
Shared Devices
I/O Services
Device Drivers
VM0
Guest OS
and Apps
VMn
Hypervisor Model
Improved Reliability and Protection
Hypervisor programs remap tables
Errant DMA is detected by hardware and reported to hypervisor / device driver
Bounce Buffer Support
Limited DMA addressability in I/O devices limits access to high memory
“Bounce buffer” is a software technique to copy I/O buffers into high memory
VT-d eliminates need for “bounce buffer”
Above equally useful for standard OSes
VT-d does not require a VMM to function
Pro: Higher Performance
Pro: I/O Device Sharing
Pro: VM Migration
Con: Larger Hypervisor

20. VT-d Applied to Service VM Model
Shared Devices
I/O Services
Hypervisor
Device Drivers
Service VMs
VMn
VM0
Guest OS
and Apps
Guest VMs
Service VM Model
Device Driver Deprivileging
Device drivers run above hypervisor as part of a “Service OS”
Guest device drivers program devices in DMA-virtual address space
Service VM
Forwards DMA API calls to hypervisor
Hypervisor sets up DMA-virtual to host-physical translation
Further Improvements in Protection
Guest device driver cannot compromise hypervisor code or data
Pro: High Security
Pro: I/O Device Sharing
Pro: VM Migration
Con: Lower Performance

21. VT-d Applied to Pass-through Model
Assigned Devices
Hypervisor
VM0
Guest OS
and Apps
Device Drivers
VMn
Pass-through Model
Direct Device Assignment to Guest OS
Guest OS directly programs physical device
For legacy guests, hypervisor sets up guest- to host-physical DMA mapping
For remapping aware guests, hypervisor involved in map/unmap of DMA buffers
PCI-SIG I/O Virtualization Working Group
Activity towards standardizing natively sharable I/O devices
IOV devices provide virtual interfaces, each independently assignable to VMs
Pro: Highest Performance
Pro: Smaller Hypervisor
Pro: Device-assisted sharing
Con: VM Migration Limits

22. DMA Remapping: IOTLB Scaling
Address Translation Services (ATS) extensions to PCIe* enable IOTLB scaling
ATS endpoint implements ‘Device IOTLBs’
Device-IOTLBs can be used to improve performance
E.g., Cache only static translations (e.g. command buffers)
Pre-fetch translations to reduce latency
Minimizes dependency on root-complex caching
Support device-specific demand I/O paging
*Other names and brands may be claimed as the property of others

23. Address Translation Services (ATS)
ATS Translation Flows
Device issues Translation Requests to root-complex
Root-complex provides Translation Response
Device caches translation locally in ‘Device IOTLB’
Devices can issue DMA with translated address
Translated DMA from enabled devices bypass address translation
Root Complex
Translation Request
Endpoint Device
Remap Hardware
IOTLB
Translate Address
Translation Response
Translated DMA Request
Device IOTLB
DMA using Translated Address
VT-d supports per-device control of ATS
*Other names and brands may be claimed as the property of others

24. Invalidation Architecture
Invalidation enforces consistency of caches
Required when software updates translation structures
Invalidation primitives
Global, domain-selective, and page-range invalidations
Support for Device-IOTLB invalidation (through ATS)
Invalidation software interfaces
Synchronous interface through MMIO registers
Queued interface through invalidation queue

25. ATS Invalidations ATS Invalidation Flow Invalidation details
Root-complex issues invalidation request to device
Device invalidates specified mappings from Device IOTLB
Device issues Invalidation response
Invalidation details
Invalidation request contains unique Invalidation Tag
Invalidation Responses may be coalesced
Endpoint Device
Root Complex
Remap Hardware
IOTLB
Device IOTLB
Invalidation Request
Invalidation Response
Invalidate Device-IOTLB
*Other names and brands may be claimed as the property of others

26. Mapping to VMM Software Challenges
VMn
Virtual Machines (VMs) …
Apps
Apps
Apps
Apps OS OS OS OS
Higher-level VMM Functions: Resource Discovery / Provisioning / Scheduling / User Interface
VMM (a.k.a., hypervisor)
Processor Virtualization
Memory Virtualization
I/O Device Virtualization
Ring Deprivileging
Virtual CPU Configuration
EPT Configuration
DMA and Interrupt Remapping Configuration
I/O Device Sharing
VT-x
VT-x2
VMDq
VT-d2
PCI SIG
VT-d
Binary Translation
Page-table Shadowing
I/O DMA Remapping
Interrupt Remapping
I/O Device Emulation
CPU0
CPU0
Storage
Physical Platform Resources
CPUn
CPUn
Network
Processors
Memory
I/O Devices

27. Example 6: Virtualization overhead on Intel® experimental client VMM
Example 6: Virtualization overhead on Intel® experimental client VMM* (vs. Native OS)
Relatively low Virtualization overheads for client benchmark
Targeting <10% overhead with improved SW techniques
Further VMM SW optimization and Next generation VT features to reduce virtualization overheads
* Pre beta version
Source: Intel Corporation Projections and technical specifications are based on internal analysis and
subject to change

28. Summary: A better IA platform
4/22/ :31 AM
Summary: A better IA platform
First to Market & Massive Ecosystem Support:
Choice: Broadest virtualization software support in the industry
Robust: First x86 hardware assisted virtualization technology (Intel VT)
Innovation: common specification = enhanced virtualization on x86 and will set the standard
Flexibility: Leverage Intel® Xeon® processor-based servers widely deployed infrastructure for advanced failover and dynamic load balancing
Better Platform Reliability:
Critical for more applications on the same server
More reliability features
Proven Platform Architecture - almost 40X more IA based servers than AMD based since 19961
“Choose the right basket”
Performance Headroom
Intel® Xeon® processors have key performance features for virtualization: dual-core, hyper-threading, I/O, memory, and larger caches
Slide Purpose: Intel based servers are the best solution for virtualization
Key Messages:
Strong 1st to market message with hardware assisted virtualization
Broadest level of ecosystem enablement
Leverage the install base of Intel-based servers (Advanced virtualization features, like VMWare’s VMotion* requires a common infrastructure (i.e. cannot mix Xeon and Opteron) set for underlying hardware)
When using virtualization for server consolidation, the end user is putting more reliance (more OS/applications) on a single system…therefore the single system (basket) better have the best level of reliability
1 – source: Q4’05 IDC server Tracker, total system shipped
Whitepaper on Virtualization benefits:
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Backup
Q & A

30. Example 1: SysBench Running with VMware*’s ESX Server*
Figure 1: Normalized SysBench results for the two test servers in the one, two, and four virtual machine environments. Higher numbers are better.
Significant performance lead over competition
*Source: Principled Technologies (PT) performance report
System configuration in backup foils

31. Example 2: SPECjbb Running with VMware*’s ESX Server**
Figure 2: Normalized SPECjbb2005 results for the two test servers in the one, two, and four virtual machine environments. Higher numbers are better.
performance lead over competition
* *Other names and brands may be claimed as the property of others
**Source: Principled Technologies (PT) performance report
Comparing Dual-Core AMD Opteron 285 with Dual-Core Intel® Xeon ® Processor 5160

32. Example 3: Microsoft* Virtual Server*
VMM – Microsoft* Virtual Server* 2005 R2 SP1 Java JFT workload
Guest OS - Windows 2003* Enterprise Edition R2 (32 bit)
Benchmark - JVM BEA WebLogic x`® (build R _ linux-x86_64)
Systems –
HP DL385 2 AMD Opteron 2.6GHz 2x1MB
Intel® Dual-Core Intel ® Xeon® Processor 3.0G SuperMicro SDP 16x1GB
Up to 53%
gain
Source: Intel Corporation Projections and technical specifications are based on internal analysis and subject to change
*Other names and brands may be claimed as property of others. System Configuration details in backup. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit or call (U.S.) or

33. Example 4: Energy Efficient Performance
Intel® CoreTM 2 Duo based system provide Energy Efficient Performance (EEP ) Leadership in virtualized environment
Source: Intel Corporation Projections and technical specifications are based on internal analysis and subject to change

34. Example 5: MS VS SpecJBB 2005*
Host OS
Microsoft* Server 2003 X64 Enterprise Edition SP1 RTM
Virtualization
Microsoft* Virtual Server* R2 Beta SP1 ver EE
Drop B1036 vmm.sys
Microsoft* Virtual Machine Windows* Guest Editions ver
Guest OS
RedHat V kernel (32-bit)
Workload
SpecJBB 2005
1.18X
1.66X
1.95X
Intel® XEON® SW Virtualized guest performance is 1.66x of Opteron
Intel® XEON® VT performance is 1.18x of Software (no VT) Intel® XEON®
Intel® XEON® VT performance is 1.95x of Opteron SW (no Pacifica)
Source: Intel Corporation Projections and technical specifications are based on internal analysis and subject to change