1. SQL Server Virtualization
Jonathan Kehayias
MCITP Database Administrator
SQL Server MVP

2. Agenda SQL Initiative Overview Typical SQL Server Workloads
Performance Metrics
I/O Metrics
Processor
Performance Counters
High Availability Configuration
Manageability Gains
Alternatives to Virtualization

3. Who am I? SQL Database Administrator OSI Restaurant Partners
5/05/2018 9:54 PM
Who am I?
SQL Database Administrator OSI Restaurant Partners
SQL Server MVP
MSDN Forums Moderator
Founder SQLCLR.net
Member Tampa SQL Users Group
PASS Member
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. 3

4. SQL Initiative Primary Objective Success Criteria Disaster Recovery
Consolidation
High Availability
Success Criteria
Immediate failover to remote data center with minimal data loss
Performance
Reduced TCO and higher ROI’s
Reduced work loss from Server Failures
Our primary reason for virtualizing was to create a total DR solution. This also allowed us to consolidate multiple physical servers onto a single host as Virtual Machines. We also gain High Availability since in the event of a hardware failure on a host, we can start the Virtual Servers on a different host.
To measure success, we had to be fully DR capable, and the environment had to provide equivalent performance to a physical implementation

5. Typical SQL Server Workloads
Small Servers 2 CPU 4GB RAM
Easily Virtualized in most Virtualization Technologies
Medium Servers 4CPU 8-16GB RAM
Requires more planning, but can be virtualized with most Virtualization Technologies
Enterprise Servers 8+CPU 16+GB RAM
Should not be virtualized, look at other alternatives
When it comes to SQL Server, I generally classify the servers I see and find into 3 groups.
Small Servers have 1-2 processors and up to 4GB of RAM. These servers are the perfect candidates for virtualization, because their size is easily managed in all of the Virtual Server Host Environments.
Medium Servers have between 2-4 processors and 8-16GB. These servers are potential candidates for virtualization. However, they require planning due to their size. If you allocate 8GB+ RAM to a virtual machine, your options for hosts can become limited in the event that you have to move the virtual machine off its primary host.
Enterprise Servers have more than 4 processors and more than 16GB of RAM. These servers are not candidates for virtualization, and if DR is required, the other technologies available should be considered.

6. Host Server Requirements
Multiple Processors/ Multiple Cores
Large allocations of RAM
Redundant NICS
Redundant HBA’s
Redundant Power
When planning to virtualize, the host servers physical hardware is a key to the overall performance you will experience on your virtual machines. The host should have multiple processor sockets and ideally each socket will support multi-core CPU’s. The host should have at least as much physical RAM as the virtual machine allocations that will be running on it, with roughly 4GB reserved for the virtual host OS.
Redundancy is extremely important for the host, and it should have redundant Networking, Fiber channels, and Power. This removes the single point of failure from affecting all of the virtual machines on the host, and allows for load balancing across the various interfaces.

7. Performance VMware ESX Server, 3.5
This is an example host from our environment

8. Performance SQL Virtual Machine Configuration
This is an example SQL Server that has been virtualized in our environment. Note the separation of disk arrays, and the SAN RAID configuration. This is not an ideal configuration for SQL Server disk wise, but as you will see it does work very well.

9. Performance SQLIO Benchmarks
SQLIO is a tool provided by Microsoft which can also be used to determine the I/O capacity of a given configuration.
SQL Server stores data 8K pages allocated in blocks of 8 as 64K extents. Typical SQL I/O operations involve Random Reads of extents from disk.
SQLIO benchmarks on this SQL Server for 64K Random Read I/O were equivalent or better than common physical hardware.
This slide should be self explanatory. You can download SQLIO from Microsoft for free:
Download details: SQLIO Disk Subsystem Benchmark Tool

10. Performance (Vitrualized) Random Read I/O By The Numbers
Drive Format Test IOs/sec MBs/sec
READ
Default Read 8KB random Default Read 64KB random Default Read 128KB random Default Read 256KB random Default Read 8KB sequential Default Read 64KB sequential Default Read 128KB sequential Default Read 256KB sequential WRITE Default Write 8KB Random Default Write 64KB Random Default Write 128KB Random Default Write 256KB Random Default Write 8KB Sequential Default Write 64KB Sequential Default Write 128KB sequential Default Write 256KB sequential
This is an Averaged output for SQLIO for one of our primary production SQL Servers which was shown spec wise earlier.

11. Dell PowerEdge 6650 2x 2.7 GHz 2GB RAM 4x 36GB RAID 10 array
Drive Format Test IOs/sec MBs/sec
READ
Default Read 8KB random Default Read 64KB random Default Read 128KB random Default Read 256KB random Default Read 8KB sequential Default Read 64KB sequential Default Read 128KB sequential Default Read 256KB sequential WRITE Default Write 8KB Random Default Write 64KB Random Default Write 128KB Random Default Write 256KB Random Default Write 8KB Sequential Default Write 64KB Sequential Default Write 128KB sequential Default Write 256KB sequential
Physical SQLIO outputs were obtained from the SQL Server Performance website Forums:
These numbers are for comparison and were gathered off the link above. Keep in mind that I don’t know the total configuration of these disk arrays and some of these numbers are very small for what the listed configuration is. This could be indicative of other problems for these specific servers, but I can’t speak to that.

12. Dual 3.0Ghz Xeon Dual 2GB FC HBA IBM DS4300 SAN RAID 10 (10 x 74GB 15K)
Drive Format Test IOs/sec MBs/sec SQLIO 8k sector
Read - Random Read - Sequential Write - Random Write - Sequential SQLIO 32k sector
Read - Random Read - Sequential Write - Random Write - Sequential SQLIO 64k sector Read - Random Read - Sequential Write - Random Write - Sequential
Physical SQLIO outputs were obtained from the SQL Server Performance website Forums:

13. DELL 6850 Quad Xeon 3.4ghz w/4GB RAM EMC Clariion CX-700 RAID 10
Drive Format Test IOs/sec MBs/sec
8k random write: k random write: k random write: k random write: k seq. write: k seq. write: k seq. write: k seq. write: k random read: k random read: k random read: k random read: k seq. read: k seq. read: k seq. read: k seq. read:
Physical SQLIO outputs were obtained from the SQL Server Performance website Forums:

14. Performance Perf Counter Monitoring
To understand how well a SQL Server is performing the SQL Server as well as Windows Subsystem Performance Counters need to be Monitored.
Key Counters to monitor include:
Processor/%Processor Time should remain below 80%.
Processor/%Privileged Time should remain below 20%
SQL Server General/User Connections
Batches/Sec
Page Life Expectancy
Pages/Sec
Memory Grants Pending
Lazy Writes/sec
For further information, take a look at the following Screencast series by Kevin Kline (SQL Server MVP and Professional Association for SQL Server (PASS) President):
Performance monitoring should be a key to any Database environment. All numbers on the next series of slides are based on Kevin Klines Screencast above.

15. Performance Perf Counters (%Processor Time)
< 80% average
This is a 21 day graph of the listed counter. This server performs within standards for a SQL Server, even though virtualized.

16. Performance Perf Counters (Processor\%Privileged Time)
< 20% Average
This is a 21 day graph of the listed counter. This server performs within standards for a SQL Server, even though virtualized.

17. Performance Perf Counters (User Connections)
This is just a reference counter to be used in tandem with other counters to view system activity.
This is a 21 day graph of the listed counter. This server performs within standards for a SQL Server, even though virtualized.

18. Performance Perf Counters (Batches/sec)
This is a reference to the amount of activity the Server is performing. It is used along with other counters like Page Splits/sec to determine if there are problems.
This is a 21 day graph of the listed counter. This server performs within standards for a SQL Server, even though virtualized.

19. Performance Perf Counters (Buffer Cache)
Should remain as close to 100% as possible. Consistent drops below 95-90% signals Memory Pressure
This is a 21 day graph of the listed counter. This server performs within standards for a SQL Server, even though virtualized.

20. Performance Perf Counters (Memory pages/sec)
The rate at which pages are read from or written to disk. If > 100 on a slow disk subsystem or > 600 on a fast disk subsystem it should be investigated.
This is a 21 day graph of the listed counter. This server performs within standards for a SQL Server, even though virtualized.

21. Performance Conclusions
1: Compared to various disk configurations of physical implementations with local storage, we experience 10x performance for disk subsystem I/O
2: Critical performance counters SQL maintains industry acceptable performance
3: Ability to consolidate multiple VMs along with SQL server, ~5 to 7:1 to save costs on physical hardware, rack space, power, cooling and integrate into DR plan.

22. Consolidation Architecture
Single SQL server per ESX host
Multiple SQL Servers never reside on the same ESX host
Logical placement of VM’s to eliminate contention of resources
Web and App server never communicate with the SQL server on the same host
In planning/configuring our environment, no 2 SQL servers are ever on a single host. They also don’t share hosts with other IO intensive servers like Exchange. In addition, the associated application servers are generally located on a different host. The reason behind this is that when an app server comes under use, it requires resources from the host. If it makes a request from SQL, then SQL also requires resources from the host at the exact same time. By placing them on different hosts, no single host has to serve up resources for the request.

23. High Availability
SAN Storage Mirrored for Disaster Recovery to Chicago Datacenter.
Quarterly Failover Tests of key SQL Servers on Chicago Network with zero data loss at failover.
In the event of a host server hardware failure, the virtual machines can be in some cases live migrated to another host. If the host physically crashes, the virtual machines can be started on a different host with a few mouse clicks. This is only available for SAN based virtualization where the VHD/VMDK sits on a SAN LUN and can be assigned to any host for startup.

24. High Availability Live Migration / VMotion
Eliminates need for Mirroring Solutions for Hardware redundancy. In the event of a host hardware failure the Virtual Machines can be hot migrated to another host
Allows live migration of Servers during high load operation to shift load to more powerful hosts as needed.
During High load operations, servers can be moved between hosts with minimal downtime, 1-5 seconds on VMware, and as of the last test, approximately 30 seconds on HyperV. This allows the server managers to manage the impact of heavy load on the environment.

25. Manageability Snapshot Technology / Undo Disks Hot Add Disk Arrays
Provides a point in time recovery point for risky operations such as upgrading Server OS and or SQL Server.
Hot Add Disk Arrays
Allows Zero Down Time additions of new Storage LUNS as database grow in size
HBA Load Balancing
Allows Disk I/O Load balancing across redundant paths to the SAN storage.
One of the greatest benefits of virtualized servers is snapshot technology that allows for immediate undo operations of changes to the server. This must be enabled prior to the change occurring, and does require disk space so you don’t want to leave this enabled on a production server all the time. This feature allowed a disastrous upgrade to SQL 2005 to be undone in a matter of seconds. It also allowed for instant removal of CU6 for SQL 2005 SP2 after it corrupted the master database on one of our servers.
Adding disk space to a virtual server is also a major benefit. All you need to do is present the new LUN from the SAN to the host, and it can be added to the server while it is running as long as the servers OS supports this.
With the right host configuration, the server managers can balance the IO load across multiple HBA’s to load balance requests to and from the SAN.

26. Manageability Rapid Scale Up
Adding additional vCPU’s and Memory is only a reboot away, provided the host has available resources.
Easily upsize a server for end of month processing when it requires the most power while minimizing its footprint while under minimal load.
Adding power to any virtual server is only a reboot away. You can easily scale a 2 processor 4GB Ram virtual machine up to 4 processors and 8GB of RAM at the end of the month where heavy processing requires more power, to allow closeouts to occur twice as fast or with less impact on the environment.
Keep in mind that you should be maintaining the appropriate licensing for a 4 processors server should you do this to be complaint in licensing your server.

27. Manageability
VMWare Infrastructure Client / System Center Virtual Machine Manager
Immediate shared console level access to the SQL Server provides remote administration and rapid response for critical server outages.
No risk of a BSOD requiring physical access to the server or of a stuck ILO interface on a server.
Integrated performance monitoring of processor, memory, disk and networking counters.
The Infrastructure Client/Virtual Machine Manager give you console level access to your virtual server as if you were sitting in front of it from anywhere that you can access your network. If a virtual server happens to experience a Blue Screen of Death, you no longer have to drive to the datacenter to realize it. You can pop onto the console and restart the server instantly from anywhere. What’s better is that you can have multiple people looking at the console from multiple locations. This has been key when a server has had problems after hours. The server team can be looking at the same screen I am while we try to troubleshoot the issue from home on a phone call.
These tools also provide virtual machine performance monitoring and alerting in the event that the virtual machine begins to exceed established tolerances.

28. Manageability Manageability (cont’d)
Running history of Events occurring through the client including Server Resets, Migrations, and Reconfigurations.
Integrated Alarms are configurable for out of tolerance counter statuses.
The management tools also maintain a history of what has occurred through the tool, and who did it. This helps determine what happened in the event a server goes offline after a live migration, or is powered off by mistake.

29. Alternatives to Virtualization
Boot From SAN
Allows for disaster recovery replication.
Requires identical hardware for easy failover boot for hardware failures.
Ideal for environments with standardized hardware configurations.
Centralizes storage for ease of maintenance.
Allows rapid redeployment of SAN image for testing purposes.
Alternatives to Virtualization include Booting from the SAN. This still allows for SAN based replication for DR planning, but it requires a 1 for 1 server allocation in your datacenters. For every server that you plan to bring up in a DR scenario, a second server must exist in your DR site.

30. Alternatives (cont’d)
HP PolyServe
Microsoft Supported Always On technology for SQL Server.
Pools resources to allow Live Migration between physical hardware.
Allows for higher utilization of physical platforms.
HP PolyServe is another alternative that is 100% Microsoft supported for SQL Server Always On Technologies. This allows you to virtualize your SQL Instance and separates your servers into a pool, and your disk storage into a pool that is shared by the server pool separating the resources from each other.

31. Alternative (cont’d) HP PolyServe (cont’d)
Server Pool isolation from Disk Array pool allows for multiple SAN arrays to be managed as a single shared resource in the Server Pool.
Run SQL 2005 and SQL 2000 in the same cluster without compatibility issues.
Volume Manager allows extension beyond the existing 2TB limit of a single LUN through striping.
You can pool multiple SAN Arrays to allow you to spread large databases across multiple SAN arrays, and allows you to expand past the 2TB limit for a Single LUN through striping.
You can also host SQL 2000 and SQL 2005 in the same cluster without compatibility issues.

32. Alternatives (cont’d)
HP PolyServe (Consolidation)
Typical Server Sprawl
This is a typical SQL Server environment. Note that the clustered instances have a complete server dedicated to nothing but waiting for a disaster. The blue area on each server represents the amount of utilization for that particular SQL Server.

33. Alternatives (cont’d)
HP PolyServe (Consolidation cont’d)
Through PolyServe, this same environment can be reduced by 9 SQL Servers, saving licensing costs, power costs, cooling costs, and rack space while providing a highly available redundant implementation for your environment.

34. Lessons Learned
1: Don’t lock pages in memory if you plan to do VMotion /Live Migration.
2: Be cautious when implementing Microsoft Recommended Best Practices that affect system configurations and test in Development before deploying in Production. Some may not be compatible with VMotion / Live Migration.
3: Don’t accept a vendor’s statement that Virtualization is the problem – look deeper, and you can generally disprove this statement.
Not all best practice recommendations work with SQL Server when it is virtualized. Locking pages in memory specifically can lead to Server crashes if you choose to use VMotion or Live Migration. Considering what the purpose of this setting is, this makes sense. If you migrate from one host to another, the memory addressing is going to change as it shifts the load from one host to the other.
Often times Virtualization gets blamed by vendors for performance issues incorrectly. It would seem logical that for all the money you pay the vendor, that they should have to provide evidence to the fact the Virtualization is the issue, but this rarely is the case. Instead is left to you to prove that it isn’t the case. In worst case scenarios this can lead to having to reproduce the problem on physical hardware before a vendor continues to provide support. We have done this twice at OSI and both times, the problems existed on the physical hardware just as they did in VMware. If you dig deeper, you can often find the problem that proves it isn’t virtualization.

35. Conclusions Performance Metrics High Availability Configuration
I/O Metrics – 10x performance, Processor and Performance Counters
High Availability Configuration
VMotion / Live Migration to load balance during end of month reconciliations keeps ahead of the business
Reduced downtime with hot add disk
Manageability Gains
SOX compliance for Disaster Recovery
Virtual Center Console for team collaboration
Cost Avoidance
Reduces need for Mirroring hardware and software
Consolidation of hardware both primary and DR

36. Resources Boot from SAN with Windows 2003 Microsoft Virtualization
VMWare Virtualization
Microsoft Assessment and Planning Tool for Virtualization
Using SQL Server 2005 in a Virtual Environment
Virtualizing SQL Server with VMware or Hyper-V
The first link is a whitepaper from Microsoft explaining the technology and how to configure it.
The second is a link to Hyper-V.
The third is a link to VMware
The fourth is a new tool from Microsoft that assesses a server to determine how good of a candidate it is for virtualization.
The fifth is a whitepaper by Microsoft on Licensing requirements for SQL Virtualized.
The last is a new article out from SQL-Mag on comparing VMware and Hyper-V for virtualized SQL Server.
One last thing is that there is a free Webcast on Virtualizing SQL with Kevin Kline. You can sign up for it at: