Improving Disk Latency and Throughput with VMware Presented by Raxco Software, Inc. March 11, 2011

Today’s Agenda Provide technical information on how NTFS impacts VMware I/O performance Examine ESX I/O test results Economic impact of Windows guests Solutions

Virtualization Benefits Server consolidation Less physical space for data centers Lower energy costs Easier management Eco-friendly alternative These benefits can only be realized if everything works as it should. The one thing the virtualization layer never touches is the guest system. If the guest systems overload the hypervisor, I/O contention occurs and performance suffers.

Identifying and Correcting Problems Latency is your best indicator of a performance problem Device latency is vSphere’s report of the physical storage response time Kernel latency is vSphere’s report of ESC’s ability to manage IO Experts disagree on specifics, but most agree that… Device latency in excess of 15ms is worth inspection Device latency in excess of 30ms is likely a problem Kernel latency in excess of 2ms means ESX queues are overflowing High device latency can result in ESX queuing So, correct slow hardware first! Then, consider reducing VMDKs on a VMFS volume Only then consider changing queue depths In a presentation at EMC World in 2010, VMware’s Scott Drummonds, discussed the importance of disk latency. When latency is > 15 ms you should be concerned, when it exceeds 30ms, you have a real problem. File and free space fragmentation increases disk latency because they increase the time it takes to read/write a file. © Copyright 2010 EMC Corporation. All rights reserved.

Storage Contention Solution: Storage IO Control SIOC calculates data store latency to identify storage contention Latency is a normalized, average across virtual machines IO size and IOPS included SIOC enforces fairness when data store latency crosses threshold Default of 30ms Fairness enforced by limiting VMs access to queue slots Net effect: trade throughput for latency With Storage IO Control Actual Disk Resources utilized by each VM are in the correct ratio even across ESX Hosts VMware’s Storage I/O Control monitors latency and trades off throughput to improve latency. While this treats the symptom (slow latency), the solution (reduced throughput) is still an issue. The real question is why are the disks slow in the first place? © Copyright 2010 EMC Corporation. All rights reserved.

NTFS I/O Storms All of your work is done in the guest systems. With Windows guests, the NTFS file system can work against you by creating I/O storms resulting from file and free space fragmentation. These I/O storms lead poor disk latency, less throughput, and poor performance.

NTFS Behavior NTFS fragments files and free space Increases logical I/O to storage controller More logical I/O = More physical I/O Multiple instances of Windows on host can lead to I/O contention The Windows NTFS file system is notorious for fragmenting files and free space. This fragmentation is related to how NTFS sees the disk. Each fragment results in a scsi command going to the disk controller. A file in one piece issues one logical I/O; a file in 100 pieces issues 100 logical I/O. Each logical I/O maps to the disk in one or more physical I/O. In a virtual system each logical I/O is overhead for the hypervisor.

What is Fragmentation? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Fragmentation happens in the NTFS file system and on the physical disk, but the two are different. NTFS fragmentation is a result of how the file system allocates space logically, it’s just how the file system works. NTFS fragments files before anything is written to the disk. Once the data is passed to the disk controller the firmware in the controller maps the logical data to physical blocks on the disk. If a file is in a single contiguous string of logical clusters, and it is mapped to a stripe set across three platters, you could have the best possible solution. The file system sends a single logical I/O and the controller mapped it for efficient retrieval. This breaks down when the file system sends the controller 100 logical I/O for a s fragmented file requiring 350 physical I/O to the disk to accommodate the file. This is how logical and physical fragmentation combine to hammer performance. ?

Logical v Physical Logical Level NTFS needs disk and cluster size, enumerates LCNs Creates $MFT and $Bitmap metadata $Bitmap is how NTFS “sees” the disk Has no idea about physical/virtual disk types When a disk is formatted NTFS needs to know its size and the cluster size. It does not know what kind of disk it is or if it is physical of virtual. When a file is created a record for the file is created in the $MFT, the volume index. NTFS then asks for space from the $Bitmap file. $Bitmap offers space in one chunk if it can locate it (a contiguous file). If contiguous space is not available $Bitmap offers space wherever it can find it (a fragmented file).

Anatomy of an MFT Record (vcn, lcn, run length): (8a85, 9189a, 7) This is a look inside a MFT record. The Extent List is where information about the logical location of a file is kept. The VCN is the virtual cluster number and indicates which part of the file this piece is 1st, 2nd, 3rd… The LCN is the starting logical cluster number for the fragment and it is followed by the run length which is the number of clusters from the LCN. The MFT record is 1KB in size. Severely fragmented files will have multiple linked MFT records which are needed to accommodate lengthy Extent Lists. This also has an adverse effect on performance.

File Allocation Create $MFT record (one or more) $Bitmap accessed to locate free space $MFT record is updated with content NTFS creates a file record in the $MFT for a new file. It then goes to $Bitmap to find space. The logical clusters allocated are recorded in the MFT Extent List. Bitmap Access MFT Update Create

File Access Load portion of MFT with correct record via directory Locate file in the MFT Pass starting LCN’s and run lengths to disk controller Number of logical fragments influences number of physical seeks A file that is badly fragmented at the file system level often results in excess physical I/O to the disk. Locate File # Physical Seeks Load # LCN’s

Logical v. Physical Physical Level Disk controller Maps LCNs to PCNs Writes data to disk The logical I/O is sent to the disk controller and mapped to physical blocks on the disk.

Wasted Seeks 58.30% 12.72% Fragmented After PerfectDisk Partition State Total Number of I/O Requests Sent to the File System Total Number of Resulting Disk Accesses/Seeks Net Wasted Seeks When Running SYSmark Percent Net Wasted Seeks When Running SYSmark Fragmented 1,320,686 2,090,649 769,963 58.30% After PerfectDisk 1,434,454 1,616,847 182,393 12.72% After Built-In 1,411,613 1,931,395 519,782 36.82% David Goebel, a Microsoft engineer and one of the original 4-man team that wrote the NTFS, did an analysis to measure the importance of consolidated files and free space. Goebel ran a benchmark against two identical disks, one with the free space fragmented and the other with the free space consolidated. He wrote software that counted every request from the file system and then counted how many physical I/O were required to write the data. Ideally, each file system request should result in a single disk I/O. The results show that the fragmented disk performed 58% more seeks than there were file system requests. When the disk was defragmented and the free space consolidated only 13% of file system requests required additional disk I/O. The extra logical I/O create extra physical I/O with its attendant impact on performance. In this benchmark the I/O count difference between the fragmented disk and the PerfectDisk disk was 587,570. At 15ms per I/O this is an additional 2.4 hours of I/O processing time.

How This Affects A Virtual Environment P2V Conversion Extra Hypervisor Overhead Disk Latency Degradation Overall Performance System Throughput Wasted Space Costly File and free space fragmentation can bloat the size of going P2V. Each extra command (one per fragment) across the virtual storage stack increases hypervisor overhead Excess disk I/O increases disk latency time with an impact on overall performance Disk latency affects system throughput Fragmentation wastes valuable disk space Problems associated with fragmentation cost more in troubleshooting, downtime and potentially unnecessary hardware upgrades.

P2V Conversion Physical Drive No Optimization Optimization 24GB 22GB 2GB Smaller If you go P2V on a fragmented volume the virtual disk is almost the same as the physical disk. When you go P2V with defragmentation, free space consolidation and then zero-fill the free space you get a much smaller image. This saves P2V storage costs.

ESX Cluster Testing Identical disks - 40% free space Optimized one set, the other “as is” Installed MS Office and MS SQL Captured metrics with VMware’s vscsiStats utility Using VMware’s vscsiStats utility we measured the impact of defragmentation on two identical sets of disks. The first set was fragmented the second was optimized with Raxco’s PerfectDisk. The test involved installing MS Office and MS SQL to both sets of disk and collecting the I/O data on both.

Total I/O Count 37191 29238 21.3 Total IO Count Fragmented PerfectDisk % Improvement Total IO Count 37191 29238 21.3 Read IO Count 3066 2799 8.7 Write IO Count 34125 26439 22.5 The PerfectDisk disk reduced the total I/O crossing the stack by 21.3%. VMware has a mantra that says “More IOPS are bad, less IOPS are good.” Optimizing the Windows guest reduced the total I/O by 7953 .

49% Reduction in Latency! 40,442 20,764 30ms 50ms 100ms >100ms Total Fragmented I/O 12749 9877 8700 9116 40,442 PerfectDisk I/O 6707 4923 4081 5053 20,764 Disk latency is the time it takes an I/O to complete. As noted in slide 5, VMware and EMC state that disk latency in excess of 30ms is a problem. In our test the PerfectDisk disk reduced the number of I/O taking more than 30ms by 49%. VMware’s Storage I/O Control monitors latency and limits access to HBA-LUN queues when latency exceeds 30ms. In effect, trading throughput for latency. By contrast, optimizing the Windows guest delivered a huge reduction in latency with no adverse effect on throughput.

Disk Latency Actual vscsiStat histogram. Note that the red lines (PerfectDisk) are about 50% of the blue for all I/O greater than 1ms. This means PerfectDisk reduced the total number of I/O in each time interval by half.

12X More Large I/O 247 2959 Total IO > 524K Fragmented Disk PerfectDisk Disk Total IO Equal to 524K 2512 848 Total IO > 524K 247 2959 Read IO Equal to 524K 33 7 Read IO >524K 125 65 Write IO Equal to 524K 2480 841 Write IO >524K 122 2894 One of the benefits of disk optimization is you can do larger I/O and thereby improve throughput. Our testing showed that PerfectDisk performed 12 times as many of the largest I/O as measured by vscsiStats.

12 times more of the largest IO Large I/O Actual vscsiStats histogram. Note right-most red line for I/O >524K. 12 times more of the largest IO

Improved Sequential I/O Fragmented PerfectDisk Improvement Percent Sequential 17% 27% 58% Total IO 127703 90526 25% Sequential IO 22126 24340 33% Not surprisingly, disk optimization improves sequential data access since the files are contiguous. Testing showed PerfectDisk improved sequential I/O by 58%.

Improved Sequential I/O Center column shows that PerfectDisk exceed the fragmented disk in sequential I/O. Conversely, the left and right-most columns show the fragmented disk outdid PerfectDisk in the worst category where the I/O were the farthest apart (500,000 LBNs).

Installation Time Comparison Fragmented PerfectDisk % Improvement MS Office Install 20 min 15 min 25 MS SQL Install 76 min 51 min 33 The software installation times in the benchmark improved dramatically with the PerfectDisk disk.

The Cost of Fragmentation EXAMPLE: 20 files x 6 seconds = 2 minutes 300 users x 2 min = 10 hours/day 10 hrs x $25/hr = $250/day Annual cost = $62,500 The primary cost of the fragmentation is the cost of waiting. Whether it is on workstations or servers, as files and free space fragments the time is takes to read/write files increases. The figures above were taken from two separate customer ROI analyses. Neither company had ever defragmented their machines. An analysis of several disks showed files and free space were fragmented and performance was degrading. An assumption was made that each user had al least 20 frequently used files that took at least 6 seconds too long to open. This was a loss of 2 minutes per day per user. There were 300 users so lost time equaled 10 hours per day with a loaded cost of $25/hour. The annual cost of fragmentation was $62,500. Management also recognized that if they did nothing the read/write times would continue to increase, performance-related problems would increase and they might buy hardware they didn’t really need. The cost of PerfectDisk for the workstations and servers here was about $10,000. The ROI based on lost productivity alone was 40 workdays.

Virtual Guest Fragmentation Windows guests have all the same NTFS behavior Fragmentation produces more IOPS Fragmentation reduces ESX throughput Fragmentation increases ESX disk latency Fragmentation creates resource contention between host & guests Everything described thus far about the NTFS file system applies to virtual Windows servers. NTFS behavior will increase IOPS, increase latency, degrade throughput and potentially create I/O contention issues.

Solutions Expensive Inexpensive More disks and faster disks Upgrade Fibre Channel Troubleshooting Inexpensive Optimize the Windows guest systems The expensive options are more and better hardware. This means fragmentation, a natural byproduct of using Windows, can happen faster. The inexpensive solution is to optimize the Windows guests with PerfectDisk.

PerfectDisk 12 vSphere Virtualization Awareness/host & client NEW Virtualization Awareness/host & client OptiWrite Fragmentation Avoidance “Zero-fill” free space NEW NEW PerfectDisk offers: Optimization awareness between host and guests to avoid contention issues OptiWrite fragmentation avoidance that prevents 95% of the fragmentation that would occur A “zero-fill” free space option that aids in P2V conversion and offers a better VM compaction option. If you have a zero-detect SAN you can recover white space after the zero-fill completes

PerfectDisk 12 vSphere “Short stroking” for thin provisioned disks Schedule guest compaction Snapshot & Linked Clone recognition NEW NEW NEW Short-stroking makes it possible to defragment a thin-provisioned volume without blowing out all the storage. New algorithms minimize file movement so defragmentation and zero-fill can take place Snapshot and linked clone recognition. These are disks you generally don’t want to defrag so PerfectDisk flags them for you. PerfectDisk also differentiates between dependent/independent disks and persistent/non-persistent disks for use in virtual desktop environment.

PerfectDisk Benefits on ESX Saves $$$ in productivity and admin Reduces resource contention for VM’s Reduces total IO workload Improves throughput Reduces disk latency Delivers optimal performance Optimizing Windows guest systems eliminates a well –documented problem with Windows. Keeping guest systems in good shape reduces the potential for I/O issues related to latency and throughput. It is always better to be proactive than reactive.

Contact Raxco Free Evaluation Software Excellent Support to Get You Started White Papers Great ROI www.raxco.com Toll Free: 1.800.546.9728