© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. IO, IO! Why’s my SQL slow? Brian Flynn

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Define Speed? »Acceleration »Burst/Top Speed »Sustained Speed »Constraints »Total Trip Time »Average Speed 2

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. »Imagine transporting things a long distance. »Just you and an over night bag vs everything you own… 3 Latency, Speed, & Throughput High Acceleration (Low Latency)Medium Acceleration (Medium Latency)Low Acceleration (High Latency) High Top SpeedMedium Top SpeedLow Top Speed Small CargoMedium CargoLarge Cargo Low CapacityMedium CapacityHigh Capacity

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Latency, Speed, & Throughput »Electricity:  Volts / Ohms = Amps  Volts * Amps = Watts »Data:  Buffer Size / Latency = Transfer Rate  Buffer Size * Transfer Rate = Throughput 4 AnalogySizeCurrentResistanceCapacity ElectricityVoltsAmpsOhmsWatts WaterPressureFlow Rate Inverse of channel diameter Volume/Time DataBuffer Size * compression * jumbo frames Transfer Rate *IOPS Latency * Function of Distance * Function of storage type Throughput

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Latency, Speed, & Throughput »When speed is constant, packet size governs throughput. »Larger packets increase throughput. »Greater distance increases latency which lowers perceived throughput. 5

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. True speed is all about latency »1 byte or 1000 bytes, same trip time. »Overhead accounts for more constraint than sum of the serialized trips. 6

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Aren’t we talking about storage? »HDD latency  Rotating the platter  Moving the head »SSD latency  One controller is a gateway for multiple chips »RAID latency  Mirror – Improves read IOPS if split reads is supported.  Stripe – Improves read & write IOPS.  Distributed Parity – Write IOPS penalty for calculating parity bits.  Raid 10 – Mirror & Stripe. No write penalty. Twice the price of stripe only. »SAN latency  One controller is the gateway to many disks.  Limited bandwidth to arrays »NAS latency  Network is usually the weakest link 7

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Enough already, just make it faster! »There’s more than straight line acceleration. »Complex systems force us to tease out the latency bottlenecks »Balancing 3 Options 1.Get the same work done with fewer operations 2.Do more in parallel 3.Make individual operations faster 8

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Get the same work done with fewer operations »In our transportation example, same work with fewer trips … The big semi does this… larger cargo or payload. »With data, however…  Eliminate redundant or superfluous database work Create indexes that reduce table scans Keep tables defragmented to increase read-ahead Keep frequently used data in application memory  Caching techniques to reduce re-reading from storage SAN cache Server memory SQL 2014 Buffer Pool Extension  Compress data More pages in memory increases page life expectancy Compressed data will fit into fewer Physical I/Os ** CPU latency must be lower than storage latency 9

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Do more in parallel »Adding disks to an array increases maximum IOPS  Helps where I/O can be parallelized Multiple CPU issuing synchronous I/Os SQL Server Pre-fetch & Read-ahead Reference: MSDN: Sequential Read AheadMSDN: Sequential Read Ahead »Defragmentation increases Read-ahead, which is asynchronous »Evaluate for parallel capabilities end to end  Multiple file groups on isolated storage  Multiple, isolated LUNs  Multiple OS disk queues  Consider SAN paths & controllers »Bottom line, you can’t parallelize everything  Business Analogy: The Mythical Man Month  Humorous Analogy: The Pregnancy Problem 10

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Make individual operations faster »Almost always, a winner! But often, the most expensive solution. »Lower latency storage  HDD Moving parts  Mechanical latency  Random reads  More mechanical latency  Thus to optimize, maximize sequential reads before random reads  Of course, with RAID arrays, almost every read is random  SSDs have no moving parts  No mechanical latency  Thus all reads are equal. No benefit to sequential vs random »Who’s IO are we talking about?  Typical disk I/Os are usually 4K  SQL Server I/Os range from 8K to 512K »REF: 11

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. How Do I Choose? It depends… :-\ »It depends… At best, all three. Next best, any combination.  3 rd party software may not permit your query tuning  Budget constraints may not permit low latency storage  The size of I/Os may not lend to parallelizing workloads »General rules:  Reducing total # I/Os is always a winner.  For large result sets or concurrency  Increasing IOPS can help a lot  For small result sets  Raw speed (Latency) may be your only hope. 12

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Disk Queue Depth »I hate queues, especially at the DMV! Waiting in line stinks! »Disk Queue Length is a measure of I/O constraint. »How high is OK? It depends on your environment.  Different performance expectation between OLTP and OLAP systems »If it’s consistently high, adding disks to the array may help. »You should be watching this one and DPA does that… 13

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Other WMI Counters »Latency-Like  Avg. Disk Sec/Read is the average time, in seconds, of a read of data from the disk. Any number:  Avg. Disk Sec/Write is the average time, in seconds, of a write of data to the disk. Please refer to the guideline in the previous bullet. »Current-Like  Avg. Disk Reads/Sec is the rate of read operations on the disk.  Avg. Disk Writes/Sec is the rate of write operations on the disk. »Physical Disk: %Disk Time is the percentage of elapsed time that the selected disk drive was busy servicing read or write requests. REF: 14

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Virtual File Stats DMV 15

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Crystal Disk Mark »A great, free tool that tells you the throughput of your storage under a few mixtures of I/O size, queue depth and sequential or random I/O. You may not get your storage admin’s blessing to run this.  But it’s a decent way to measure throughput. 16

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. SQLIO 17 REF:

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 18 Query Plans & Process Monitor »Query plans tell you when you’re scanning and seeking »ProcMon tells you how many and how big are the storage I/Os

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Server memory, SAN Cache, Tierd Storage etc »You’ll have to clear a few obstacles. »SQL Server caches pages in RAM so when testing you’ll need dump the buffer cache to ensure you are going to disk. »And unless your storage admin is a gift from the DBA gods, you’ll have to get real creative to control for SAN cache in any experiments. »Tiered storage can cause unpredictability too. 19

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Free Tools for SAN & VM environments

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Design Choices : Fragmentation Impacts »Design for Read Ahead  Understand & avoid physical and logical fragmentation  Clustered indexes that monotonously increment on isolated storage  Pre-allocate files in larger swatches rather than auto-grow »Use file groups and different storage based on I/O demand  Rarely accessed things on less performant media  Frequently accessed things on highly performant media »Plan ahead! 21

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. DMV Wait Types »Multiple wait types indicate storage I/O »DPA breaks down query execution by wait type over time. 

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. DEMONSTRATION TIME! 23

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. DPA Storage I/O feature 24

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Think you have Database I/O issues? »Trial Database Performance Analyzer FREE for 14 days. »DPA can identify and recommend action on I/O issues. 25

© 2012 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. Thank you! »Brian Flynn »Twitter »LinkedIn: brianpaulflynn »Scripts : server-table-scans-bf01/ server-table-scans-bf01/ 26