Troubleshooting SQL IO issues with Xperf Joe McTaggart Microsoft Premier Field Engineer © 2012 Microsoft Corporation Microsoft Confidential

Conditions and Terms of Use Microsoft Confidential This training package is proprietary and confidential, and is intended only for uses described in the training materials. Content and software is provided to you under a Non-Disclosure Agreement and cannot be distributed. Copying or disclosing all or any portion of the content and/or software included in such packages is strictly prohibited. The contents of this package are for informational and training purposes only and are provided "as is" without warranty of any kind, whether express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non-infringement. Training package content, including URLs and other Internet Web site references, is subject to change without notice. Because Microsoft must respond to changing market conditions, the content should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred. Copyright and Trademarks © 2012 Microsoft Corporation. All rights reserved. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. For more information, see Use of Microsoft Copyrighted Content at http://www.microsoft.com/about/legal/permissions/ Microsoft®, Internet Explorer®, Outlook®, SkyDrive®, Windows Vista®, Zune®, Xbox 360®, DirectX®, Windows Server® and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners. © 2012 Microsoft Corporation Microsoft Confidential

Joe McTaggart joemct@Microsoft.com Joined Microsoft in 2001 SQL CTS (customer technical support) through 2006 Premier Field Engineer since 2009 Reactive Troubleshooting Proactive engagements Training

Event Tracing for Windows

Event Tracing for Windows (ETW) General purpose, high speed tracing Kernel and Application events Built in to the OS Can be turned on and off on the fly Added in Windows 2000 Has since become the key instrumentation technology for windows. Provides the foundation for xperf/wpa, netmon, procmon, processesplorer, SQL XEvents, etc © 2012 Microsoft Corporation Microsoft Confidential

ETW Architecture Four main components --Event Provider --logical entity the writes events to ETW Sessions --Controller --Starts and Stops ETW Sessions --Consumer --application that reads trace files or listens for realtime events --Tracing Session --buffering and logging of events © 2012 Microsoft Corporation Microsoft Confidential

Xperf

Xperf Overview Consists of a controller and a consumer Xperf – trace controller Xperfviewer – consumer - reads etl traces Moving away from Xperf to WPA and WPR Windows Performance Recorder (WPR) for collecting traces Windows Performance Analyzer (WPA) for view traces Very similar to xperf/xperfviewer but with UI enhancements Part of the Windows Performance ToolKit Downloaded as part of the Windows SDK Can install just the Windows Performance Toolkit to get Xperf binaries © 2012 Microsoft Corporation Microsoft Confidential

When to use xperf? Kernel and application level tracing. Provides end to end tracing. Can capture both kernel flags and application providers at the same time High CPU CPU Sampling Stackwalk can show thread stack of high cpu consumers Slow IO MiniFilter tracing Device Driver tracing (storport tracing requires a hotfix in Windows pre-2012) Memory leak Can track memory allocations Can track Handle allocations (starting in Windows 8/Windows Server 2012)

Windows IO Stack or or NIC NIC iSCSI HBA iSCSI HBA FC HBA FC HBA Application Port driver iSCSIprt.sys I/O Subsystem NIC NIC File System NTFS.sys or Software Initiator MSiSCSI.sys or iSCSI HBA Volume Snapshot VOLSNAP.sys Multi-path MPIO.sys iSCSI HBA Port driver Storport.sys Volume Manager DMIO.sys Miniport Vendor specific FC HBA Cluster Clusdisk.sys FC HBA Time measured by Performance Monitor counters such as Avg. Disk Sec/read includes the time spent in these drivers Class driver DISK.sys diskperf.sys

Xperf Gotcha’s Windows 8 SDK shows it requires .NET Framework 4.5 Only necessary for other component and is not required for the Windows Performance Toolkit DisablePagingExecutive Regkey must be enabled to allow for proper Stackwalk information http://technet.microsoft.com/en-us/library/cc959492.aspx Trace files can get large quickly Precise tracing necessary Use circular files Lost Events If the target drive for the etl logs is not fast enough events can be lost. Increase memory buffer to reduce lost events Whenever possible, test your xperf cmd before running in production

Xperf Commands Xperf –providers KF --lists kernel flags available Xperf –providers KG --lists kernel groups available Xperf –on <KF or KG> -stackwalk <KF> -MaxBuffers <##> -BufferSize <KB> -file circular --start an xperf trace Xperf –d <filename> --stop an xperf trace

Xperf Commands – Detailed High CPU Issues: xperf.exe –on Latency –stackWalk Profile CPU with Wait Analysis: Xperf -on Latency+DISPATCHER –stackWalk CSwitch+ReadyThread+ThreadCreate+Profile Disk Issues with Mini-Filter tracing xperf -on PROC_THREAD+LOADER+CSWITCH+FILENAME+FILE_IO+FILE_IO_INIT+DR IVERS+ FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE -f kernel.etl -stackwalk CSwitch+DiskReadInit+DiskWriteInit+DiskFlushInit+FileCreate+FileCleanup +FileClose+FileRead+FileWrite+MiniFilterPreOpInit+MiniFilterPostOpInit

Xperf Demo © 2012 Microsoft Corporation Microsoft Confidential SQL IO --usbdrive.etl --Open sqldiag --pick out 15 sec error message --start with FILEIO graphs --reorder to process on left --duration, size, filename, offset on the right --order by duration --convert error message offset to decimal --sort by offset and find decimal value. --Point out all we can see event type, size of IO, time of IO, flags, etc New Window --Device IO --Sort by duration --Zoom in and cut off the very end (bad data) --Show how one driver passes to another. --looking for big drops --left should have module, process --show sql server and times spent in ntfs.sys --show how it is compressed.ndf --FILEIO split into smaller IO’s by ntfs. Not a 1:1 mapping --show storeport.sys --not a disk issue New windows --Mini Filter delay by process --sort by duration. --add process --mpfilter.sys = forefront endpoint protection --show total time of 31 sec matches diff between filtermgr.sys and ntfs.sys High CPU: --Sqlhighcpu.etl --show lining up symbols --dig into CPU Sampling graph --follow the cpu time (parallel, sort, hash) © 2012 Microsoft Corporation Microsoft Confidential

SQL PFE Blog SQL PFE Team Blog http://blogs.msdn.com/b/sql_pfe_blog/ General Xperf and looking at SQL CPU: http://blogs.msdn.com/b/sql_pfe_blog/archive/2013/03/19/troubleshootin g-sql-server-high-cpu-usage-using-xperf.aspx Xperf and SQL IO: http://blogs.msdn.com/b/sql_pfe_blog/archive/2013/04/23/identifying- cause-of-sql-server-io-bottleneck-using-xperf.aspx

Premier Field Engineer (317) 319-4943 Contact Joe McTaggart Premier Field Engineer (317) 319-4943 www.microsoft.com/microsoftservices © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION © 2012 Microsoft Corporation Microsoft Confidential