Will Query Store fix ‘ALL’ my Performance Issues? Surbhi Pokharna How many of you just love to tell your database that ‘Don’t run slow and always use good query plans!’ We can use QueryStore for that but you know what..you may not want to do that. There is lot more to offer in Query Store than just forcing the good Query Plans and that’s what we are going to talk about during next hour.. My name is Surbhi Pokharna I am a Production DBA, juggling with SQL Server since version 6.5 So, Does it sound familiar that database is running slow today, everything is big mess but it was much faster yesterday. Your boss expects that you should know what’s wrong and immediately fix it.

What changed??? @Surbhi_Pokharna And here you are wondering. What Changed?? Was there * An upgrade to the latest SQL version and suffered performance issues? Microsoft claim SQL 2016 is 30% faster so why has your database suddenly gone slow after you moved it to the shiny new server! * Released new Application code and suffered a slow down? New code should run faster. But when you’ve released it to production its going slower and you can’t see why. * Experience unpredictable performance? One minute its fast, the next minute is gone slow. What’s happening? In this situation, you want a way to track queries over time to understand workload patterns and changes from the baseline. You want to be able to answer quickly questions like “Why has this query gone slow? What was the execution plan this morning compared to now? Why has it changed?” @Surbhi_Pokharna

Query Store @Surbhi_Pokharna There is a tool called QueryStore, Microsoft introduced it first in Azure SQL DB in year 2015 and to SQL Server in version 2016, is basically a data recorder for your database. * Once enabled, it automatically captures and retains a history of the queries that run on your system, their execution plans and the runtime execution statistics. You can then use this information for troubleshooting performance problems. And find answers of ‘What Changed?’ @Surbhi_Pokharna

Tools for Troubleshooting Performance Performance Monitor DMVs DBCC Commands Extended Events Trace Query Plans Let’s take quick peek on other Performance Troubleshooting tools What tools have we used up to now to resolve those sort of issues? * Many of use will have used Profiler to capture traces of what’s happening on the server. It’s been with us since SQL 2000 and it can capture query performance metrics and other things and log it to a table or file It can provide very useful information for a wide range of issues. * But a trace can take quite an overhead on the server, not always but its quite often the case so Profiler isn’t the sort of thing that you can run continuously on your server, * you’ll typically start it to capture some information then stop it after a few minutes or an hour maybe. Trouble is by the time you’ve set it up and turned it on you’re probably too late to capture the problem you wanted to look at, it will have mysteriously cleared and you’ll be no wiser as to what happened. * Extended Events were introduced with SQL 2008. * Go-to measure for precision. These were a major improvement on traces because they added a much wider range of events that could be captured. There are lots of things you can do with extended events that you couldn’t do with profiler and traces. * but again there is an overhead and you have to start the extended event to capture the information you want So many of the shortcomings of traces can be applied to Extended Events * Dynamic Management Views were introduced in SQL 2005. These provide system related data and performance related data that you can easily query. DMVs provide a wealth of information but have a couple of drawbacks * There is no concept of time, its either real time data or data since the SQL server was last restarted. * Your server may have been running for months but you can’t say “give me the data just from yesterday”. * And all data is lost when the server is restarted. So if your server crashes you will have no information about what happened prior to the crash And if you have a cluster failover, this involves a server restart and you loose the data in the DMVs. Probably at the moment you want it the most. Query Store is a new tool in your armoury to give you an extra layer of help. SP/CU Release Notes @Surbhi_Pokharna

What can Query Store do for us? Top resource consuming Over duration Need to be parametrized Best of All Regressed (slowed) Have too many plans Force SQL plan Using the built in reports you can quickly and easily find the worst culprits that are causing performance problems on your server. For instance you can; Identify the most expensive queries in your database by CPU, execution time, memory consumption read and writes and so forth. Analyse the resource usage patterns for a particular database. See when the busiest times are and what queries are being run at that time Determine the number of times a query was executed in a given period Audit the history of query plans for a given query. See if you have queries that are changing execution plan regularly. Identify all the non parametrised queries. So you can fix the code to use parameterised queries. And Best of all You can find all the queries that have slowed down, what Microsoft calls, regressed (in other words where a new execution plan has been created which is not as efficient as the older one) You can then easily force the server to use a previous query plan which will give you back the performance you had “yesterday” without having to rewrite an code. @Surbhi_Pokharna

Enabling Query Store SSMS QueryStore can be enabled using SSMS or using T-SQL * It is integrated into the SQL Server engine so it is part of the normal processing and execution process. And the great news is that it is supported on all SQL 2016 Server editions , Enterprise down to Express. * It also has quite a limited performance impact of around 3-5% on average (so Microsoft says) so you can leave it running without causing too much of a problem to your server. * It is enabled at database level so you can choose which database on a server you want to collect data which means you don’t have to collect unnecessary data and avoid unnecessary overhead. *It is integrated into SQL Server Management Studio. Right click the database, select Query Store Turn On Click OK and query store is collecting data. It’s that simple There a number of parameters that you can choose. We’ll will discuss in more detail in a moment. General Operation Mode Off, Read Only or Read/Write Monitoring Data Flush Interval How often data is written to disk Default 900s Statistics Collection Interval Metrics will be aggregated on this interval Default 60 mins Query Store Retention Maximum Storage Size Maximum amount of space the Query Store will consume When this is hit, turns read only Default 100Mb Query Capture Mode What queries to capture (ALL, AUTO, NONE) Size based Cleanup Mode Controls whether the clean-up process will automatically activate when the total amount of data gets close to the maximum Default Off Stale Query Threshold How long to keep old queries in the store Default 30 Days NOT in SSMS Maximum plans per query Default 200 @Surbhi_Pokharna

Enabling Query Store – T-SQL ALTER DATABASE QS_Test SET QUERY_STORE = ON –- or OFF GO ALTER DATABASE QS_Test SET QUERY_STORE ( OPERATION_MODE = READ_WRITE, -- READ_ONLY MAX_STORAGE_SIZE_MB = 100, INTERVAL_LENGTH_MINUTES = 60, -- aggregate statistics DATA_FLUSH_INTERVAL_SECONDS = 900, -- frequency data is written to disk QUERY_CAPTURE_MODE = ALL, -- AUTO, NONE CLEANUP_POLICY = (STALE_QUERY_THRESHOLD_DAYS = 30), -- retention period SIZE_BASED_CLEANUP_MODE = AUTO, -- clean up process MAX_PLANS_PER_QUERY=200) Enabling Query Store –T-SQL Naturally, you can also enable and configure Query Store using T-SQL There is a simple alter database command to enable it. This will use the defaults for the parameters ON or OFF Off. Its not collecting data and the data is not available for queries On. Its one of read/write or read only Operation Mode Read/Write is the normal mode. Here it will collect and update Query Store with your performance data Read Only In this case no more data is added to Query Store but it can be read and queried. You might use this to freeze the performance data for analysis offline Maximum Storage Size This is the maximum amount of space that Query Store will use. Query Store does not auto grow so when this maximum is reached, Query Store becomes read only and stops collecting any more data. Default 100Mb Interval length Statistics Collection Interval The time window that the metrics will be aggregated on Default 60 mins Data is always in motion Data Flush Interval Initially the data that Query Store captures is stored in-memory then after a period it is written asynchronously to disk. This option determines how often you want to flush this to disk. Default 900s – 15mins Query Capture Mode Specifies the query capture policy for the Query Store. (ALL, AUTO, NONE) ALL- capture every single query that is run on the database. Auto. Tries to decide which queries are worth capturing . It will try to ignore infrequently executed or insignificant queries . You have no say over what that definition is, it is only known to Microsoft. None – won’t capture nay NEW queries Default All Why a bad plan ----------------- Clean-up Policy Stale Query Threshold Defines how long to keep old queries in the store Default 30 Days Size Based Cleanup Mode Controls whether the clean up process will automatically activate and start deleting the oldest data when the total amount of data gets close to the maximum Default Off It is strongly recommended to activate size-based clean-up to makes sure that Query Store always runs in read-write mode and collects the latest data. There is one parameter which is NOT in SSMS Maximum plans per query Default 200 The default parameters are good for a quick start except set QUERY_CAPTURE_MODE = AUTO but you should monitor how Query Store behaves over time and adjust its configuration accordingly @Surbhi_Pokharna

Query Store Reports @Surbhi_Pokharna *Its really easy to setup and there are a series of built-in reports in Management Studio that allow you analyse your queries and their performance. * It includes several Catalog Views and Extended Events to access the captured information to create your own reports and alerts. And some people have used these to create their own reports in SSRS, Power BI. Query Store Reports There are four built in reports that allows you analyse your queries Regressed Queries A regressed query is one that has performed worse than it did previously This reports will show queries for which the execution metrics have recently regressed If we want to we can force a query to use a previous faster plan. What is called Plan forcing. More later Overall Resource Consumption This report shows the overall resource consumption for the database during a certain period, hourly, daily, monthly for several metrics such as duration, execution count CPU, read and writes. You can use this to identify resource patterns , see the busiest part of the day for example Top Resource Consuming Queries This report shows the top n plans during a certain period, hourly, daily, monthly You can see the queries, the time it was run and which execution plan was used each time . Tracked Queries Each query has an ID. If you have a particular query you wish to follow you can use this report to look at a particular query ID. There are plenty of dmvs to allow you to create you own SSMS reports or connect them to Power BI There is a free report file available here: http://www.sqlservercentral.com/blogs/dotnine-sql-server-and-more/2016/11/02/query-store-custom-ssms-performance-dashboard/ @Surbhi_Pokharna

What happens under the bonnet @Surbhi_Pokharna

Query Store Architecture Query Plan In memory Plan Store Query Store is integrated into the SQL Server engine so it is part of the normal processing and execution process. Query Optimization Process occurs as normal. When a query is submitted to the system, an execution plan is created and stored in the ‘plan cache’. ‘WITH’ QueryStore, an asynchronous process runs for the QueryStore, picks execution plan, query text and writes to a separate piece of memory for temporary storage. Another Asynchronous process will then write these execution plan to the QueryStore tables in the database. All these are Asynchronous processes to ensure that there is minimal impact on other processes with-in the system. When a query gets compiled and executed for the first time the query text and query plan are sent to the plan store and the runtime statistics of the query are sent to the runtime stats store. Each further time that the query is executed the run time statistics are updated and aggregated over the aggregation period into the runtime stats store Initially the data is saved to in-memory objects. When the flush interval is reached the data is placed into an asynchronous queue and an internal process then writes the data inside the queue to the tables. The default flush interval is 15 minutes which will give an average performance overhead of 3-5%. (so Microsoft claims) If you lower the interval so it flushes more often then there will be a higher impact on performance. If you have a longer interval there will be less overhead but there is more risk of losing data that is left in memory in the event of a crash. In practice the query text and execution plans are sent to the async queue straight away to avoid losing this information Its only the runtime statistics that are kept in memory until the flush interval is reached. Query Text Query Store Schema SQL Async Queue Execute Runtime Stats Data flush interval Query Store captures data in memory to minimise i/o overhead Data is persisted to disk asynchronously in the background

query_store_query_text query_context_settings Query Store Schema query_store_query_text query_context_settings Context Setting Id Query Text Id query_store_query QueryStore collects extremely rich set of data, information stored in QueryStore breaks down into two basic sets. Information about query itself And information about run Time statistics * The metrics and data is held in a series of new system tables inside the database. This means that the information is available even after server restarts which is a great improvement. Remember that Run Time metrics connects back to the plan, not the query Run time and Wait Statistics are aggregated based upon a defined time interval – Default is 1 hour * The information is automatically aggregated over a defined time interval so we can now look at what happened an hour ago or last night or see trends over the last month rather than just seeing the sum or average since the server started or what is happening right now. * It works for nearly all types of queries including in-memory OLTP queries which I believe weren't captured by DMVs. Query Id query_store_plan Plan Id query_store_runtime_stats Run Time Stats Interval Id query_store_runtime_stats

DEMO @Surbhi_Pokharna

Overall Resource Consumption @Surbhi_Pokharna

Top Resource Consuming Queries @Surbhi_Pokharna

Regressed Queries @Surbhi_Pokharna

Force Plan @Surbhi_Pokharna

What is Tracked? Select Insert Delete Update NOT Included DDL Any T-SQL DML statement Select Insert Delete Update NOT Included DDL Bulk Insert DBCC Backup etc. What is Tracked by Query Store? Any DML statement, regardless of whether its cached, or has a recompile query hint, ad-hoc queries, even internal queries. Everything else is ignored @Surbhi_Pokharna

Use Cases - Upgrade instance/App Don’t change compatibility level Enable Query Store Run workload, get a baseline Change compatibility level * Based on version Run the workload again Query Store - find plan regressions Force plans or fix code Continue to monitor Use Case Upgrading SQL One use case that Microsoft quote is that of moving a database from an old version of SQL to SQL 2016 They recommend leaving the compatibility level at the previous version then allowing query store to run and capture a baseline of query performance. Then change the compatibility level to SQL 2016 and using query store to alert you to any queries which have degraded. In SQL 2014 the query optimizer was completely rewritten and caused a number of issues for people when they upgraded so its something worth considering if you are upgrading from earlier versions. --------- Use Case Upgrading Application Another example that Microsoft quote is using Query Store to monitor updates to your application. Again use query store to track a baseline for your queries before upgrading then use it to alert you to any queries which have degraded This may require a bit more effort because if the query has changed, even adding a new comment in the middle of a DML statement then query store will treat it as a new query. So you’ll need to use the Catlog Views and procedures to obtain the comparison. @Surbhi_Pokharna

Use Cases – Regressed Queries Identify Execution patterns Activity by time Excessive Resource Consumption Failed Queries Adhoc Queries Regressed Queries Different Query Plans Typical Use Cases Typically Query Store usage will be divided into two types Firstly it to look at performance and execution patterns What time of day do we see the most activity? When is our busiest time, morning afternoon or evening? How does performance vary over the last month? Is it getting worse or better? Which queries are causing the biggest performance hit on our database which may need looking at? Are there queries that get cancelled by the user or by the system. Maybe work out why, lack of resources, users doing silly things. Maybe you have one of these reporting tools that allow users to design their own query without any understanding of the database and end up running an impossibly big query Are there people running adhoc queries? Are there non parameterised queries which need converting? Secondly Using Query Store to find bad execution plans or too many execution plans and fix them by forcing the query optimiser to use a previous “good” plan. @Surbhi_Pokharna

Limitations! @Surbhi_Pokharna Limitations There are plenty of things to be aware of when you start to use Query Store Not so much limitations but things you must consider when you start to use query store First off it is enabled at the database level, it is not a server wide setting. This can be seen as a benefit because you can limit the databases you want to manage so reducing the overhead caused by Query Store. But if you want to manage another database you must remember to turn it on. And if you build a new instance you must remember to turn on Query Store for the databases that you wish to monitor. Also it cannot be enabled for either master or tempdb. People have tried, it just doesn’t work. The reason this is important is that the query is captured in the context of the database. If you connect to master and run a query on your enabled database, query store will not capture the information. You must do a “USE Database” command to change context to that database for query store to capture the information. So the “default database” setting for logins may become very important You cannot capture read only databases and this includes replicas such as Always On Availability Group secondaries. Many people use Always On secondary databases in read only mode for reporting purposes. You won’t be able to capture the usage and statistics of these. There is no control over the Query Store. Multiple DBAs could change the settings without reference to what others may have done. So management could be difficult Db_owner permissions allow you to configure and force execution plans To just view the Query Store reports you require VIEW DATABASE STATE. --------------------------------------------------- Limitations 2 There is no auto growth setting for Query Store like you find in a SQL database. You have to set the maximum size that you want the query store to grow to. You must keep an eye on the size of the Query Store because if it reaches its maximum size it automatically becomes read only and no more data will be added. You may find that query Store isn’t collecting data just at the time you really need it. You will have to keep a balance between the size and how much history you really require. Check these parameters to ensure you don’t run out of space. Max Size of 100 MB may not be sufficient if your workload generates large numbers of different queries and plans or if you want to keep query history for a longer period of time. Statistics Collection Interval:  The aggregation interval of your runtime statistic will directly affect the amount of data in the runtime statistics table The default is 1 hour which will give one row per query per plan per hour. If you lower the value for a finer granularity the larger the query store will grow. If you change from 60 minutes to 1 minute you’ll have 60 times the amount of data. Stale Query Threshold defines the retention period for inactive queries. By default, Query Store is configured to keep data of inactive queries for 30 days. Which may not be long enough, but you’ll need more space. Size Based Cleanup Mode is the parameter that can stop your database going read only. If this is turned on then Query Store will purge the oldest data when the database gets to 90% of its maximum size and delete data to get it down to 80% of the maximum size. By default this is off. But Microsoft strongly recommends that this is to activated to makes sure that Query Store always runs in read-write mode and collects the latest data. But you’ll have to make sure that the maximum size is big enough not to lose the data you want. It will just remove oldest data, it won’t distinguish what is worth keeping or not Query Store Capture Mode determines what queries are captured. All, which is the default, captures every single query regardless of how infrequent or insignificant they are. Whereas Auto  just captures what it thinks are queries worth capturing. Infrequent, insignificant queries and adhoc queries are ignored. How it does this and what defines an insignificant query is not published. You may miss some queries that you really wanted. None – Query Store stops capturing new queries. –this means it will only continue to capture statistics for queries it already knows about but won’t add any new queries ------------------------------------------------------ @Surbhi_Pokharna

Forcing an execution plan <may not be the answer!> May be temporary Schema changed? Query changes (even slightly)? Data volume changes? Significant time has passed? Overly “pinned” systems Limitations 4 The Query store data is stored in the primary file group. You cannot change this. If you have your user tables in your primary file group as well then you may get file contention where Query store is trying to write to the database at the same time as the users are trying to update their data. The database will be naturally larger for backups and restores. Forcing Execution Plans is obviously what Microsoft has been promoting and it does offer a quick fix. But is it a long term solution? If you start using query store to force the optimizer to use a different query plan, you must go back and check that it is still valid What if significant amount of data has changed in the table or a new index has been added? Is that forced execution plan still valid? You should see forcing a plan as a quick fix giving you time to find a long term solution And make sure you periodically check what forced plans you have otherwise you could be making the problem worse Query Store is clever enough to see that if the forced query plan can not be used (for example dropping an index which the forced query plan uses), it will use a new query plan instead of failing the query and breaking the application. @Surbhi_Pokharna

Summary Easy to use Embedded within the database engine Database specific Data not lost on restart Data aggregated over time Data is captured at query level Automatic storage management Support natively compiled procedures Highly customizable Rich set of statistics @Surbhi_Pokharna

Questions?? @Surbhi_Pokharna

Thank You! linkedin.com/in/surbhipokharna @Surbhi_Pokharna Surbhi.Pokharna@gmail.com Tune, Juggle, Improve SQL Server (since version 6.5)

References Query Store Plural Sight courses by Erin Stellato https://app.pluralsight.com/player?course=sqlserver-query-store-introduction&author=erin-stellato&name=sqlserver-query-store-introduction-m0&clip=0&mode=live Erin Stellato’s Query Store Best Practices https://www.sqlskills.com/blogs/erin/query-store-best-practices/ David Postlethwaite’s Intro to Query Store presentation SQL Server 2017 Query Performance Tuning book by Grant Fritchey @Surbhi_Pokharna