1. TempDB Best Practices, Advanced Configs. & Troubleshooting Omar Gonzalez Sr. DBA at Dell Fourth MeetUp DBA200

2. About my experience Job Experience: Web Developer at Grupo Alica.NET/DB Developer at Coca-Cola.NET Support & SQL DBA at TCS DBA & Web Admin at Intel – contractor DB Admin Advisor at Dell Sr. DBA & HealthCare DB Team Lead at Dell

3. What is TempDB? TempDB is a system database in SQL Server Tempdb globally stores both internal and user objects and the temporary tables, objects, and stored procedures that are created during SQL Server operation. The tempdb can become overloaded in terms of space available and excessive DDL/DML operations. This can cause unrelated applications running on the server to slow down or fail.

4. Common Issues Running out of storage space in tempdb Queries that run slowly due to the I/O bottleneck in tempdb Excessive DDL operations leading to a bottleneck in the system tables Allocation contention. Metadata contention.

5. TempDB Usage User Objects Table and index Global and local temp tables Table variables Internal Objects Has Joins Sorts Cursors (session scope) LOB usage (batch scope) Version Store Online indexing Triggers Snapshot-based isolation levels Row versioning Free Space

6. Monitoring Space Space used = Internal + User + VersionStore Free Space = Reserved space – Space used NOTE: These calculations don’t account for pages in mixed extents. *DEMO

7. Trace Flag T1118 Trace flag 1118 forces uniform extent allocations instead of mixed page allocations. The trace flag is commonly used to assist in TEMPDB scalability by avoiding SGAM and other allocation contention points. Blocking and performance problems may occur when you enable trace flag 118 in SQL Server 2005 if the temporary table creation workload is high. Fixed in SQL 2005 with SP2 + KB936185, or with SP3. New feature in SQL Server 2016 allows to specify uniform extents at DB level instead of instance level (trace), by an alter statement. ALTER DATABASE tempdb SET MIXED_PAGE_ALLOCATION OFF; This option is OFF by default, this option is ON for all system DBs. TempDB is the only system DB that supports OFF.

8. TempDB Space Usage – User Objects BreakDown Siginificant space used by user objects may indicate poor designated queries, most of the time queries used for import-export or for processing purposes. TempDB used as a staging area OLAP / Data WareHouses You can use the sys.dm_db_file_space_usage dynamic management view to monitor the disk space used by the user objects, internal objects, and version stores in the tempdbfiles. sys.dm_db_file_space_usage Additionally, to monitor the page allocation or deallocation activity in tempdb at the session or task level, you can use the sys.dm_db_session_space_usage andsys.dm_db_task_space_usage dyna mic management views. These views can be used to identify large queries, temporary tables, or table variables that are using a large amount of tempdb disk space. sys.dm_db_session_space_usagesys.dm_db_task_space_usage

9. TempDB Space Usage – Internal Objects BreakDown Work Table LOB usage, large object data type Text, Ntext, Image, Varchar(MAX), NVarchar(MAX), FileStream, XML String functions such as SUBSTRING, REPLICATE UDF (User Defined Functions) Cursors Use set-based functionality instead temporary LOB storage is batch scoped and cursor worktable is session scoped Sort Runs Hash Joins or known as Work File ALTER INDEX WITH REBUILD, SORT_IN_TEMPDB = ON; Service Broker performs caching of objects in the context of query execution. The size is approx. 1 KB per dialog. Used by Database Mail and Event notifications DBCC CHECKDB Run DBCC CHECKDB WITH ESTIMATEONLY Space required = Largest index + “index create memory” option Min memory per query option has precedence over the index create memory option.

10. TempDB Space Usage – Version Store Triggers Snapshot-based isolation levels based on row versioning Online maintenance ALTER INDEX WITH REORGANIZE; ALTER INDEX WITH REBULID, ONLINE=ON; BulkLoad Operations CTE

11. TempDB Config. Recommendations Part 1 Size it properly Avoid small sizes and small growths Minimum growth for 0-100 MB TempDB, 10 MB When TempDB is above 100 MB use a growth of at least 10% or equivalent in MBs. Work around to presizing it. Enable Trace flag 1117 to grow all files in a filegroup equally. It not only impacts TempDB but all other DBs on the instance.

12. TempDB Config. Recommendations Part 2 Set recovery model to SIMPLE Automatically reclaims log space Create as many files as needed to maximize disk bandwith. As a general rule, create one file per core, up to 8 files. Make each data file the same size; this allows for optimal proportional-fill performance. Put the tempdb database on a fast I/O subsystem. Use disk striping if there are many directly attached disks. Add faster or additional I/O controllers. Add more cache for the controllers. Update SAN software for disks Align it to 64 KB allocation unit size, cluster size.

13. TempDB Config. Recommendations Part 3 Put the tempdb database on disks that differ from those that are used by user databases. Reduce I/O waits Make sure SWAP or pagefile is not on the same drive as TempDB. Increase I/O bandwith by adding more physical drives to the disk array and/or replace current disks with faster drives. SSD for StandAlone or SSD for TempDB in cluster SQL 2012 Set PAGE_VERIFY option to CHECKSUM starting in SQL Server 2008. In earlier versions of SQL Server, the PAGE_VERIFY option is set to NONE for the tempdb database and cannot be modified. In SQL 2008, the default value is CHECKSUM. For upgrades from SQL 2005 the default remains as NONE. When set to CHECKSUM, the PAGE_VERIFY option discovers damaged database paged caused by disk I/O path errors and reports these errors

14. Maintenance Tasks Set all the data files to the same size DBCC SHRINKFILE: Page 1:1942632 could not be moved because it is a work table page. Reduce VLFs Add/move files Monitor space Did I mention monitor space?

15. How can I monitor my TempDB? Check for free space at the DB Generate custom alerts Create events Create response jobs Make notifications Run periodic reports Schedule auto-growths

16. Common Performance Counters Avg. Disk Sec/Read Avg. Disk Sec/Write Physical Disk: %Disk Time Physical Disk: %Idle Time

17. Common Errors Codes 1101/1105 – Any session must allocate space in tempdb. 3959 – The version store is full. Usually appears after a 1101 or 1105 error. 3967 – The version store is forced to shrink because tempdb is full 3958/3966 – A transaction cannot find the required version record in tempdb.

18. Common Related Issues – High DB locks What is it? SH-DATABASE lock Symptoms General rule, keep less than 1K locks Sys.dm_tran_locks Fix/Resolution SQL 2008 SP3 CU17 SQL 2008 R2 SP2 CU13 SQL 2012 SP1 CU9 SQL 2014 CU1

19. Common Related Issues – High WRITELOG waits What is it? Long waits for SQL to write information into the transactional log file of the DB (LDF). The log pool is an in-memory cache of the transaction log. By default, log pool uses a single global PMO (Pointer to Memory Object) for all databases. When many threads try to write logs by using the log pool, all the threads have to wait for the pointer to a memory object resulting in contention. What is LSN? Symptoms Long WRITELOG waits detected on sys.dm_os_wait_stats DMV Long recovery times at AOAG Long rollback times at transactional replication General long recovery time after reboots or after restoring DBs If you saw this also check for high VLFs

20. Common Related Issues – High WRITELOG waits Fix/Resolution SQL 2012 SP1 CU3 + Enable trace flag T9024 If the issues persist even after enabling trace flag T9024, you can also enable trace flag T8048 which partitions the pointer to memory object at CPU level. SQL 2012 SP3 SQL 2014 RTM + Enable trace flag T9024 If the issues persist even after enabling trace flag T9024, you can also enable trace flag T8048 which partitions the pointer to memory object at CPU level. SQL 2014 SP1

21. Common Related Issues – High CPU waits What is it ? Talk about the phases of a process, RUNNABLE, RUNNING, SUSPENDED How can I monitor it? Dm_os_wait_stats CMEMTHREAD https://blogs.msdn.microsoft.com/psssql/2011/09/01/sql-server-20082008-r2-on-newer-machines-with-more- than-8-cpus-presented-per-numa-node-may-need-trace-flag-8048/ https://blogs.msdn.microsoft.com/psssql/2011/09/01/sql-server-20082008-r2-on-newer-machines-with-more- than-8-cpus-presented-per-numa-node-may-need-trace-flag-8048/ If SQL 2008/2008R2 & CPUs>8. Think about enabling T8048. SOS_SUSPEND_QUEUE CXPACKET Fix/Resolution DO NOT change max worker threads Check/Change MAXDOP instead & its associated cost Evaluate and determinate if you have the adequate number of CPUs and cores Check if any network offload settings are enabled. Most of those features are too old and have seen environments with issues when there are more than 32 cores present.

22. Common Related Issues – OS Level Update BIOS version Apply OS hotfix as per KB2207548 Power Plan Check that it is not set as Power Saving Mode for Prod instances, put it at least on Balanced mode, High Performance is preferred. Follow KB981285 to disable Advanced Power Management in the system’s BIOS for VMs

23. Common Related Issues – Query Level Check for top queries by CPU, disk, etc. Optimization for Ad-Hoc Workloads SQL compilations per second SQL re-compilations per second SPs WITH RECOMPILE option. Reduce contention at DB level This action will minimize locking contention while protecting transactions from dirty reads of uncommitted data modifications. ALTER DATABASE SET ALLOW_SNAPSHOT_ISOLATION ON ALTER DATABASE SET READ_COMMITTED_SNAPSHOT ON

24. Common Related Issues – Memory Pressure Buffer Cache Hit Ratio Page Life Expectancy Checkpoint pages/sec Lazywrites/sec DBCCs DBCC FREESYSTEMCACHE(‘SQL Plans’) DBCC FRESSIONCACHE DBCC DROPCLEANBUFFERS DBCC FREESYSTEMCACHE(‘All’) DBCC FREEPROCCACHE; --WARNING! DBCC SQLPERF(‘sys.dm_os_latch_stats’, CLEAR) DBCC SQLPERF(‘sys.dm_os_wait_stats’, CLEAR)

25. Sources: Optimizing tempdb Performance https://technet.microsoft.com/en-us/library/ms175527(v=sql.105).aspx Capacity Planning for TempDB https://technet.microsoft.com/en-us/library/ms345368(v=sql.105).aspx Troubleshooting Insufficient Disk Space in tempdb https://technet.microsoft.com/en-us/library/ms176029(v=sql.105).aspx Recommendations to reduce allocation contention in SQL Server tempdb database https://support.microsoft.com/en-us/kb/2154845

26. Q & A Thank You!!!