1. Troubleshooting SQL Server When You Cannot Access The Machine
Vicky Harp
Sr. Manager of SQL Server Product Management
IDERA

2. Let’s talk bugs All software has failure conditions
Corruption
Tampering
Most software has bugs
Errors in logic
Errors in execution

3. Let’s talk bugs Grey areas
Scalability
Platform and Hardware Compatibility
Language and Regional Support
When a customer experiences a problem, it’s no longer a grey area - it’s a bug.

4. Let’s talk bugs You will have a bug in your database.
A customer is going to experience that bug.
They will not be able to let you log in to their database.
This is going to happen

5. Troubleshooting without access to the machine is very hard
But it’s the best way to troubleshoot!

6. Why don’t you have access?
Security policies
Topological constraints
Logistics and time zones
Language barriers
Availability of key personnel
Inability to reproduce issue on demand

7. why shouldn’t you want access?

8. why shouldn’t you want access?
Need for NDAs or security policies on customer data
Responsibility for unintended effects
No test case enrichment
Skillset mismatch
Single threaded troubleshooting

9. It’s better to have scripts
More secure
More relevant
Build up a library which can be used by others
Can be re-used in the QA process
Usable by wide variety of personnel
Easier to train support staff

10. Build supportability into your apps
Distinct application names
Metadata information in database
Application and database logging
Self-tracing or verbose output flags for data access layer
Human readable object and column names

11. Build a support toolbox
Create both generic scripts and problem specific scripts
Document and maintain these scripts like your production code
Use your toolbox during your QA cycle

12. Build a support toolbox
Almost always collect low-cost, useful data
Application Metadata
Database Topology
Fundamental Performance Metrics
Scope and Scale Data

13. Build a support toolbox
Make it easy to collect problem-specific data
Reproduction Cases
Errors
Performance Data

14. Build a support toolbox
Create your scripts with security in mind
Assume it might be code reviewed
Clearly document what the script is doing
Do not return potentially sensitive data (you can mask it)

15. Remote troubleshooting techniques
DMV Queries
SET STATISTICS IO
SET STATISTICS TIME
Query Plans
Annotated Stored Procedures
XEvent Sessions

16. DMV Queries SQL Server ships with 100s of dynamic management views
Most DMVs are lightweight and return almost instantly
Very well documented and uncontroversial for users

17. DMV Queries Practical Uses: Table sizes, rowcounts
File and filegroup topology
Database status
Backup history, job history
Performance metrics and blocking chains
Index utilization
Schema verification

18. DMV Queries
Thoughtfully written scripts will not need modification for different environments
DMVs are partially dependent on SQL Server version, but less so with more contemporary releases

19. DEMO – DMV Queries

20. SET Statistics IO SET STATISTICS IO ON
Verbose and valuable performance data
Needs to be paired with the statement to be tuned
set statistics io on exec

21. Set Statistics IO set statistics io on exec sp_help
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0. Table 'Workfile'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0. Table 'sysschobjs'. Scan count 1, logical reads 41, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'spt_values'. Scan count 1, logical reads 2, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0. Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'sysscalartypes'. Scan count 1, logical reads 2, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

22. Set Statistics IO Look for the tables and worktables with a lot of IO
Table Name Scan Count Logical Reads Physical Reads Read Ahead Reads Lob Logical Reads Lob Physical Reads Lob Read Ahead Reads
Look for the tables and worktables with a lot of IO
Look for tables you don’t expect to be accessed

23. SET STATISTICS TIME SET STATISTICS TIME ON
Returns parse, compile, and execution time
Works well in conjunction with SET STATISTICS IO ON

24. SET STATISTICS TIME
select count(*) from MyCustomers where left(CustomerName,1) = 'a'
select count(*) from MyCustomers where CustomerName like 'a%‘
SQL Server parse and compile time: CPU time = 0 ms, elapsed time = 5 ms.
(1 row(s) affected)
SQL Server Execution Times: CPU time = 93 ms, elapsed time = 97 ms.
SQL Server Execution Times: CPU time = 32 ms, elapsed time = 30 ms.

25. DEMO – SET STATISTICS IO / SET STATISTICS TIME

26. Query Plans Gold standard in diagnosing query performance issues
Does not require user to actually execute the problem code
Requires some level of expertise to interpret

27. DEMO – Query PLans

28. Annotated Stored Procedures
Add error statements to help troubleshoot logical flow
Helpful for complex procedures where query plans are cumbersome
Either ship annotated procedures or have them available for swap-in

29. DEMO – Annotated Stored Procedures

30. Xevent Sessions
Watch what is happening in your application and interpret it after the fact
Can be tuned for a very exact use case or a broader diagnostic one
May return sensitive information!

31. DEMO – Xevent Sessions

32. A note on reproducing issues
You do not have to fully reproduce an issue: you may simulate it in code
Hard coding wait times
Contrived blocking chains
Query hints
Alternate tables
Simulation and reproduction cases should be archived like source code

33. Questions?