1. Janis Griffin Senior DBA
Geekin’ Out on Extended Events -Deconstructing the system_health event to solve SQL Server performance
Janis Griffin
Senior DBA

2. Who Am I Senior DBA for Confio Software Current – 25+ Years DBA
@DoBoutAnything
Current – 25+ Years DBA
Primarily SQL Server, Sybase, Oracle
Former – Database Design & Implementation
Specialize in Performance Tuning
Review Database Performance for Customers and Prospects
Interest in System Health Events

3. Agenda Extended Events Overview
System_Health - Event Session & Targets
Viewing Info via SSMS
Review what’s collected?
All Event details
SP_SERVER_DIAGNOSTICS
How to query XML data type
Examples & Usages
Appendix – Error Descriptions / More Code

4. Extended Events Overview
Lightweight event-handling mechanism
Captures event information like SQL Profiler / SQL Trace
More information plus you can now configure
When events are triggered
they can be sent to a target for further analysis
Introduced in SQL Server 2008
Very complex to code and read (parse xml)
Much Improved in 2012 with many more Events
SSMS has Extended Event Interface
Deprecation Announcement
SQL Trace is No More
won’t be in Versions > 2012
select instance_name, cntr_value
from sys.dm_os_performance_counters
where object_name like '%deprecate%'
and instance_name like '%trace%'

5. System_Health Session
Included by default with SQL Server 2012
Session starts when database engine starts
No noticeable performance effects
Collects system data to troubleshoot issues
Uses the ring buffer & event file targets
Microsoft recommends you leave it running
If session dropped, recreate by script:
u_tables.sql
C:\Program Files\Microsoft SQL Server\MSSQL11.<instanceid>\MSSQL\Install

6. System_Health (SH) Targets
Query to view the session data from different targets:
SELECT CAST(xst.target_data as xml) target_session_data
FROM sys.dm_xe_session_targets xst
JOIN sys.dm_xe_sessions xs
ON (xs.address = xst.event_session_address)
WHERE xs.name = 'system_health'
View data for specific targets in SSMS Extended Events UI
Ring buffer info is in XML format

7. Viewing Info via SSMS View the ring_buffer data in xml format
Select the ring_buffer target
Data will appear in the xml editor
Options to view event_file target, (.XEL file):
Use File -> Open in SSMS.
Drag and Drop the file into SSMS.
Double click the .XEL file.
Default location – SQL Server log directory
In SSMS > Extended Events session > select ‘View Target Data’.
fn_xe_file_target_read_file
To view more than one .XEL file:
Merge Extended Event Files from File -> Open menu.

8. Viewing SH via SSMS UI

9. Viewing SH via Function
SELECT object_name,count(*)
FROM sys.fn_xe_file_target_read_file
('C:\Program Files\Microsoft SQL Server\MSSQL11.SPEEDTY\MSSQL\Log\system_health*.xel',null,null,null)
group by object_name

10. Merging SH Files

11. What’s Collected? System Data Collected:
Sql_text & session_id for sessions that encounter:
Memory-related errors , 701, 802, 8645, 8651, 8657 and 8902
Or that have encountered errors with a severity >= 20
Non-yielding scheduler issues ( i.e. error in sql server log)
Deadlocks
Callstack, sql_text, & session_id for sessions waiting on:
on resources for > 15 seconds.
on locks for > 30 seconds.
for a long time for preemptive waits
Duration varies by wait type.
Preemptive wait is when SQL Server is waiting for external API calls
Callstack & session_id for CLR and virtual allocation failures
System component results from sp_server_diagnostics
System Health, Memory, Connectivity & Security errors using:
Their respective ring buffers - e.g. connectivity_ring_buffer_recorded.

12. What’s Collected in detail?
System_Health Session – Events

13. Detailed Events scheduler_monitor_system_health_ring_buffer_recorded
wait_info
What the heck is a ‘callstack’? Review Jonathan Kehayias’s reference:
security_error_ring_buffer_recorded

14. Detailed Events “(cont.)”
error_reported
Only shows errors with severity >= 20
Consider changing for Development environments to check that code!
memory_broker_ring_buffer_recorded
Shows Memory allocations between various components

15. Detailed Events “(cont.)”
connectivity_ring_buffer_recorded
xml_deadlock_report

16. Even More Information sp_server_diagnostics_component_result
Originally used for Cluster / Always On Functionality
Great Info Regardless!
Can be run manually
sp_server_diagnostics 5
Interval has to be 5 seconds or more
Captures diagnostic data & health information to detect potential failures. The procedure runs in repeat mode and sends results periodically. It can be invoked from either a regular or a DAC (dedicated admin) connection.

17. SP_SERVER_DIAGNOSTICS
How to run the sp_server_diagnostics package
Default is 0 – run once
May want to set up as a job that continuously runs
States: Unknown, clean, warning, error

18. SP_SERVER_DIAGNOSTICS
SYSTEM Section
<RingBufferTarget truncated="1" processingTime="98772" ... Etc…
<event name="sp_server_diagnostics_component_result" package="sqlserver" timestamp=" T02:41:47.499Z">
<data name="component">
<type name="sp_server_diagnostics_component" package="sqlserver"></type>
<value>0</value>
<text><![CDATA[SYSTEM]]></text>
</data>
<data name="state">
<type name="sp_server_diagnostics_state" package="sqlserver"></type>
<value>1</value>
<text><![CDATA[CLEAN]]></text>
<data name="data">
<type name="xml" package="package0"></type>
<value>
<system spinlockBackoffs="0" sickSpinlockType="none" sickSpinlockTypeAfterAv="none" latchWarnings="0" isAccessViolationOccurred="0" writeAccessViolationCount="0" totalDumpRequests="0" intervalDumpRequests="0" nonYieldingTasksReported="0" pageFaults="158" systemCpuUtilization="18" sqlCpuUtilization="3" BadPagesDetected="0" BadPagesFixed="0" LastBadPageAddress="0x0" />
</value>
</event>

19. System XML Query DECLARE @x XML = (select CAST(xet.target_data as xml)
from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT 'varchar(50)') AS EventName
'datetime') AS DateAndTime
,t.e.value('(data/text)[1]','varchar(255)') as Component
as [System CPU Utilizaton]
as [SQL Cpu Utilization]
as [Latch Warnings]
as [Access Violations]
as [Non Yields Reported]
as [Bad Pages Detected]
as [Bad Pages Fixed]
AS t(e)
WHERE 'varchar(50)') = ’sp_server_diagnostics_component_result'
AND t.e.value('(data/text)[1]','varchar(255)') = 'SYSTEM'

20. SP_SERVER_DIAGNOSTICS
RESOURCE Section
<text><![CDATA[CLEAN]]></text>
<resource lastNotification="RESOURCE_MEMPHYSICAL_HIGH" outOfMemoryExceptions="0" isAnyPoolOutOfMemory="0" processOutOfMemoryPeriod="0">
<memoryReport name="Process/System Counts" unit="Value">
<entry description="Available Physical Memory" value=" " />
<entry description="Available Virtual Memory" value=" " />
<entry description="Available Paging File" value=" " />
<entry description="Working Set" value=" " />
<entry description="Percent of Committed Memory in WS" value="73" />
<entry description="Page Faults" value=" " />
<entry description="System physical memory high" value="1" />
<entry description="System physical memory low" value="0" />
<entry description="Process physical memory low" value="0" />
<entry description="Process virtual memory low" value="0" />
</memoryReport> <memoryReport name="Memory Manager" unit="KB">
<entry description="VM Reserved" value=" " />
<entry description="VM Committed" value="410592" />
<entry description="Locked Pages Allocated" value="0" />
<entry description="Large Pages Allocated" value="0" />
<entry description="Emergency Memory" value="1024" />
<entry description="Emergency Memory In Use" value="16" />
<entry description="Target Committed" value="606656" />
<entry description="Current Committed" value="410592" />
<entry description="Pages Allocated" value="207536" />
<entry description="Pages Reserved" value="1024" />
<entry description="Pages Free" value="125120" />
<entry description="Pages In Use" value="241216" />
<entry description="Page Alloc Potential" value="839616" />
<entry description="NUMA Growth Phase" value="0" />
<entry description="Last OOM Factor" value="0" />
<entry description="Last OS Error" value="0" />
</memoryReport>
</resource>

21. Resource XML Query SELECT
XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT
'varchar(50)') AS EventName
'datetime') AS DateAndTime
,t.e.value('(data/text)[1]','varchar(255)') as Component
as [Last Notification]
as [OOM Exceptions]
as [AvaiilPhysical Mem(GB)]
as [Avail Paging File (GB)]
as [Percent of Committed Memory in WS]
as [Page Faults]
as [VM Committed (MB)]
as [Locked Pages Allocated (GB)]
as [Large Pages Allocated (GB)]
as [Target Committed (GB)]
as [Current Committed (GB)]
AS t(e)
WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result'
AND t.e.value('(data/text)[1]','varchar(255)') = 'RESOURCE'

22. SP_SERVER_DIAGNOSTICS
Query Processing Section
<queryProcessing maxWorkers="512" workersCreated="103" workersIdle="67" tasksCompletedWithinInterval="231" pendingTasks="0" oldestPendingTaskWaitingTime="0" hasUnresolvableDeadlockOccurred="0" hasDeadlockedSchedulersOccurred="0" trackingNonYieldingScheduler="0x0">
<topWaits>
<nonPreemptive>
<byCount>
<wait waitType="PAGEIOLATCH_SH" waits=" " averageWaitTime=" " maxWaitTime="5691" />
<wait waitType="WRITELOG" waits="465276" averageWaitTime="761491" maxWaitTime="4446" />
<wait waitType="IO_COMPLETION" waits="453320" averageWaitTime="800811" maxWaitTime="3369" />
<wait waitType="LCK_M_S" waits="369871" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="PAGEIOLATCH_UP" waits="281647" averageWaitTime="107218" maxWaitTime="2779" />
<wait waitType="ASYNC_NETWORK_IO" waits="242675" averageWaitTime="582969" maxWaitTime="4392" />
<wait waitType="SLEEP_BPOOL_FLUSH" waits="204757" averageWaitTime="102009" maxWaitTime="809" />
<wait waitType="HADR_FILESTREAM_IOMGR_IOCOMPLETION" waits="68978" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="PAGEIOLATCH_EX" waits="68281" averageWaitTime="128510" maxWaitTime="2751" />
<wait waitType="FFT_RECOVERY" waits="45976" averageWaitTime="883294" maxWaitTime="16094" />
</byCount>
<byDuration>
<wait waitType="CLR_AUTO_EVENT" waits="324" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="FT_IFTSHC_MUTEX" waits="10637" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="LCK_M_SCH_S" waits="198" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="LCK_M_U" waits="29596" averageWaitTime=" " maxWaitTime="45376" />
<wait waitType="XE_LIVE_TARGET_TVF" waits="2050" averageWaitTime=" " maxWaitTime="14034" />
<wait waitType="LCK_M_IS" waits="33" averageWaitTime=" " maxWaitTime="659798" />
</byDuration>
</nonPreemptive>

23. SP_SERVER_DIAGNOSTICS
Query Processing Section – “cont.”
<preemptive>
<byCount>
<wait waitType="PREEMPTIVE_OS_GENERICOPS" waits="811791" averageWaitTime="106905" maxWaitTime="1349" />
<wait waitType="PREEMPTIVE_OS_FILEOPS" waits="432272" averageWaitTime="596381" maxWaitTime="15461" />
<wait waitType="PREEMPTIVE_XE_CALLBACKEXECUTE" waits="398977" averageWaitTime="1082" maxWaitTime="12" />
<wait waitType="PREEMPTIVE_OS_CREATEFILE" waits="373487" averageWaitTime="221855" maxWaitTime="15461" />
<wait waitType="PREEMPTIVE_OS_WAITFORSINGLEOBJECT" waits="239927" averageWaitTime="575229" maxWaitTime="4392" />
<wait waitType="PREEMPTIVE_OS_GETFILEATTRIBUTES" waits="229899" averageWaitTime="55036" maxWaitTime="1998" />
<wait waitType="PREEMPTIVE_FILESIZEGET" waits="183914" averageWaitTime="15479" maxWaitTime="46" />
<wait waitType="PREEMPTIVE_OS_AUTHORIZATIONOPS" waits="96115" averageWaitTime="998" maxWaitTime="62" />
<wait waitType="PREEMPTIVE_OS_FLUSHFILEBUFFERS" waits="91945" averageWaitTime="114737" maxWaitTime="5071" />
<wait waitType="PREEMPTIVE_OS_REPORTEVENT" waits="47370" averageWaitTime=" " maxWaitTime=" " />
</byCount>
<byDuration>
<wait waitType="PREEMPTIVE_XE_DISPATCHER" waits="6" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="PREEMPTIVE_XE_GETTARGETSTATE" waits="10015" averageWaitTime="732191" maxWaitTime="1531" />
<wait waitType="PREEMPTIVE_OS_WRITEFILE" waits="2368" averageWaitTime="225333" maxWaitTime="4521" />
</byDuration>
</preemptive>

24. Query Processing XML Query
XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT
'varchar(50)') AS EventName
'datetime') AS DateAndTime
,t.e.value('(data/text)[1]','varchar(255)') as Component
as [Max Workers]
as [Workers Created]
as [Idle Workers]
as [Pending Tasks]
as [Unresolvable Deadlock]
as [Deadlocked Schedulers]
AS t(e)
WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result'
AND t.e.value('(data/text)[1]','varchar(255)') = 'QUERY_PROCESSING'

25. SP_SERVER_DIAGNOSTICS
IO_SUBSYSTEM Section
<event name="sp_server_diagnostics_component_result" package="sqlserver" timestamp=“snipped”
<data name="component">
<type name="sp_server_diagnostics_component" package="sqlserver"></type>
<value>3</value>
<text><![CDATA[IO_SUBSYSTEM]]></text>
</data>
<data name="state">
<type name="sp_server_diagnostics_state" package="sqlserver"></type>
<value>1</value>
<text><![CDATA[CLEAN]]></text>
<data name="data">
<type name="xml" package="package0"></type>
<value>
<ioSubsystem ioLatchTimeouts="0" intervalLongIos="0" totalLongIos="0">
<longestPendingRequests></longestPendingRequests>
</ioSubsystem>
</value>
</event>

26. IO_Subsystem XML DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT
'varchar(50)') AS EventName
'datetime') AS DateAndTime
,t.e.value('(data/text)[1]','varchar(255)') as Component
as [ioLatchTimeout]
as [intervalLongIos]
as [totalLongIos]
as [longestPendingRequests]
AS t(e)
WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result'
AND t.e.value('(data/text)[1]','varchar(255)') = 'IO_SUBSYSTEM'

27. XML Queries
System
Resource
Query Processing
IO Subsystem

28. Cool ‘free’ view
Check out Extended Events ‘system health’ Dashboards for SQL Server 2012

29. Add SH metrics to Ignite

30. Add SH metrics to Ignite

31. Alerts in Ignite

32. Alerts in Ignite
Historical view of all alerts for 1 year

33. How to Alert in Ignite

34. XE Wait types in Ignite XE_LIVE_TARGET_TVF – Watch Live Data
XE_BUFFERMGR_ALLPROCESSED_EVENT -Happens when Extended Events session buffers are flushed to targets. This wait Happens on a background thread.

35. Summary System_health session - use to trouble shoot issues
Can view easily in SSMS or query XML to alert & record history over time.
Show security & connectivity errors, system health & memory broker info
Deadlocks & many errors are captured
Consider changing error severity levels for deve/test instances
Consider changing file target size to capture more history
Can set up sp_server diagnostics for more info
Remember Trace has been deprecated
Sql Server Profile for Trace Capture
Replaces Sql Trace Stored procedures, functions and catalog views

36. About Confio Wait-Based Performance Tools Ignite8 / IgniteVM
Ignite for SQL Server, Oracle, DB2, Sybase
Helps show which SQL to tune
Based in Colorado, worldwide customers
Free trial at
– Free Current View

37. Q & A
APPENDIX

38. Errors Collected Memory-related errors
17803, 701, 802, 8645, 8651, 8657 and 8902
Non-yielding scheduler issues - error 17883

39. Errors Collected Memory-related errors – “Cont.”
17803, 701, 802, 8645, 8651, 8657 and 8902
Non-yielding scheduler issues - error 17883

40. System Health Severity Levels
Severity level 10 messages are informational 
indicate problems caused by mistakes in the information you have entered.
Severity levels from 11 through 16 are generated by the user
can be corrected by the user.
Severity levels from 17 through 25 indicate software or hardware errors.
when a level 17, 18, or 19 error occurs, you can continue working
might not be able to execute a particular statement.
Severity levels from 20 through 25 indicate system problems.
fatal errors, which means that the process is no longer running.
process freezes before it stops
client connection closes, and the client might not be able to reconnect.
Severity Level 20: SQL Server Fatal Error in Current Process
Severity Level 21: SQL Server Fatal Error in Database (dbid) Processes
Severity Level 22: SQL Server Fatal Error Table Integrity Suspect
Severity Level 23: SQL Server Fatal Error: Database Integrity Suspect
Severity Level 24: Hardware Error
Severity Level 25: Some type of system error

41. Scheduler_monitor Query
XML =
(select CAST(xet.target_data as xml)
from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'int') AS process_utilization
'int') as system_idle
'float') as page_faults
'float') as working_set_delta
'int') as memory_utilization
AS t(e)
WHERE 'varchar(50)') = 'scheduler_monitor_system_health_ring_buffer_record'

42. Wait_info Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'int') AS Duration
AS Session_id
'varchar(max)') AS wait_type
'varchar(max)') AS wait_type_no
'varchar(max)') AS ErrMsg
'varchar(max)') AS sql_text
AS t(e)
WHERE 'varchar(50)') = 'wait_info'
ORDER BY Duration desc

43. Error_reported Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'int') AS ErrNo
'int') AS Severity
'varchar(max)') AS ErrMsg
'varchar(max)') AS sql_text
AS t(e)
WHERE 'varchar(50)') = 'error_reported' '

44. Security Error Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
select 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'varchar(max)') AS calling_api_name
'int') AS error_code
'int') AS session_i
AS t(e)
WHERE 'varchar(50)') = 'security_error_ring_buffer_recorded'

45. Connectivity Error Query
XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
select 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'varchar(max)') AS type
'varchar(max)') AS tds_flags
'int') AS session_id
'varchar(max)') AS source_type
'varchar(max)') AS source_text
'varchar(max)') AS local_host
'varchar(max)') AS remote_host
'int') AS tds_input_buffer_bytes
AS t(e)
WHERE 'varchar(50)') = 'connectivity_ring_buffer_recorded'

46. Xml_dead_lock Query DECLARE @XMLData xml SELECT TOP 1
@XMLData = CAST(target_data AS xml)
FROM
sys.dm_xe_session_targets xst
JOIN sys.dm_xe_sessions xs
ON xs.address = xst.event_session_address
WHERE
xs.name = 'system_health'
and xst.target_name = 'ring_buffer'
SELECT
COUNT(1) AS cnt
AS X (Col)