1. Troubleshooting Windows Applications for IT Admins Using DebugDiag – A Taste

2. Debugging Discussion
[troubleshooting][managed][Native]

3. Types of Problems in Production
System hang or deadlock
Fatal exception
Nonfatal exception
Data loss or inconsistency
Performance problems
Memory leaks or excessive memory usage
[troubleshooting][managed][Native]

4. Causes of Problems in Production
Interaction with other systems
Hardware differences
Software differences
Unexpected load
Transient network conditions
Resource contention
[troubleshooting][managed][Native]
Resource contention: in testing there was one app/application pool but in production there are more applications contesting for the same app pool resources.

5. Challenges of Debugging in Production
Real users
Pressure to resolve quickly
Reluctance to ”make things worse”
Change control
Concern over tool installation
[troubleshooting][managed][Native]

6. Challenges of Debugging in Production (Cont.)
Hard to access server
Can’t use normal tools
No repro
Infrequent problem
Hard to characterize problem
The problem only appears at a customer site
[troubleshooting][managed][Native]

7. Production Debugging Timeline
Before a problem occurs
Pre-install debugging tools in production
Build symbols as appropriate
Build in application exception handling and logging
Design, build, and test for performance and scalability
Build in monitoring and diagnostics
Know the metrics of your system
Find and resolve problems before they become crises
Plan your deployments
[troubleshooting][managed][Native]

8. Production Debugging Timeline (Cont.)
When a problem does occur
Don’t panic; be methodical
Evaluate the symptoms
Document what you do
Consider mitigation strategies
Gather data
After the problem has occurred
Analyze data
Plan and implement steps to resolve
Evaluate the improvement and repeat as necessary
[troubleshooting][managed][Native]

9. Tools and environments
Least-invasive
Debugging tools
Environment
Debug Diagnostics, Perfmon, Event Logs, WER, Dr Watson, Netmon, procdump
Production
Testing/Staging
WinDbg, cdb, profilers, remote Debug with VS, SysInternals Suite,..
[troubleshooting][managed][Native]
Development
Most Invasive
Visual Studio

10. Installing DebugDiag Download the latest version
You will get redirected to the Microsoft Download site that points to the latest version of DebugDiag

11. 11/23/2018
DebugDiag Components
DebugDiag is composed of the following components:
The Debugging Service - DbgSvc.exe
The Debugger Host - DbgHost.exe
The Collection UI - DebugDiag.Collection.exe
The Analysis UI - DebugDiag.Analysis.exe
The Rule Designer - DebugDiag.RuleDesigner.exe
[troubleshooting]

12. Troubleshooting Exceptions
[managed][troubleshooting]

13. What Are Exceptions
Unexpected behavior (for example, an error) encountered by an executing program
Managed exception inherits from System.Exception
Contains properties that describe and provide more info about the exception
All managed languages use the same exception-handling system
[managed][troubleshooting]
An exception is an error condition or unexpected behavior encountered by an executing program. It can be thrown due to
A fault in your code or code you called
Unexpected conditions that the runtime encounters
Operating System resources not being available etc.,

14. Perfmon Counters ASP.NET version Application Restarts
.NET CLR Exceptions
# of Exceps Thrown
# Exceps Thrown/sec
Memory
Available MBytes
ASP.NET version
Application Restarts
Worker Process Restarts
[managed][troubleshooting]

15. Troubleshooting Hangs

16. Defining Application Hang
Application stops responding to user input or service requests but the process continues to exist. In other words, it is “stucked”
The scope of a hang can exist between threads and spread to process wide
Localized thread level hang involves 1 or more threads
Process wide hangs causes entire application to stop responding
Types of hangs
100% or high CPU hang
Process is non-responsive – and consumes high CPU continuously or for a noticeable period of time
0% or low CPU hang
Process stops responding – but the CPU utilization of the process is very low
[Managed][troubleshooting]

17. Troubleshooting Sequence
Obtain:
- Perfmon log (started preferably during process startup)
- Memory dumps during problem occurrence
Non-responsive
Identify process(s) in TaskMgr
Yes
[Managed][troubleshooting]
Server-side App?
Often during debugging you will need to repeat the steps to better define and root cause the issue
Obtain:
Memory dumps during problem occurrence No

18. Request Execution Time Requests Queued Request Wait Time
Performance Counters
General
Process
% Processor Time
ThreadCount
Thread
ID Thread
ID Process
.NET CLR Memory
% Time in GC
ASP.NET
Request Execution Time
Requests Queued
Request Wait Time
Requests Current
[Managed][troubleshooting]

19. Troubleshooting Memory Issues

20. Memory Issues >.NET 4.0 set complus_GCBreakOnOOM=1
Unmanaged
Symptom: Private Bytes increase while #Bytes In All Heaps remains flat
Troubleshooting path: Various tools (DebugDiag, LeakDiag, etc.)
Managed
Symptoms:
Private Bytes and #Bytes In All Heaps both increase
% Time in GC is high
Troubleshooting path: For most cases, one or more dump files
set complus_GCBreakOnOOM=1 can used to get DebugBreak() on OOM
[Managed] [troubleshooting]
.NET 2.0: HKLM\SOFTWARE\Microsoft\.NETFramework\GCBreakOnOOM (DWORD) registry key 0x2 can be used to get DebugBreak() on OOM
>.NET 4.0 set complus_GCBreakOnOOM=1

21. Symptoms in ASP.NET memoryLimit PeriodicRestartMemory
Out Of Memory Exceptions
Memory Recycling
memoryLimit
PeriodicRestartMemory
PeriodicRestartPrivateMemory
System event log: worker process recycles if memory limit set
Log Name: System
Source: Microsoft-Windows-WAS
Date: /16/2010 1:15:27 AM
Event ID:
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: <COMPUTER NAME>
Description:
A worker process with process id of '7420' serving application pool 'DefaultAppPool' has requested a recycle because it reached its private bytes memory limit.
[Managed][troubleshooting]

22. Perfmon Counters General .NET CLR Loading Current AppDomains
Current Assemblies
Bytes In LoaderHeap
.NET CLR Memory
# Bytes In All Heaps
% Time in GC
Process
Handle Count
Private Bytes
Virtual Bytes
[Managed][troubleshooting][native]

23. Memory – Troubleshooting sequence
memory leak
Yes
.NET CLR Memory: Bytes in all Heaps go up Process: Private Bytes go up
net issue? No
.NET CLR Memory: Bytes in all Heaps stay flat Process: Private Bytes are going up
at least 2 hang dumps during problem occurrence
-> with some time gap between to see the memory growth
Analyse with Windbg or Perfview
Obtain Perfmon log (started preferably during process startup) and process dump during problem occurrence
Obtain Perfmon log (started preferably during process startup) and 2 process dumps 5-7 minutes apart during problem occurrence
[Managed][troubleshooting][native]
Use DebugDiag or Xperf, LeakDiag

24. Your Instructor Theo.Browning@Microsoft.com @CodeHeadTheo
11/23/2018
Your Instructor @CodeHeadTheo
Support Engineer
Sr. Support Engineer
Technical Lead
Escalation Engineer
Premier Field Engineer
Sr. Premier Field Engineer
[troubleshooting][managed][Native]
Instructor: replace with your information ?
2007
2011
2012
2017
Exchange and Platforms
Azure, Debugging, Code, Script
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.