1. Improve production debugging of .NET web applications
Microsoft 2016
7/2/2018 8:48 PM
THR4003
Improve production debugging of .NET web applications
Ido Flatow
Senior Architect, Microsoft MVP & RD
Sela Group
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. For the Next 20 Minutes…
Worker process crashes
Unexplained exception

3. How Are we Going to Do This?
What did the client report?
Which steps we used to troubleshoot the issue?
What did we find?
How did we fix it?
What were those tools we used?

4. If You Touch This, It Will Crash
Client
House call doctor service
Reported
IIS worker process crashes after a specific flow in the application
The flow itself completes successfully
Worker process starts automatically afterwards
Workaround
None – it’s a SYMPTOM

5. Troubleshooting Checked Event Viewer for ASP.NET crash messages
Crashed during a finalizer!
When a finalizer method crashes, so will your application!
Always use try/catch if at risk of an exception in a finalizer

6. Troubleshooting - cntd
DirectoryNotFoundException, but which directory?
Logger was a 3rd-party component, that created a private log file
Developers didn’t even know there was a log being created
Of course, no source code available
Checked the code to locate the file path
Used a free decompiler (ILSpy, dotPeek, JustDecompile)
Code used a DI-based configuration sub-system
Tracking the file path through code was taking too long

7. Troubleshooting - cntd
Sysinternals Process Monitor to the rescue!
File path found – created required folder, and voila!

8. The Tools in Use Event Viewer (eventvwr.exe)
First tool to open when a .NET application crashes
Shows application, security, and system logs
ASP.NET health monitoring writes exceptions to the Application log
W3WP hang and recycle events are written to the system log (source=WAS)
You can log application events to a custom log
Sysinternals Process Monitor
Real-time process monitor that shows registry, file, and TCP/UDP activity
Filter activities by process name, file paths, result, etc.
View call stack of each activity
Can also be used to detect slowness due to excessive file/registry access

9. The Tools in Use Decompilers
Browse content of .NET assemblies (.dll and .exe)
Decompile IL to C# or VB
Find usage of a field/method/property
Some tools support extensions and Visual Studio integration

10. When SSL/TLS Fails… Client Reported Workaround
Airport shuttle service site
Reported
Application suddenly fails to communicate with external services over HTTPS
Error is “Could not establish trust relationship for the SSL/TLS secure channel”
Cannot reproduce the error in dev/test
Workaround
Restart IIS (iisreset.exe )

11. Troubleshooting Checked Event Viewer for any related error
Found the SSL/TLS error in the Application logs
And a Schannel error in the System logs
According to MSDN documentation, error code 70 is protocol version support

12. Troubleshooting - cntd
Used Microsoft Message Analyzer (network sniffer) to watch the TLS handshake messages
Before issue starts – client asks for TLS 1.0, handshake completes
After issue starts – client asks for TLS 1.2, handshake stops

13. Troubleshooting - cntd
Microsoft Ignite 2016
7/2/2018 8:48 PM
Troubleshooting - cntd
Checked the Server Hello, it used TLS 1.1, not TLS 1.2
Switched to TCP view to verify client’s behavior
Client indeed sends a FIN, and server responds with an RST
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Troubleshooting – moment of clarity
Microsoft Ignite 2016
7/2/2018 8:48 PM
Troubleshooting – moment of clarity
Developer remembered adding code to support new Paypal standards of using only TLS 1.2
Code set to only use TLS 1.2, removing support for TLS 1.0 and 1.1
Suggested fix
Use enum flags to support all TLS versions – Tls | Tls11 | Tls12
This is the actual default for .NET 4.6 and on
For .NET and below – default is Ssl3 | Tls
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. The Tools in Use Microsoft Message Analyzer
Replaces Microsoft’s Network Monitor (NetMon)
Captures, displays, and analyzes network traffic
Can listen on local/remote NICs, loopback, Bluetooth, and USB
Supports capturing HTTPS pre-encryption, using Fiddler proxy component
Event Viewer (eventvwr.exe)
Discussed previously

16. Additional Tools (you will see some tomorrow)
Process monitoring
IIS Request Monitoring, Windows Performance Monitor, Sysinternals Process Explorer and VMMap
Tracing and logs
PerfView (CLR/ASP.NET/IIS ETW tracing), IIS Failed Request Tracing, IIS/HTTP.sys logs, Log Parser Studio
Dumps
DebugDiag, WinDbg, Visual Studio dump debugger, Sysinternals ProcDump
Decompilers
ILSpy, JustDecompile, DotPeek
Network sniffers
Fiddler, Wireshark

17. How to Start? Understand what is happening
Be able to reproduce the problem ”on-demand”
Choose the right tool for the task
When in doubt – get a memory dump!

18. Resources You had them throughout the slides 
Video and slides will be available soon on Channel9
channel9.msdn.com/Events/Speakers/ido-flatow

19. Related Content Find Me Later At. . . Relevant Breakout Sessions
Unlock the essential toolbox for production debugging of .NET Web Applications Wednesday, 9:00am, A
If you got your memory dumps, join me at my Hub Talk Thursday, 10:20am, MVP Hub Talk 1
Relevant Breakout Sessions
Case of the unexplained: Windows Troubleshooting with Mark Russinovich Wednesday, 4:00pm, Thomas Murphy Ballroom 2&3
Learn debugging tips and tricks for .NET Developers Thursday, 2:15pm, A302

20. Join the conversation on the Microsoft Tech Community
Microsoft Ignite 2016
7/2/2018 8:48 PM
Join the conversation on the Microsoft Tech Community
THR Improve production debugging of .NET web applications
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Please evaluate this session
7/2/2018 8:48 PM
Please evaluate this session
Your feedback is important to us!
From your PC or Tablet visit MyIgnite at
From your phone download and use the Ignite Mobile App by scanning the QR code above or visiting
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.