Improve production debugging of .NET web applications Microsoft 2016 7/2/2018 8:48 PM THR4003 Improve production debugging of .NET web applications Ido Flatow Senior Architect, Microsoft MVP & RD Sela Group © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

For the Next 20 Minutes… Worker process crashes Unexplained exception

How Are we Going to Do This? What did the client report? Which steps we used to troubleshoot the issue? What did we find? How did we fix it? What were those tools we used?

If You Touch This, It Will Crash Client House call doctor service Reported IIS worker process crashes after a specific flow in the application The flow itself completes successfully Worker process starts automatically afterwards Workaround None – it’s a SYMPTOM

Troubleshooting Checked Event Viewer for ASP.NET crash messages Crashed during a finalizer! When a finalizer method crashes, so will your application! Always use try/catch if at risk of an exception in a finalizer

Troubleshooting - cntd DirectoryNotFoundException, but which directory? Logger was a 3rd-party component, that created a private log file Developers didn’t even know there was a log being created Of course, no source code available Checked the code to locate the file path Used a free decompiler (ILSpy, dotPeek, JustDecompile) Code used a DI-based configuration sub-system Tracking the file path through code was taking too long

Troubleshooting - cntd Sysinternals Process Monitor to the rescue! File path found – created required folder, and voila!

The Tools in Use Event Viewer (eventvwr.exe) First tool to open when a .NET application crashes Shows application, security, and system logs ASP.NET health monitoring writes exceptions to the Application log W3WP hang and recycle events are written to the system log (source=WAS) You can log application events to a custom log Sysinternals Process Monitor Real-time process monitor that shows registry, file, and TCP/UDP activity Filter activities by process name, file paths, result, etc. View call stack of each activity Can also be used to detect slowness due to excessive file/registry access https://download.sysinternals.com/files/ProcessMonitor.zip

The Tools in Use Decompilers Browse content of .NET assemblies (.dll and .exe) Decompile IL to C# or VB Find usage of a field/method/property Some tools support extensions and Visual Studio integration http://ilspy.net https://www.jetbrains.com/decompiler http://www.telerik.com/products/decompiler.aspx

When SSL/TLS Fails… Client Reported Workaround Airport shuttle service site Reported Application suddenly fails to communicate with external services over HTTPS Error is “Could not establish trust relationship for the SSL/TLS secure channel” Cannot reproduce the error in dev/test Workaround Restart IIS (iisreset.exe )

Troubleshooting Checked Event Viewer for any related error Found the SSL/TLS error in the Application logs And a Schannel error in the System logs According to MSDN documentation, error code 70 is protocol version support

Troubleshooting - cntd Used Microsoft Message Analyzer (network sniffer) to watch the TLS handshake messages Before issue starts – client asks for TLS 1.0, handshake completes After issue starts – client asks for TLS 1.2, handshake stops

Troubleshooting - cntd Microsoft Ignite 2016 7/2/2018 8:48 PM Troubleshooting - cntd Checked the Server Hello, it used TLS 1.1, not TLS 1.2 Switched to TCP view to verify client’s behavior Client indeed sends a FIN, and server responds with an RST © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Troubleshooting – moment of clarity Microsoft Ignite 2016 7/2/2018 8:48 PM Troubleshooting – moment of clarity Developer remembered adding code to support new Paypal standards of using only TLS 1.2 Code set to only use TLS 1.2, removing support for TLS 1.0 and 1.1 Suggested fix Use enum flags to support all TLS versions – Tls | Tls11 | Tls12 This is the actual default for .NET 4.6 and on For .NET 4.5.2 and below – default is Ssl3 | Tls © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

The Tools in Use Microsoft Message Analyzer Replaces Microsoft’s Network Monitor (NetMon) Captures, displays, and analyzes network traffic Can listen on local/remote NICs, loopback, Bluetooth, and USB Supports capturing HTTPS pre-encryption, using Fiddler proxy component https://www.microsoft.com/en-us/download/details.aspx?id=44226 Event Viewer (eventvwr.exe) Discussed previously

Additional Tools (you will see some tomorrow) Process monitoring IIS Request Monitoring, Windows Performance Monitor, Sysinternals Process Explorer and VMMap Tracing and logs PerfView (CLR/ASP.NET/IIS ETW tracing), IIS Failed Request Tracing, IIS/HTTP.sys logs, Log Parser Studio Dumps DebugDiag, WinDbg, Visual Studio dump debugger, Sysinternals ProcDump Decompilers ILSpy, JustDecompile, DotPeek Network sniffers Fiddler, Wireshark

How to Start? Understand what is happening Be able to reproduce the problem ”on-demand” Choose the right tool for the task When in doubt – get a memory dump!

Resources You had them throughout the slides  Video and slides will be available soon on Channel9 channel9.msdn.com/Events/Speakers/ido-flatow

Related Content Find Me Later At. . . Relevant Breakout Sessions Unlock the essential toolbox for production debugging of .NET Web Applications Wednesday, 9:00am, A404-405 If you got your memory dumps, join me at my Hub Talk Thursday, 10:20am, MVP Hub Talk 1 Relevant Breakout Sessions Case of the unexplained: Windows Troubleshooting with Mark Russinovich Wednesday, 4:00pm, Thomas Murphy Ballroom 2&3 Learn debugging tips and tricks for .NET Developers Thursday, 2:15pm, A302

Join the conversation on the Microsoft Tech Community Microsoft Ignite 2016 7/2/2018 8:48 PM Join the conversation on the Microsoft Tech Community THR4003 - Improve production debugging of .NET web applications © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Please evaluate this session 7/2/2018 8:48 PM Please evaluate this session Your feedback is important to us! From your PC or Tablet visit MyIgnite at http://myignite.microsoft.com From your phone download and use the Ignite Mobile App by scanning the QR code above or visiting https://aka.ms/ignite.mobileapp © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.