2. SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright © 2006 Mark Russinovich

3. 3 Mark Russinovich Co-founder and chief software architect of Winternals Software (www.winternals.com) www.winternals.com Co-author of Windows Internals, 4th edition and Inside Windows 2000, 3rd Edition with David Solomon Author of tools on www.sysinternals.com www.sysinternals.com Home of Mark’s blog and forums Microsoft Most Valuable Professional (MVP) Senior Contributing Editor to Windows IT Pro Magazine Ph.D. in Computer Engineering

4. 4 David Solomon President of David Solomon Expert Seminars (www.solsem.com) www.solsem.com Founded in 1992 1982-1992: VMS operating systems development Teach public and private live classes on Windows Internals and Advanced Troubleshooting Microsoft Most Valuable Professional (MVP) Books Windows Internals, 4th edition Inside Windows 2000, 3rd edition Inside Windows NT, 2nd edition Windows NT for OpenVMS Professionals Videos Windows Internals COMPLETE New! Sysinternals Video Library (see DVD in bag)

5. 5 Outline Introduction Troubleshooting with Filemon Troubleshooting with Regmon Using Filemon and Regmon Together

6. 6 Troubleshooting Application Failures Most applications do a poor job of reporting file- related or registry-related errors Permissions problems Missing files Missing or corrupt registry data Errors manifest in several different ways Misleading error messages Crashes Silently exiting Hangs

7. 7 Troubleshooting Application Failures When in doubt, run Filemon and Regmon! Filemon monitors file I/O Regmon monitors registry activity Ideal for troubleshooting a wide variety of application failures Also useful for to understand and tune file system and Registry access Understand hard drive activity Optimize application installation and configuration Filemon and Regmon run on Microsoft Windows 95, Windows 98, Windows Me, Windows 2000, Windows XP, Windows Server 2003, x64 64-bit Editions, Windows Vista

8. 8 Using Regmon/Filemon Two basic techniques: Go to end of log and look backwards to where problem occurred or is evident and focused on the last things done Compare a good log with a bad log Often comparing the I/O and Registry activity of a failing process with one that works may point to the problem Have to first massage log file to remove data that differs run to run Delete first 3 columns (they are always different: line number, time, process id) Easy to do with Microsoft Office Excel by deleting columns Then compare with FC (built in tool) or Windiff (Resource Kit)

9. 9 Outline Introduction Troubleshooting with Filemon Troubleshooting with Regmon Using Filemon and Regmon Together

10. 10 How Filemon Works Filemon is based on a file system “filter driver” Extracts the driver to \Windows\System32\Drivers Installs the driver Deletes the driver file Requires “Debug Programs” user right First run requires the “Load Driver” user right Application FilemonDriver FilemonGUI File System Driver User Mode Kernel Mode

11. 11 Using Filemon # - operation number Process: image name + process id Request: internal I/O request code Result: return code from I/O operation Other: flags passed on I/O request

12. 12 Controlling Filemon Start/stop logging (Control/E) Clear display (Control/X) Open Microsoft Internet Explorer window to folder containing file: Double click on a line does this Find – finds text within window Save to log file Advanced mode Network option

13. 13 What Filemon Monitors By default Filemon traces all file I/O to: Local non-removable media Network shares It saves all output for display Can exhaust virtual memory in long runs You can limit captured data with history depth You can limit what is monitored: What volumes to watch in Volumes menu What paths and processes to watch in Filter dialog What operations to watch in Filter dialog (reads, writes, successes and errors)

14. 14 Filemon Filtering and Highlighting Include and exclude filters are substring matches against the process and path columns Exclude overrides include filter Be careful that you don’t exclude potentially useful data Capture everything and save the log Then apply filters (you can always reload the log) Highlight matches all columns

15. 15 Understanding Disk Activity Use Filemon to see why you’re hard disk is crunching Process performance counters show I/O activity, but not to where System performance counters show which disks are being hit, but not which files or which process Filemon pinpoints which file(s) are being accessed, by whom, and how frequently You can also use Filemon on a server to determine which file(s) were being accessed most frequently Import into Excel and make a pie chart by file name or operation type Move heavy-access files to a different disk on a different controller

16. 16 Polling and File Change Notification Many applications respond to file and directory changes A poorly written application will “poll” for changes A well-written application will request notification by the system of changes Polling for changes causes performance degradation Context switches including TLB flush Cache invalidation Physical memory usage CPU usage Alternative: file change notification When you run Filemon on an idle system you should only see bursty system background activity Polling is visible as periodic accesses to the same files and directories File change notification is visible as directory queries that have no result

17. 17 Demo: Change Notify Explorer posts change notify to know when directory contents change for open Internet Explorer windows Open Internet Explorer window to a folder Create or delete a file

18. 18 Demo: Understanding Notepad’s File Save  Run Filemon  Set filter to only include Notepad.exe  Run Notepad  Type some text  Save file as “test.txt”  Go back to Filemon  Stop logging  Set highlight to “test.txt”  Find line representing creation of new file Hint: look for create operation

19. 19 Basic vs. Advanced Mode Basic mode massages output to be sysadmin- friendly and target common troubleshooting Things you don’t see in Basic mode: Raw I/O request names Various internal file system operations Activity in the System process Page file I/O Filemon file system activity

20. 20 Example: Word Crash While typing in the document Microsoft Office Word XP would intermittently close without any error message To troubleshoot ran Filemon on user’s system Set the history depth to 10,000 Asked user to send Filemon log when Word exited

21. 21 Solution: Word Crash Working backwards, the first “strange” or unexplainable behavior are the constant reads past end of file to MSSP3ES.LEX User looked up what.LEX file was Related to Word proofing tools Uninstalled and reinstalled proofing tools and problem went away

22. 22 Example: Build Fails While building a program using nmake on a command line link reported an error: “error writing to program database, check for insufficient disk space, invalid path, or insufficient privileges”

23. 23 Solution: Build Fails Saw sharing violation in Filemon: Performed a handle search for the file in Process Explorer Saw Windbg had it opened from an earlier debug session even though debug session was closed Closed Windbg

24. 24 Example: Useless Excel Error Message Excel reports an error “Unable to read file" when starting

25. 25 Solution: Excel Error Message Filemon trace shows Excel reading file in XLStart folder All Microsoft Office apps autoload files in their start folders Should have reported: Name and location of file Reason why it didn’t like it

26. 26 DLL Problems Process Explorer may solve a DLL versioning issue, but may not if: A DLL is missing The order of DLL loads is relevant So, use Filemon! Look at the last DLL opened before the application died Compare the startup of a working with a failing application Missing or inaccessible DLLs often not reported correctly Look for “NOTFOUND” or “ACCESS DENIED” May be opening wrong versions due to wrong versions being in folders in PATH

27. 27 Example: Word Dies Word starts and a few seconds later gets a Dr. Watson (access violation) Customer tried re-installing Microsoft Office – still failed Solution: Ran Filemon, looked at last DLL loaded before Dr. Watson It was a printer DLL Uninstalled printer – problem went away

28. 28 Problem: Perfmon Hangs Perfmon hung when starting IIS performance counter DLL was last thing Perfmon referenced

29. 29 Solution: Perfmon Hangs Services snapin showed IIS was hung starting Investigation revealed an IIS configuration problem

30. 30 Example: Help Fails The Help command in an application failed on Windows 95, but worked fine on Windows 98/Windows Me/Windows NT4/Windows 2000/Windows XP Failed with meaningless error message Ran Filemon on failing system and working system Reduced log to file opens Compared logs

31. 31 Solution: Help Fails At the point logs diverged, looked backwards to last common thing done An OLE system DLL was loaded Noticed this OLE DLL was loaded from a directory in the user’s PATH on Windows 95, but from \Windows\System on other versions Conclusion: DLL loaded on Windows 95 system was not for Windows 95 Got proper version for Windows 95, problem went away

32. 32 Example: Access Hangs Problem: Access would hang when trying to import an Excel file Worked fine on other users’ workstations Traced startup of Access on failing and working systems

33. 33 Solution: Access Hangs Compared logs and looked for first unexplainable difference First unexplainable difference was that Accwiz.dll was being loaded from two different directories Failing system was loading an old Access DLL from \windows\system32 due to having installed older Access previously Solution: Removed DLL in \windows\system32 and problem went away

34. 34 Example: Pinnacle Studio Hangs User had a hang when launching Pinnacle Studio Filemon showed accesses to CyberPatrol's DLL, an Internet filtering tool Cyberpatrol monitors processes by loading a DLL into them Uninstalling CyberPatrol fixed the problem

35. 35 Example: Misleading AOL Error AOL worked in one user’s account, but failed with this on another account on the same system: User reinstalled AOL, but problem persisted

36. 36 Example: Misleading AOL Error User did not have admin rights to AOL directory This version of AOL was not limited-user account friendly

37. 37 Example: Microsoft Office Outlook Application Error For example, an Outlook application failed with this error: Ran Filemon and found it was getting Access Denied Someone had misread a request to remove EDIT rights and removed all rights

38. 38 Example: Microsoft Software Installer Misleading Error User received this message trying to install something: Filemon showed the real reason:

39. 39 Demo: Permission Problems and Misleading Error Messages  In Explorer, create a folder c:\noaccess  Remove all rights to the folder  Run Notepad & type some text  Run Filemon – set filter to Notepad.exe  In Notepad, File->Save As to c:\noaccess\test.txt Note error reported  Look at Filemon trace and find Access Denied

40. 40 Outline Introduction Troubleshooting with Filemon Troubleshooting with Regmon Using Filemon and Regmon Together

41. 41 Configuration Problems Missing, corrupted or overly-secure Registry settings often lead to application crashes and errors Some applications don’t completely remove registry data at uninstall Regmon may yield the answer

42. 42 How Regmon Works Regmon uses a driver to intercept Registry operations Up until now Regmon has relied on system call “hooking” to intercept Registry accesses Hooking isn’t supported by the kernel As of Windows XP the system call table is write-protected by default if a system has <256 MB, requiring a trick Windows Server 2003 introduces a Registry callback mechanism Driver can see and modify Registry behavior Latest version of Regmon comes with two drivers: one for Windows Server 2003 and one for previous versions Defined in a DDK header file and used by antivirus products Application RegmonDriver RegmonGUI RegistrySubsystem

43. 43 Regmon UI is similar to Filemon Request: OpenKey, CreateKey, SetValue, QueryValue, CloseKey Path HKCU=HKEY_CURRENT_USER (per-user settings) HKLM=HKEY_LOCAL_MACHINE (system wide settings) Result – return code from Registry operation Other – extended information or results

44. 44 Polling and Registry Change Notification Many applications want to respond to Registry changes Polling the Registry is just as bad for performance Applications can request to be notified of changes Like with Filemon, Regmon should be idle on an idle system

45. 45 Registry Troubleshooting If you suspect registry data is causing problems, rename the key and rerun the application Most applications recreate user settings when run In this way, the data won’t be seen by the application Can always rename the key back Use Regmon to discover application settings location

46. 46 Demo: Finding Notepad’s Settings  Run Notepad  Change Font  Run Regmon and filter to Notepad.exe  Exit Notepad  In Regmon log, find location of user-specific Notepad settings  Double click on a line to jump to Regedit  Delete top level Notepad user settings key  Re-run Notepad and confirm font resets to default setting

47. 47 Example: Missing Word Toolbar Problem: User somehow disabled all toolbars and menus in Word No way to open files, change settings, etc. Solution: With Regmon, captured startup of Word Found location of user-specific settings for Word Deleted this Registry key Re-ran Word, which recreated user settings from scratch

48. 48 Example: Misleading Internet Explorer Error Message Internet Explorer failed to start with this error: First, looked on system for ICFGNT.DLL Not there Not on other systems in the network, either

49. 49 Solution: Misleading Internet Explorer Error Message Captured Regmon trace and looked backwards from end of Regmon log Saw query of Completed value in Internet Connection Wizard key Value read was 0 Value was 1 on other systems Solution: Set value to 1 and problem went away

50. 50 Example: Internet Explorer Hangs Internet Explorer hung when started unless user manually dialed ISP Captured a Regmon trace and looked backwards from point Internet Explorer was hung Found references to ATT under a RAS PhoneBook key Solution: renamed ATT key and problem went away Conclusion: previous ISP’s dialer had left junk behind

51. 51 Example: Misleading Microsoft Visual Basic for Applications (VBA) Error User got this error installing an application: Regmon showed permissions problem Solution: Edited permissions

52. 52 Missing Settings Sometimes queries to what is not there is more interesting than what is there Identify missing Registry keys Search for status “NOTFOUND” May reveal hidden capabilities

53. 53 Example: Hidden Capabilities User tried installing Compaq utility on non- Compaq hardware Regmon showed hidden key to override: Solution: created key and install succeeded

54. 54 Outline Troubleshooting with Filemon Troubleshooting with Regmon Using Filemon and Regmon Together

55. 55 Filemon and Regmon Many times its not clear whether a problem is Registry or file related And sometimes problems involve both Registry and file configuration problems Always run Filemon and Regmon when troubleshooting

56. 56 Example: Internet Explorer Hangs Internet Explorer started hanging on certain folders Hangs were up to a minute Internet Explorer would work normally for a minute and then hang again

57. 57 Solution: Internet Explorer Hangs Ran Filemon and saw network path error Contained references to decommissioned computer Regmon showed icon lookup configured for missing computer Fix: Delete Paint Shop Pro (PSP) browse files and all PSP file associations

58. 58 Running Filemon/Regmon Before Logon Sometimes need to capture I/O or registry activity during boot, the logon or logoff process Problem: when you logoff all your processes are terminated Solutions: Run Filemon/Regmon in a different logon session psexec –s –i –d Run Filemon/Regmon from a service Use Srvany (Resource Kit) Use Regmon’s log boot option

59. 59 Resources Technical Chats and Webcasts http://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asp Microsoft Learning and Certification http://www.microsoft.com/learning/default.mspx MSDN & TechNet http://microsoft.com/msdn http://microsoft.com/technet Virtual Labs http://www.microsoft.com/technet/traincert/virtuallab/rms.mspx Newsgroups http://communities2.microsoft.com/ communities/newsgroups/en-us/default.aspx Technical Community Sites http://www.microsoft.com/communities/default.mspx User Groups http://www.microsoft.com/communities/usergroups/default.mspx

60. 60 Fill out a session evaluation on CommNet and Win an XBOX 360!

61. © 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.