Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze Unit OS2: Operating System Principles 2.5. Demos

2 Copyright Notice © David A. Solomon and Mark Russinovich These materials are part of the Windows Operating System Internals Curriculum Development Kit, developed by David A. Solomon and Mark E. Russinovich with Andreas Polze Microsoft has licensed these materials from David Solomon Expert Seminars, Inc. for distribution to academic organizations solely for use in academic environments (and not for commercial use)

3 Roadmap for Section 2.5. Demos investigating: Process Execution Object Manager & Handles Interrupt Handling Memory Pools Labs System Threads System Processes

4 Lab: Examining Privileged vs. User Time 1. Run MLTITHRD Click on “Mltithrd->Bounce” menu item Observe system activity with Task Manager and Qslice 2. Run CPUSTRES (in ResKit) Change thread 1 activity to “Maximum” Observe system activity with Task Manager and Qslice

5 Tools for Obtaining Process & Thread Information Many overlapping tools (most show one item the others do not) Built-in tools in Windows 2000/XP: Task Manager, Performance Tool Tasklist (new in XP) Support Tools pviewer - process and thread details (GUI) pmon - process list (character cell) tlist - shows process tree and thread details (character cell) Resource Kit tools: apimon - system call and page fault monitoring (GUI) oh – display open handles (character cell) pviewer - processes and threads and security details (GUI) ptree – display process tree and kill remote processes (GUI) pulist - lists processes and usernames (character cell) pstat - process/threads and driver addresses (character cell) qslice - can show process-relative thread activity (GUI) Tools from Process Explorer – super Task Manager – shows open files, loaded DLLs, security info, etc. Pslist – list processes on local or remote systems Ntpmon - shows process/thread create/deletes (and context switches on MP systems only) Listdlls - displays full path of EXE & DLLs loaded in each process

6 How Process Explorer Works Uses undocumented functions for: Enumerating loaded modules with full path names Enumerating processes and handles Obtains handle names using the aid of a driver Related Tools: Handle – command-line handle viewer Listdlls – command-line DLL viewer

7 Process Explorer Lab: Refresh Highlighting 1. Press space bar to pause refresh 2. Run Notepad 3. In ProcExp, hit F5 and notice new process 4. Exit Notepad 5. In ProcExp, hit F5 and notice Notepad in red 6. Press space bar to resume normal refresh Uses: Understanding process startup sequences Detecting appearance of processes coming and going

8 Process Explorer Lab: Column Selection And Username 1. Notice additional details show for each process (icon, description) 2. Click on View->Select Columns Add username column Add username column 3. Compare username column in Task Manager with Process Explorer – what is the difference? 4. Deselect View->Show Processes From All Users

9 Process Explorer Lab: Command Line 1. Double click on date/time in task bar (lower right of screen) 2. In Process Explorer, hit F5 to refresh 3. Find new process created (RUNDLL32.EXE) 4. Examine command line arguments Example: cmd.exe process was consuming lots of CPU time Command line argument showed which.BAT file was running

10 Process Explorer Lab: Process Performance Statistics Click on Performance Tab of process properties Note: all these numbers can be configured as columns

11 Examining CPU Time Open process properties and look at CPU usage history on the performance graph page Hover the mouse over a point to see the time of that value

12 TCP/IP Endpoints Process properties TCP/IP tab shows process’ TCP and UDP endpoints Resolves addresses in the background TCPView from Sysinternals shows all endpoints

13 Process Explorer Lab: Environment Variables Click on Environment Tab of process properties Click on Environment Tab of process properties

14 Process Explorer Lab: Environment Variables 1. Open a command prompt 2. Run Notepad.exe from command prompt 3. Type “set abc=xyz” 4. In ProcExp, hit F5 and examine environment variables for Cmd.exe and Notepad.exe Notice Notepad.exe does not have abc defined Notice Notepad.exe does not have abc defined

15 Process Explorer: Thread Details Process Explorer “Threads” tab shows which thread(s) are running Start address represents where the thread began running (not where it is now) Click Module to get details on module containing thread start address Can also kill threads May be useful in an ISAPI process with a runaway provider

16 Thread Start Functions Process Explorer can map the addresses within a module to the names of functions This can help identify which component within a process is responsible for CPU usage Requires access to: Symbol file for that module Proper version of Dbghelp.dll (part of Windows Debugging Tools) Process Explorer looks for : Dbghelp.dll: in the default Debugging Tools install directory Symbols: _NT_SYMBOL_PATH environment variable Can also specify with Options->Configure Symbols

17 Viewing Call Stacks with Process Explorer Click Stack to view call stack Lists functions in reverse chronological order Note that start address on Threads tab is different than first function shown in stack This is because all user threads start in a Windows library function which calls the programmed start address

18 Example: Solving Hung Processes Problem: Powerpoint was hanging for 1 minute on startup Thread stack shows waiting on a printer driver

19 Kernel mode code always uses the current thread’s kernel mode stack Kernel stack attributes: One for each thread Mapped in system address space Normally nonpageable Because kernel mode code might be running at dispatch level IRQL or above, in which context page faults will cause a crash GDI requests a larger kernel stack size: MmGrowKernelStack( stackPointer ); Might not succeed Not documented for drivers; used internally by GDI Kernel Mode Stack

20 Suspending Processes Process Explorer can suspend a process Why would you want to do this? You’ve started a long running job but want to pause it to do something else Lowering the priority still leaves it running… You’ve started a long download but want to have your network bandwidth temporarily Some multi-service system process activity is due to other processes calling upon their services suspend a process that is consuming CPU time to see what that does to the system process in question

21 Process Explorer Lab: Suspend Start Notepad From a command prompt: 1. Suspend Notepad process with Process Explorer 2. Try to switch back to Notepad (should not respond) 3. Resume Notepad

22 PS Tools PsFile – lists & closes remote file opens PsShutdown – remote shutdown, lock workstation, log off user PsExec – run an app on a remote system PsList – list processes & threads PsUptime – system up time PsInfo – display general system info PsGetsid – displays computer or user SIDs PsService – service process control (like SC in XP) PsLoglist – dumps event log in text PsSuspend – suspend a process PsKill – kill processes PsLoggedon – lists local and remote logon sessions PsPassword – change local/remote passwords

23 HAL Choices To see the HAL list, do an “update driver” on the drivers for the “Computer” and specify manual selection from the list;

24 Variations of Routine Names… Private versions of public routines Both public (exported) and private entry points may exist Private version is not callable outside of the module that defines them Basic routine name has “p” added to the end of its prefix IopCallDriver => private version of IoCallDriver Public routine may simply be private routine with name redefined Internal routines Not callable outside of the defining module – and no public version But may be invoked by other means (traps, interrupts) Last letter of prefix changed to “i” Examples: KiTrap, MiAccessFault, KiSystemService

25 Native Images.EXEs not linked against any subsystem Interface to NT executive routines directly via NTDLL.DLL Two examples: smss.exe(Session Manager -- starts before subsystems start) csrss.exe(Windows subsystem)

26 Examining Open Handles: MS Tools Resource Kit “oh” (Open Handles) tool Can show named & unnamed handles (“-a” switch) Can select by object type (e.g. “oh -t file” shows all open files) Does not show full path name XP & 2003: openfiles /query command Both of these require the “maintain a list of objects for each type” NT “global flag” registry bit to be set Oh turns this on for you (or you can run Gflags.exe) Requires reboot to take effect See HKEY_LOCAL_MACHINE\System\CurrentControlSet \Control\Session Manager\GlobalFlag Can view this bitmask with the GFLAGS tool uses 8 bytes extra for each open handle

27 Increased System Memory Limits Key system memory limits raised in XP & Server 2003 Windows 2000 limit of 200 GB of mapped file data eliminated Previously limited size of files that could be backed up Maximum System Page Table Entries (PTEs) increased Can now describe 1.3 GB of system space (960 MB contiguous) Windows 2000 limit was 660 MB (220 MB contiguous) Increases number of users on Terminal Servers Also means maximum device driver size is now 960 MB (was 220 MB)

28 Monitoring Pool Usage Poolmon.exe (Support Tools) Shows paged and nonpaged pool consumption by data structure “tag” Must first turn on “pool tagging” with Resource Kit gflags tool & reboot On by default in Windows Server 2003 (not in XP or Win2000) Controls:“p” to toggle between nonpaged, paged pool, or both “b” to sort by total # of bytes, “a” to sort by # of allocations, “t” to sort by structure tag “?” displays help

29 Finding All the Drivers Note that while most drivers are in \Windows\System32\Drivers, they can be loaded from anywhere To check the location of all drivers: Run Msinfo32.exe, click on Software Environment->System Drivers, sort by Path Or, type “Driverquery /v” (XP & 2003) Or view loaded DLL list of System process with Process Explorer However, some drivers are deleted after they are loaded Binary file and registry key can be deleted after load Examples: Process Explorer, Filemon, Regmon To list all loaded modules, run Drivers.exe or type “lm k” in Kernel Debugger (note: only works with LiveKd) Can compare with list from Msinfo32 or Driverquery

30 Pool Usage with Kernel Debugger !poolused !poolused 1 [pooltag]Show full details !poolused 2 [pooltag]Show nonpaged pool sorted by usage !poolused 4 [pooltag]Show paged pool sorted by usage !poolused 8 [pooltag]Show session pool usage Pooltag specified can have wildcards (* or ?)

31 Troubleshooting Pool Leaks With Verifier Use Driver Verifier (described in the I/O section) Enable pool tracking for driver(s) of interest Causes system to track pool usage by driver vs Poolmon, which looks at pool usage by structure tag Reboot and monitor pool usage of driver Use “verifier /log file.txt /interval nnn” to store the output in a text file Check to see if it’s going up…

32 Troubleshooting Pool Leaks With Verifier A leaker exhibits the following Current allocations is always close to or equal to the peak The peak grows over time If the leak is significant the peak allocations or bytes will be large GUI interface to view usage:

33 Services How do services interact with the system? Must register with service control manager when started (otherwise process is killed) Get startup configuration parameters from Registry Log errors to Windows 2000 Event Log Use some form of IPC mechanism for client communication and control Likely make use of Win2K security impersonation Service implementation One.EXE may have >1 service (type code in Registry indicates) Examples of services installed by default Event Log, Task Scheduler Examples of add-on services DNS, DHCP, RAS, Web server

34 Service Control Tools Net start/stop – local system only Sc.exe (built in to XP/2003; also in Win2000 Resource Kit) Command line interface to all service control/configuration functions Works on local or remote systems Psservice (Sysinternals) – similar to SC Other tools in Resource Kit Instsrv.exe – install/remove services (command line) Srvinstw.exe – install/remove services (GUI) Why are service creation tools included in Reskit? Because Reskit comes with several services that are not installed as services when you install the Reskit

35 Understanding Svchost.exe CPU Time Consumption If a multi-service process or other multi- component process such Inetinfo.exe (IIS) or Dllhost.exe (COM) is consuming CPU time, how do you determine which service is responsible? Need to drill down to thread granularity Go to Threads tab in Process Explorer and sort by CPU usage

36 Properties of a Service General tab Logon tab Recovery tab Dependencies tab

37 Lab: Minimal Process Set Run Process Explorer Kill Smss.exe (must do this first!) Then right click on Winlogon and end process tree Kill all other processes except Process Explorer From Process Explorer, you can still run programs E.g. cmd.exe, iexplore.exe, etc. Reboot (must power off – shutdown won’t work)