System Mechanisms Santosh Kumar Singh

Trap Dispatching Trap Handler Interrupt vs. Exception What is an Interrupt ? What is an Exception ? Distinguished by the kernel

Trap Dispatching Interrupt Interrupt Service Routines System Service Calls System Services Hardware / Software Exception Ex Frame Exception Dispatcher Exception Handler Virtual address exceptions Virtual Memory manager’s pager

Trap Dispatching After an exception or interrupt, the processor records enough machine state on the kernel stack. Windows switches the thread’s kernel-mode stack and generates a “trap frame”. The kernel also has a interrupt service routine for trap handling tasks mainly for device interrupts

Task Dispatching The trap handlers in the ISR typically execute the system function KeBugCheckEx, which halts the computer when the kernel detects incorrect behavior

Interrupt Dispatching Hardware generated interrupts originates from I/O devices and they must notify the processor when they need service. System Soft wares can also generate interrupts Kernel installs interrupt trap handlers to respond to device interrupts. Interrupt trap handlers transfer control to either to ISR that handles the interrupt or to an internal kernel routine that responds to the interrupt. Device drivers supply ISRs to service device interrupts.

Hardware Interrupt Processing Hardware platforms in windows, come into one of the lines on an interrupt controller. The controller in turn interrupts the processor on a single line. Once kernel is interrupted, it asks the controller to get the Interrupt Request (IRQ). The controller translates the IRQ to an interrupt number and uses this number as an index into a structure called Interrupt dispatch table (IDT). During boot time, windows fills the IDT with pointers to kernel routines that handle each interrupt and exception. Each processor has separate IDT so that different processors can run different ISRs.

X86 / x64 / IA 64 Interrupt Controller X86 Interrupt Controller Most x86 systems has either i8259A Programmable Interrupt Controller (PIC) or i82489 Advanced Programmable Interrupt Controller. PIC work only on uniprocessor systems and has 15 interrupt lines. APIC work with multiprocessor systems and has 256 interrupt lines. X64 Interrupt Controller It has the same interrupt controllers as with x86 as it is compatible with X86. IA64 Interrupt Controllers It uses Streamlined Advanced Programmable Interrupt Controller (SAPIC), which is an evolution of APIC. The major difference is that the I/O APICs on an APIC system deliver interrupts to local APICs over a private APIC bus, where as on a SAPIC system interrupts traverse the I/O and system bus for faster delivery. Another difference is that interrupt routing and load balancing is handled by APIC private bus, but a SAPIC system requires that the system to be present in the firmware.

X86 APIC architecture CPU 0 CPU 1 Local Local APIC APIC I8259A- I/O Equivalent PIC Device Interrupts

Software Interrupt Request levels (IRQLs) Windows impose its interrupt priority scheme known as Interrupt request levels (IRQLs). The kernel represents IRQLs internally as numbers from 0 through 31 on x86 and 0 through 15 on IA64, with higher numbers representing higher-priority interrupts. The HAL maps hardware interrupt numbers to the IRQLs. Interrupts are served in priority order, and a higher priority interrupt preempts the servicing of a lower priority interrupt. Thread scheduling priority is an attribute of a thread, where as an IRQL, is an attribute of an interrupt source.

X86 interrupt request levels (IRQLs) 31 High 30 Power fail 29 Inter-processor interrupt Hardware Interrupts 28 Clock 27 Profile 26 Device n …. 3 Device 1 2 DPC/dispatch Software interrupts 1 APC Passive Normal thread execution

Mapping interrupts to IRQLs In windows, a type of device driver called a bus driver determines the presence of devices on its bus and what interrupt can be assigned to a device. The bus driver reports the information of the interrupt type to the plug and play manager for decision. Then it calls the HAL function HalpGetSystemInterruptVector, which maps interrupts to IRQLs.

Pre-defined IRQLs The kernel uses high level when its halting the system in KeBugCheckEx and masking out all interrupts. Power fail level is used for system power failure code. Inter-processor interrupt level is used to request another processor to perform an action. Clock level is used for the system’s clock for keeping track of time of day and schedule tasks. The system’s real-time clock uses profile level when kernel profiling, a performance measurement mechanism is enabled. The device IRQLs are used to prioritize device interrupts DPC/dispatch-level and APC level interrupts are software interrupts that the kernel and device drivers generate. The lowest IRQL, passive level, isn't really an interrupt level at all; it’s the setting at which normal thread execution takes place and all interrupts are allowed to occur. Interrupt Objects – The kernel provides a portable mechanism- a kernel control object called an interrupt object that allows device drivers to register ISRs for their devices.

Software Interrupts Dispatch or Deferred Procedure call Interrupts The kernel generates software interrupts for a variety of tasks Initiating thread dispatching. Non-time critical interrupt processing. Handling timer expiration. Asynchronously executing a procedure in a context of a particular thread. Supporting asynchronous I/O operations. Dispatch or Deferred Procedure call Interrupts When a thread can no longer continue executing, perhaps because it has terminated or because it voluntarily enters a wait state, the kernel calls the dispatcher directly to effect an immediate context switch. A Deferred Procedure calls (DPC) is a function that performs a system task- a task that is less time critical than the current one. The kernel uses DPCs to process timer expiration and to reschedule the processor after a thread’s quantum expires. A DPC is represented by the DPC Object, that is not visible in user mode but visible to device drivers and other system code. A DPC object contains the address of the of the sytem function that the kernel will call when it processes the DPC interrupt, which are stored in DPC queues.

Asynchronous Procedure Call (APC) Interrupts Asynchronous procedure calls provide a way for user programs and system code to execute in the context of a particular user thread. APCs are described by a kernel control object, APC Object, waiting to execute reside in a kernel managed APC queue. There are 2 kinds of APCs: Kernel mode and user mode. Kernel mode APCs don’t require permissions from a target thread to run in that thread’s context, while user-mode APCs do. Several windows APIs such as ReadFileEx, WriteFileEx, and QueueUserAPC, use user-mode APCs. Device drivers use kernel mode APCs. POSIX subsystem uses kernel-mode APCs to deliver POSIX signals to POSIX processors.

Exception Dispatching Exceptions are conditions that result directly from the execution of the the program that is running. Windows introduced a facility known as structured exception handling which allows applications to gain control when exception occur. On x86, all exceptions have predefined interrupt numbers that directly correspond to the entry in the IDT that points to the trap handler for a particular exception. All exceptions, except those simple enough to be resolved by the trap handler are serviced by a kernel module called the exception dispatcher. Encountering a breakpoint while executing a program being debugged generates an exception, which kernel handles by calling the debugger. Interrupt Number Exception Divide Error 4 Overflow B Segment Not Present E Page Fault

Dispatching an Exception The exception handler’s job is to find an exception handler that can dispose the exception. Trap Handler Debugger Port Debugger (first chance) Frame based handlers Exception Record Trap Handler Debugger Port Debugger (second chance) LPC Function calls Exception Port Environment Subsystem Kernel default handler

Unhandled Exceptions All windows threads have an exception handler at the top of stack that processes unhandled exceptions, which is declared in the internal windows start-of-process or start-of-thread function. The generic code for these internal start functions – Void win32StartofProcess( LPTHREAD_START_ROUTINE lpStartAddr, LPVOID lpvThreadParm){ _try { DWORD dwThreadExitCode = lpStartAddr(lpvThreadParm); ExitThread(dwThreadExitCode); } _except(unhandledExceptionFilter( GetExceptoinInformation())) { ExitProcess(GetExceptionCode()); The behavior is based on the contents of the HKLM\SOFTWARE\Microsoft NT\CurrentVersion\AeDebug registry key. There are two values: Auto and Debugger. Auto tells the unhandled exception filter whether to automatically run the debugger or ask the user what to do. By default, it is set to 1, which means it will launch the debugger automatically. However, installing Visual studio sets the value to 0.

Unhandled Exceptions A tool captures the state of the application “crash” and record its in a log file (Drwtsn32.log) and a process crash dump file (User.dmp) which is found in \windows\system32\Drwtsn32.exe Windows Error Reporting Windows error reporting automates the submission of both user mode process crashes and kernel mode system crashes. This can be configured in My Computer->Properties->Advanced->Error Reporting or in HKLM\Software\Microsoft\ PCHealth\ErrorReporting In environments where systems are not connected to the internet or where the administrator wants to control which error reports are submitted to Microsoft, the destination for the error report can be configured to an internal file server. Corporate Error Reporting that understands the directory structure created by Windows Error Reporting and provides the administrator with the option to take selective error reports and submit them to Microsoft.

System Service Dispatching A system service dispatch is triggered as a result of executing an instruction assigned to a system service dispatching. The instruction that windows uses for system service dispatching depends on the processor in which its executing. 32 bit System service dispatching On x86 processors prior to the Pentium II, windows used the int 0x2e instruction (46) decimal, which results in a trap. The trap causes the executing thread to transition into kernel mode and enter the system service dispatcher. On x86 Pentium II processors and higher, windows uses the special sysenter instruction. At boot time, windows detects the type of processor on which its executing and sets up the appropriate system call code to be used. The system code for NtReadFile in user mode – ntdll!NtReadFile: 77f5bfa8 b8b700000 mov eax,0xb7 77f5bfad ba0003fe7f mov edx,0x7ffe0300 77f5bfb2 ffd2 call edx 77f5bfb4 c22400 ret 0x24

Kernel mode System Service Dispatching The kernel uses this argument to locate the system service information in the system service dispatch table. User Mode Kernel Mode System Service Dispatch Table System Service call . ….. System Service Dispatcher 32 bit 1 2 System Service 2 3 n

Service Descriptor tables 31 13 11 Native API Native API Table Index 1 1 Unused Unused 2 2 IIS Spud Driver IIS Spud Driver 3 3 KeServiceDescriptor Table KeServiceDescriptor TableShadow

Generic access rights mapping Objects Windows Object Manager is an executive component responsible for creating, deleting, protecting and tracking objects. WinObj (available at www.sysinternals.com) displays the internal object manager’s namespace. Structure of an Object Process 1 Name HandleCount ReferenceCount Type Object Name Object directory Security descriptor Quota charges Open handles count Open handles list Object type Reference count Owned by the object Manager Process 2 Object header Process 3 Kernel Object Owned by the kernel Object Specific Data Object body Type Name Pool Type Default quota charges Access types Generic access rights mapping Synchronizable? (Y/N) Methods: Open, close, delete Parse, security, Query name Owned by the executive Executive Object Executive objects that contain kernel objects

Object Specifics If object tracking debug flag is set Method Process objects and the process type object Process Type Object If object tracking debug flag is set Process Object 1 Process Object 2 Process Object 3 Process Object 4 Object Methods Method When Method is called Open When an object handle is Opened Close When an object handle is closed Delete Before an object manager deletes an object Query Name When a thread requests the name of an object, such as a file, that exists in a secondary object namespace. Parse When the object manager is searching for an object name that exists in a secondary object namespace Security When a process reads or changes the protection of an object, such as a file, that exists in a secondary object namespace

Pointer to object header Object Handles When a process creates or opens an object by name, it receives a handle that represents its access to the object. Windows 2000 process handle table architecture Handle Table 255 255 255 Subhandle table Middle level pointers Top level pointers Audit on close Inheritable Process Protect from close Structure of a handle table entry Lock Pointer to object header A I P Access mask 32 bits

Types of Object Names Stored Object Properties Object Security – Whenever a process creates an object or opens a handle to an existing object, the process must specify a set of desired access rights. Object Retention – Objects are of two types: Permanent and Temporary Objects. Resource Accounting – A open object handle count indicates that some process is using that resource. Windows object manager provides a central facility for resource accounting. Windows uses quota charges. The registry values are 0 by default and is set at HKLM \ System \ CurrentControlSet \ Session Manager \ Memory Management Object Names – Means of accessing an object. Standard Object Directories Directory Types of Object Names Stored \GLOBAL?? MS-DOS device names (\DosDevices is a symbolic link to this directory.) \BaseNamedObjects Mutexes, events, semaphores, waitable timers, and section objects \Callback Callback objects \Device Device Objects \FileSystem File system driver objects and file system recognizer device objects \KnownDlls Section names and path for known DLLs (DLLs mapped by the system at startup time) \NIs Section names for mapped national language support tables \ObjectTypes Names of types of Objects \RPC Control Port objects used by remote procedure calls (RPCs) \Security Names of objects specific to the security subsystem \Windows Windows subsystem ports and window stations.

Synchronization Incorrect Sharing of Memory The scenario when two threads running on different processors both write data to a circular queue. Time Processor A Processor B Get queue tail Insert data at current location Get queue tail Increment tail pointer Insert data at current location /* Error */ Increment tail pointer Incorrect Sharing of Memory

High IRQL Synchronization Interlocked Operations Spinlocks Processor A Processor A Do Try to acquire DPC queue Spinlock Until SUCCESS Do Try to acquire DPC queue Spinlock Until SUCCESS Spinlock Begin Remove DPC from queue End Begin Remove DPC from queue End DPC DPC DPC queue Release DPC queue spinlock Release DPC queue spinlock Critical Section

High IRQL Synchronization Queued Spinlocks – Form of spinlock that scales better on multiprocessor than a standard spinlock. Windows defines a number of global queued spinlocks by storing pointers to them in an array contained in each processor’s processor control region (PCR). A global spinlock can be acquired by calling KeAcquireQueuedSpinlock with the index into the PCR array at which the pointer to the spinlock is stored. Instack Queued Spinlocks – Windows XP and Windows Server 2003 kernels support dynamically allocated queued spinlocks with the KeAcquireInStackQueuedSpinlock and DeReleaseInStackQueuedSpinlock functions. Executive Interlocked Operations – The kernel supplies a number of simple synchronization functions constructed on spinlocks for more advanced operations, such as adding and removing entries from singly and doubly linked lists. Examples include ExInterlockedPopEntryList and ExInterlockedPushEntryList for singly linked lists, and ExInterlockedInsertHeadList and ExInterlockedRemoveHeadList for doubly linked lists. All these functions require a standard spinlock as a parameter and are used throughout the kernel and device drivers.

Low – IRQL Synchronization Executive software outside the kernel also needs to synchronize access to global datastructures in multiprocessor environment. Kernel Dispatcher Objects The kernel furnishes additional synchronization mechanisms to the executive in the form of kernel objects, known as dispatcher objects. The user visible synchronization objects acquire their synchronization capabilities from these kernel dispatcher objects. The executive’s synchronization semantics are visible to Windows programmers through the WaitForSingleObject and WaitForMultipleObjects functions. Executive synchronization object executive resources provide both exclusive access ( like a mutex) as well as shared read access.

Low – IRQL Synchronization Waiting for Dispatcher Objects A thread can synchronize with a dispatcher object by waiting for the object’s handle. The kernel suspends the thread and changes the dispatcher state from running to waiting. A synchronization object can be in one of the two states: either in signaled state or the nonsignaled state. Waiting for a dispatcher object Initialized Set object to signaled state Ready Terminated Waiting Thread waits on an object handle Transition Running Standby

Low – IRQL Synchronization A thread object is in the nonsignaled state during its lifetime and is set to the signaled state by the kernel when the thread terminates. When an object is set to the signaled state, waiting threads are generally released from their wait states immediately. Data Structures Two data structures are key to tracking who is waiting for what: dispatcher headers and wait blocks. Both are publicly defined in the DDK include file Ntddk.h Typedef struct _DISPATCHER_HEADER { UCHAR Type; UCHAR Absolute; UCHAR Size; UCHAR Inserted; LONG SignalState; LIST_ENTRY waitListHead; } DISPATCHER_HEADER; Typedef struct _KWAIT_BLOCK{ LIST_ENTRY waitListEntry; Struct _KTHREAD *RESTRICTED_POINTER Thread; PVOID object; Struct _KWAIT_BLOCK *RESTRICTED_POINTER NextWaitBlock; USHORT WaitKey; USHORT WaitTYpe; } KWAIT_BLOCK, *RESTRICTED_POINTER PRKWAIT_BLOCK;

Low – IRQL Synchronization Thread objects Thread 1 Wait block list Thread 2 Wait block list Wait Data Structures Dispatcher objects Size Type State Wait List head Object type specific data Wait Blocks List Entry Thread Object Key Type Next Link Object A Thread 2 wait block Size Type State Wait List head Object type specific data List Entry Thread Object Key Type Next Link List Entry Thread Object Key Type Next Link Object B Thread 2 wait block Thread 1 wait block

Low – IRQL Synchronization Fast Mutexes and Guarded Mutexes – They avoid waiting for the event object if there’s no contention for the fast mutex. This gives better performance in multiprocessor systems. Executive defines ExAcquireFastMutex and ExAcquireFastMutexUnsafe functions. Guarded mutexes are primarily used by the memory manager, which uses them to protect global operations. Executive Resources – Executive resources is a synchronization mechanism that is used throughout the system, especially in file-system drivers. Threads waiting to acquire a resource for shared access wait for a semaphore associated with the resource, and threads waiting for acquire a resource for exclusive access wait for an event. Push Locks – They were introduced in Windows XP is an optimized synchronization mechanism built on the event object and like fast mutexes, they wait for an event object only when there’s contention on the lock. There are two types of Push Locks Normal Cache aware