Download presentation
Presentation is loading. Please wait.
1
Unit OS6: Device Management
6.3. Windows I/O Processing Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze
2
Copyright Notice © 2000-2005 David A. Solomon and Mark Russinovich
These materials are part of the Windows Operating System Internals Curriculum Development Kit, developed by David A. Solomon and Mark E. Russinovich with Andreas Polze Microsoft has licensed these materials from David Solomon Expert Seminars, Inc. for distribution to academic organizations solely for use in academic environments (and not for commercial use)
3
Roadmap for Section 6.3 Driver and Device Objects
I/O Request Packets (IRP) Processing Driver Layering and Filtering Plug-and-Play (PnP) and Power Manager Operation Monitoring I/O Activity with Filemon Most I/O operations don’t involve all the components of the I/O system. A typical I/O request starts with an application executing an I/O-related function (for example, reading data from a device) that is processed by the I/O manager, one or more device drivers, and the HAL. The I/O system is packet driven. Most I/O requests are represented by an I/O request packet (IRP), which travels from one I/O system component to another. The design allows an individual application thread to manage multiple I/O requests concurrently. An IRP is a data structure that contains information completely describing an I/O request. The I/O manager creates an IRP that represents an I/O operation, passing a pointer to the IRP to the correct driver and disposing of the packet when the I/O operation is complete. In contrast, a driver receives an IRP, performs the operation the IRP specifies, and passes the IRP back to the I/O manager, either for completion or to be passed on to another driver for further processing. Based on the IRP concept, we describe the asynchronous processing within the Windows I/O system in detail.
4
Driver Object A driver object represents a loaded driver
Names are visible in the Object Manager namespace under \Drivers A driver fills in its driver object with pointers to its I/O functions e.g. open, read, write When you get the “One or More Drivers Failed to Start” message its because the Service Control Manager didn’t find one or more driver objects in the \Drivers directory for drivers that should have started
5
Device Objects A device object represents an instance of a device
Device objects are linked in a list off the driver object A driver creates device objects to represent the interface to the logical device, so each generally has a unique name visible under \Devices Device objects point back at the Driver object
6
Driver and Device Objects
Driver Object \Device\TCP \Device\UDP \Device\IP \TCPIP Open Open(…) Write Read Read(…) When a thread opens a handle to a file object (described in the “I/O Processing” section later in this chapter), the I/O manager must determine from the file object’s name which driver (or drivers) it should call to process the request. Furthermore, the I/O manager must be able to locate this information the next time a thread uses the same file handle. The following system objects fill this need: ¦ A driver object represents an individual driver in the system. The I/O manager obtains the address of each of the driver’s dispatch routines (entry points) from the driver object. ¦ A device object represents a physical or logical device on the system and describes its characteristics, such as the alignment it requires for buffers and the location of its device queue to hold incoming IRPs. The I/O manager creates a driver object when a driver is loaded into the system, and it then calls the driver’s initialization routine (for example, DriverEntry), which fills in the object attributes with the driver’s entry points. After loading, a driver can create device objects to represent devices, or even an interface to the driver, at any time by calling IoCreateDevice or IoCreateDeviceSecure. However, most Plug and Play drivers create devices with their add-device routine when the PnP manager informs them of the presence of a device for them to manage. Non–Plug and Play drivers, on the other hand, usually create device objects when the I/O manager invokes their initialization routine. The I/O manager unloads a driver when its last device object has been deleted and no references to the device remain. Write(…) Dispatch Table Loaded Driver Image TCP/IP Drivers Driver and Device Objects
7
File Objects Represents open instance of a device (files on a volume are virtual devices) Applications and drivers “open” devices by name The name is parsed by the Object Manager When an open succeeds the object manager creates a file object to represent the open instance of the device and a file handle in the process handle table A file object links to the device object of the “device” which is opened File objects store additional information File offset for sequential access File open characteristics (e.g. delete-on-close) File name Accesses granted for convenience
8
I/O Request Packets System services and drivers allocate I/O request packets to describe I/O A request packet contains: File object at which I/O is directed I/O characteristics (e.g. synchronous, non-buffered) Byte offset Length Buffer location The I/O Manager locates the driver to which to hand the IRP by following the links: The I/O request packet (IRP) is where the I/O system stores information it needs to process an I/O request. When a thread calls an I/O service, the I/O manager constructs an IRP to represent the operation as it progresses through the I/O system. If possible, the I/O manager allocates IRPs from one of two per-processor IRP nonpaged look-aside lists: the small-IRP look-aside list stores IRPs with one stack location, and the large-IRP look aside lists contains IRPs with eight stack locations. If an IRP requires more than eight stack locations, the I/O manager allocates IRPs from nonpaged pool. After allocation and initializing an IRP, the I/O manager stores a pointer to the caller‘s file object in the IRP. File Object Device Object Driver Object
9
Putting it Together: Request Flow
Process DeviceIoControl User Mode Kernel Mode Dispatch Table NtDeviceIoControlFile File Object Device Object Driver Object Handle Table IRP DispatchDeviceControl( DeviceObject, Irp ) Driver Code
10
Environment subsystem or DLL
I/O Request Packet Environment subsystem or DLL An application writes a file to the printer, passing a handle to the file object User mode Kernel mode Services 2)The I/O manager creates an IRP and initializes first stack location I/O manager IRP stack location IRP header WRITE parameters File object Device object Driver object 3)The I/O manager uses the driver object to locate the WRITE dispatch routine and calls it, passing the IRP Handling a synchronous I/O to a single-layered driver consists of seven steps: 1. The I/O request passes through a subsystem DLL. 2. The subsystem DLL calls the I/O manager’s read or write service. 3. The I/O manager allocates an IRP describing the request and sends it to the driver (a device driver in this case) by calling its own IoCallDriver function. 4. The driver transfers the data in the IRP to the device and starts the I/O operation. 5. The driver signals I/O completion by interrupting the CPU. 6. When the device completes the operation and interrupts the CPU, the device driver services the interrupt. 7. The driver calls the I/O manager’s IoCompleteRequest function to inform it that it has finished processing the IRP’s request, and the I/O manager completes the I/O request. Dispatch routine(s) Start I/O ISR DPC routine Device Driver
11
IRP data IRP consists of two parts: Fixed portion (header):
Type and size of the request Whether request is synchronous or asynchronous Pointer to buffer for buffered I/O State information (changes with progress of the request) One or more stack locations: Function code Function-specific parameters Pointer to caller‘s file object While active, IRPs are stored in a thread-specific queue I/O system may free any outstanding IRPs if thread terminates
12
I/O Processing – synch. I/O to a single-layered driver
The I/O request passes through a subsystem DLL The subsystem DLL calls the I/O manager‘s NtWriteFile() service I/O manager sends the request in form of an IRP to the driver (a device driver) The driver starts the I/O operation When the device completes the operation and interrupts the CPU, the device driver services the int. The I/O manager completes the I/O request
13
Completing an I/O request
Servicing an interrupt: ISR schedules Deferred Procedure Call (DPC); dismisses int. DPC routine starts next I/O request and completes interrupt servicing May call completion routine of higher-level driver I/O completion: Record the outcome of the operation in an I/O status block Return data to the calling thread – by queuing a kernel-mode Asynchronous Procedure Call (APC) APC executes in context of calling thread; copies data; frees IRP; sets calling thread to signaled state I/O is now considered complete; waiting threads are released
14
Flow of Interrupts Peripheral Device Controller Interrupt Object
CPU Interrupt Service Table 2 3 n Peripheral Device Controller ISR Address Spin Lock Dispatch Code Interrupt Object CPU Interrupt Controller Raise IRQL Lower IRQL KiInterruptDispatch Grab Spinlock Drop Spinlock Read from device Acknowledge-Interrupt Request DPC Driver ISR After an I/O device completes a data transfer, it interrupts for service and the Windows kernel, I/O manager, and device driver are called into action. When a device interrupt occurs, the processor transfers control to the kernel trap handler, which indexes into its interrupt dispatch table to locate the ISR for the device. ISRs in Windows typically handle device interrupts in two steps. When an ISR is first invoked, it usually remains at device IRQL only long enough to capture the device status and then stop the device’s interrupt. It then queues a DPC and exits, dismissing the interrupt. Later, when the DPC routine is called, the device finishes processing the interrupt. When that’s done, the device calls the I/O manager to complete the I/O and dispose of the IRP. It might also start the next I/O request that is waiting in the device queue.
15
Servicing an Interrupt: Deferred Procedure Calls (DPCs)
Used to defer processing from higher (device) interrupt level to a lower (dispatch) level Also used for quantum end and timer expiration Driver (usually ISR) queues request One queue per CPU. DPCs are normally queued to the current processor, but can be targeted to other CPUs Executes specified procedure at dispatch IRQL (or “dispatch level”, also “DPC level”) when all higher-IRQL work (interrupts) completed Maximum times recommended: ISR: 10 usec, DPC: 25 usec See The advantage of using a DPC to perform most of the device servicing is that any blocked interrupt whose priority lies between the device IRQL and the DPC/dispatch IRQL is allowed to occur before the lower-priority DPC processing occurs. Intermediate-level interrupts are thus serviced more promptly than they otherwise would be. queue head DPC object DPC object DPC object
16
Interrupt dispatch table
Delivering a DPC 1. Timer expires, kernel queues DPC that will release all waiting threads Kernel requests SW int. DPC routines can‘t assume what process address space is currently mapped DPC Interrupt dispatch table high Power failure 2. DPC interrupt occurs when IRQL drops below dispatch/DPC level 3. After DPC interrupt, control transfers to thread dispatcher DPC DPC DPC Dispatch/DPC dispatcher DPC queue APC Low DPC routines can call kernel functions but can‘t call system services, generate page faults, or create or wait on objects 4. Dispatcher executes each DPC routine in DPC queue
17
I/O Completion: Asynchronous Procedure Calls (APCs)
Execute code in context of a particular user thread APC routines can acquire resources (objects), incur page faults, call system services APC queue is thread-specific User mode & kernel mode APCs Permission required for user mode APCs Executive uses APCs to complete work in thread space Wait for asynchronous I/O operation Emulate delivery of POSIX signals Make threads suspend/terminate itself (env. subsystems) APCs are delivered when thread is in alertable wait state WaitForMultipleObjectsEx(), SleepEx()
18
Asynchronous Procedure Calls (APCs)
Special kernel APCs Run in kernel mode, at IRQL 1 Always deliverable unless thread is already at IRQL 1 or above Used for I/O completion reporting from “arbitrary thread context” Kernel-mode interface is linkable, but not documented “Ordinary” kernel APCs Always deliverable if at IRQL 0, unless explicitly disabled (disable with KeEnterCriticalRegion) User mode APCs Used for I/O completion callback routines (see ReadFileEx, WriteFileEx); also, QueueUserApc Only deliverable when thread is in “alertable wait” Asynchronous procedure calls (APCs) provide a way for user programs and system code to execute in the context of a particular user thread (and hence a particular process address space). Because APCs are queued to execute in the context of a particular thread and run at an IRQL less than DPC/dispatch level, they don’t operate under the same restrictions as a DPC. An APC routine can acquire resources (objects), wait for object handles, incur page faults, and call system services. APCs are described by a kernel control object, called an APC object. APCs waiting to execute reside in a kernel-managed APC queue. Unlike the DPC queue, which is systemwide, the APC queue is thread-specific—each thread has its own APC queue. When asked to queue an APC, the kernel inserts it into the queue belonging to the thread that will execute the APC routine. The kernel, in turn, requests a software interrupt at APC level, and when the thread eventually begins running, it executes the APC. There are two kinds of APCs: kernel mode and user mode. Kernel-mode APCs don’t require “permission” from a target thread to run in that thread’s context, while user-mode APCs do. Kernel-mode APCs interrupt a thread and execute a procedure without the thread’s intervention or consent. Thread Object K APC objects U
19
Driver Layering and Filtering
Process To divide functionality across drivers, provide added value, etc. Only the lowest layer talks to the I/O hardware “Filter drivers” attach their devices to other devices They see all requests first and can manipulate them Example filter drivers: File system filter driver Bus filter driver User Mode Kernel Mode System Services File System Driver I/O Manager Volume Manager Driver A filter driver that layers over a file system driver is called a file system filter driver. The ability to see all file system requests and optionally modify or complete them enables a range of applications, including remote file replication services, file encryption, efficient backup, and licensing. Every commercial on-access virus scanner includes a file system filter driver that intercepts IRPs that deliver IRP_MJ_CREATE commands that issue whenever an application opens a file. Before propagating the IRP to the file system driver to which the command is directed, the virus scanner examines the file being opened to ensure it’s clean of a virus. If the file is clean, the virus scanner passes the IRP on, but if the file is infected the virus scanner communicates with its associated Windows service process to quarantine or clean the file. If the file can’t be cleaned, the driver fails the IRP (typically with an access-denied error) so that the virus cannot become active. IRP Disk Driver
20
Driver Filtering: Volume Shadow Copy
New to XP/Server 2003 Addresses the “backup open files” problem Volumes can be “snapshotted” Allows “hot backup” (including open files) Applications can tie in with mechanism to ensure consistent snapshots Database servers flush transactions Windows components such as the Registry flush data files Different snapshot providers can implement different snapshot mechanisms: Copy-on-write Mirroring Volsnap is the built-in provider: Built into Windows XP/Server 2003 Implements copy-on-write snapshots Saves volume changes in files on the volume Uses defrag API to determine where the file is and where paging file is to avoid tracking their changes A limitation of many backup utilities relates to open files. If an application has a file open for exclusive access, a backup utility can’t gain access to the file’s contents. Even if the backup utility can access an open file, the utility runs the risk of creating an inconsistent backup. Consider an application that updates a file at its beginning and then at its end. A backup utility saving the file during this operation might record an image of the file that reflects the start of the file before the application’s modification and the end after the modification. If the file is later restored the application might deem the entire file corrupt because it might be prepared to handle the case where the beginning has been modified and not the end, but not vice versa. These two problems illustrate why most backup utilities skip open files altogether. Windows XP introduces the Volume Shadow Copy service (\Windows\System32\Vssvc.exe), which allows the built-in backup utility to record consistent views of all files, including open ones. The Volume Shadow Copy service acts as the command center of an extensible backup core that enables independent software vendors (ISVs) to plug-in “writers” and providers. A writer is a software component that enables shadow-copy-aware application to receive freeze and thaw notifications to ensure that backup copies of their data files are internally consistent, whereas providers allow ISVs with unique storage schemes to integrate with the shadow copy service. For instance, an ISV with mirrored storage devices might define a shadow copy as the frozen half of a split mirrored volume.
21
Volume Shadow Copy Driver
Volume Snapshots Writers Backup Application Oracle 5. Backup application saves data from volume Shadow copies 2. Writers told to freeze activity Backup application requests shadow copy Volume Shadow Copy Service SQL 4. Writers told to resume (“thaw”) activity 3. Providers asked to create volume shadow copies The Microsoft Shadow Copy Provider (\Windows\System32\Drivers\Volsnap.sys) is a provider that ships with Windows to implement software-based snapshots of volumes. It is a type of driver called a storage filter driver that layers between file system drivers and volume drivers (the drivers that present views of the disk sectors that represent a logical drive) so that it can see the I/O directed at a volume. When the backup utility starts a backup operation, it directs the Microsoft Shadow Copy Provider driver to create a volume shadow copy for the volumes that include files and directories being recorded. The driver freezes I/O to the volumes in question and creates a shadow volume for each. For example, if a volume’s name in the Object Manager namespace is \Device\HarddiskVolume0, the shadow volume would have a name like \Device\HarddiskVolumeShadowCopyN, where N is a unique ID. Volume Shadow Copy Driver (volsnap.sys) Mirror provider Providers
22
Volsnap.sys Snapshot Backup Application Application Shadow Volume C:
File System Driver Volsnap.sys This figure demonstrates the behavior of applications accessing a volume and a backup application accessing the volume’s shadow volume. When an application writes to a sector after the snapshot time the Volsnap driver makes a backup copy, like it has for sectors a, b, and c of volume C: in the figure. Subsequently, when an application reads from sector c, Volsnap directs the read to volume C:, but when an the backup application reads from sector c, Volsnap reads the sector from the snapshot. When a read occurs for any unmodified sector, such as d, Volsnap routes the read to the volume. Backup read of sector c Application read of sector c a b c All reads of sector d Snapshot a d … b c
23
Shadow Copies of Shared Folders
When enabled, Server 2003 uses shadow copy to periodically create snapshots of volumes Schedule and space used is configurable
24
Shadow Copies on Shared Folders
Shadow copies are only exposed as network shares Clients may install an Explorer extension that integrates with the file server and let’s them View the state of folders and files within a snapshot Rollback individual folders and files to a snapshot
25
The PnP Manager In NT 4.0 each device driver is responsible for enumerating all supported busses in search of devices they support As of Windows 2000, the PnP Manager has bus drivers enumerate their busses and inform it of present devices If the device driver for a device not already present on the system, the PnP Manager in the kernel informs the user-mode PnP Manager to start the Hardware Wizard
26
The PnP Manager Once a device driver is located, the PnP Manager determines if the driver is signed If the driver is not signed, the system’s driver signing policy determines whether or not the driver is installed After loading a driver, the PnP Manager calls the driver’s AddDevice entry point The driver informs the PnP Manager of the device’s resource requirements The PnP Manager reconfigures other devices to accommodate the new device
27
The PnP Manager Enumeration is recursive, and directed by bus drivers
Bus drivers identify device on a bus As busses and devices are registered, a device tree is constructed, and filled in with devices Root ACPI PCI USB Video Disk Key- board Battery Device Tree As the bus drivers report detected devices to the PnP manager, the PnP manager creates an internal tree called the device tree that represents the relationships between devices. Nodes in the tree are called devnodes, and a devnode contains information about the device objects that represent the device as well as other Plug and Play–related information stored in the devnode by the PnP manager.
28
Resource Arbitration Devices require system hardware resources to function (e.g. IRQs, I/O ports) The PnP Manager keeps track of hardware resource assignments If a device requires a resource that’s already been assigned, the PnP Manager tries to reassign resources in order to accommodate Example: Device 1 can use IRQ 5 or IRQ 6 PnP Manager assigns it IRQ 5 Device 2 can only use IRQ 5 PnP Manager reassigns Device 1 IRQ 6 PnP Manager assigns Device 2 IRQ 5
29
Plug and Play (PnP) State Transitions
PnP manager recognizes hardware, allocates resources, loads driver, notifies about config. changes Query-remove command Not started Remove command Start-device command Pending remove Started Removed Start-device command Query-stop command Remove command Surprise remove The PnP manager is the primary component involved in supporting the ability of Windows to recognize and adapt to changing hardware configurations. The PnP manager automatically recognizes installed devices, a process that includes enumerating devices attached to the system during a boot and detecting the addition and removal of devices as the system executes. Hardware resource allocation is a role the PnP manager fills by gathering the hardware resource requirements (interrupts, I/O memory, I/O registers, or bus-specific resources) of the devices attached to a system and, in a process called resource arbitration, optimally assigning resources so that each device meets the requirements necessary for its operation. Because hardware devices can be added to the system after boot-time resource assignment, the PnP manager must also be able to reassign resources to accommodate the needs of dynamically added devices. Loading appropriate drivers is another responsibility of the PnP manager. The PnP manager determines, based on the identification of a device, whether a driver capable of managing the device is installed on the system, and if one is, instructs the I/O manager to load it. If a suitable driver isn’t installed, the kernel-mode PnP manager communicates with the user-mode PnP manager to install the device, possibly requesting the user’s assistance in locating a suitable set of drivers. The PnP manager also implements application and driver mechanisms for the detection of hardware configuration changes. Applications or drivers sometimes require a specific hardware device to function, so Windows includes a means for them to request notification of the presence, addition, or removal of devices. Pending stop Surprise-remove command Stop command Stopped Device Plug and Play state transitions
30
The Power Manager A system must have an ACPI-compliant BIOS for full compatibility (APM gives limited power support) A number of factors guide the Power Manager’s decision to change power state: System activity level System battery level Shutdown, hibernate, or sleep requests from applications User actions, such as pressing the power button Control Panel power settings The system can go into low power modes, but it requires the cooperation of every device driver - applications can provide their input as well
31
The Power Manager There are different system power states:
On Everything is fully on Standby Intermediate states Lower standby states must consume less power than higher ones Hibernating Save memory to disk in a file called hiberfil.sys in the root directory of the system volume Off All devices are off Device drivers manage their own power level Only a driver knows the capabilities of their device Some devices only have “on” and “off”, others have intermediate states Drivers can control their own power independently of system power Display can dim, disk spin down, etc.
32
Power Manager based on the Advanced Configuration and Power Interface (ACPI) State Power Consumption Software Resumption HW Latency S0 (fully on) Maximum Not applicable None S1 (sleeping) Less than S0, more than S2 System resumes where it left off (returns to S0) Less than 2 sec. S2 (sleeping) Less than S1, more than S3 2 or more sec. S3 (sleeping) Less than S2, processor is off Same as S2 S4 (sleeping) Trickle current to power button and wake circuitry System restarts from hibernate file and resumes where it left off (returns to S0) Long and undefined S5 (fully off) Trickle current to power button System boot System Power-State Definitions
33
Troubleshooting I/O Activity
Filemon can be a great help to understand and troubleshooting I/O problems Two basic techniques: Go to end of log and look backwards to where problem occurred or is evident and focused on the last things done Compare a good log with a bad log Often comparing the I/O activity of a failing process with one that works may point to the problem Have to first massage log file to remove data that differs run to run Delete first 3 columns (they are always different: line #, time, process id) Easy to do with Excel by deleting columns Then compare with FC (built in tool) or Windiff (Resource Kit)
34
Filemon # - operation number Process: image name + process id
Request: internal I/O request code Result: return code from I/O operation Other: flags passed on I/O request
35
Using Filemon Start/stop logging (Control/E) Clear display (Control/X)
Open Explorer window to folder containing file: Double click on a line does this Find – finds text within window Save to log file Advanced mode Network option
36
What Filemon Monitors By default Filemon traces all file I/O to:
Local non-removable media Network shares Stores all output in listview Can exhaust virtual memory in long runs You can limit captured data with history depth You can limit what is monitored: What volumes to watch in Volumes menu What paths and processes to watch in Filter dialog What operations to watch in Filter dialog (reads, writes, successes and errors)
37
Filemon Filtering and Highlighting
Include and exclude filters are substring matches against the process and path columns Exclude overrides include filter Be careful that you don’t exclude potentially useful data Capture everything and save the log Then apply filters (you can always reload the log) Highlight matches all columns
38
Basic vs Advanced Mode Basic mode massages output to be sysadmin-friendly and target common troubleshooting Things you don’t see in Basic mode: Raw I/O request names Various internal file system operations Activity in the System process Page file I/O Filemon file system activity
39
Understanding Disk Activity
Use Filemon to see why you’re hard disk is crunching Process performance counters show I/O activity, but not to where System performance counters show which disks are being hit, but not which files or which process Filemon pinpoints which file(s) are being accessed, by whom, and how frequently You can also use Filemon on a server to determine which file(s) were being accessed most frequently Import into Excel and make a pie chart by file name or operation type Move heavy-access files to a different disk on a different controller
40
Polling and File Change Notification
Many applications respond to file and directory changes A poorly written application will “poll” for changes A well-written application will request notification by the system of changes Polling for changes causes performance degradation Context switches including TLB flush Cache invalidation Physical memory usage CPU usage Alternative: file change notification When you run Filemon on an idle system you should only see bursty system background activity Polling is visible as periodic accesses to the same files and directories File change notification is visible as directory queries that have no result
41
Example: Word Crash While typing in the document Word XP would intermittently close without any error message To troubleshoot ran Filemon on user’s system Set the history depth to 10,000 Asked user to send Filemon log when Word exited
42
Solution: Word Crash Working backwards, the first “strange” or unexplainable behavior are the constant reads past end of file to MSSP3ES.LEX User looked up what .LEX file was Related to Word proofing tools Uninstalled and reinstalled proofing tools & problem went away
43
Example: Useless Excel Error Message
Excel reports an error “Unable to read file" when starting
44
Solution: Useless Excel Error Message
Filemon trace shows Excel reading file in XLStart folder All Office apps autoload files in their start folders Should have reported: Name and location of file Reason why it didn’t like it
45
Further Reading Mark E. Russinovich and David A. Solomon, Microsoft Windows Internals, 4th Edition, Microsoft Press, 2004. I/O Processing (from pp. 561) The Plug and Play (PnP) Manager (from pp. 590) The Power Manager (from pp. 607) Troubleshooting File System Problems (from pp. 711)
46
Source Code References
Windows Research Kernel sources \base\ntos\io – I/O Manager \base\ntos\inc\io.h – additional structure/type definitions \base\ntos\verifer – Driver Verifier \base\ntos\inc\verifier.h – additional structure/type definitions
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.