 Demand Technology Software, Inc. 32-bit Virtual Memory Constraints in Windows: an Update Mark Friedman Demand Technology 1020 Eighth Avenue South, Suite 6, Naples, FL USA phone: (239) fax: (239)

 Demand Technology, Inc. 32-bit VM Constraints 2 Outline Topics  Virtual Memory concepts  Virtual Memory constraints in 32-bit Windows  /3 GB option  Physical Address Extension (PAE)  Address Windowing Extensions (AWE)  64-bit relief

Intel x86

 Demand Technology, Inc. 32-bit VM Constraints 4 Intel x86 in PAE mode

 Demand Technology, Inc. 32-bit VM Constraints 5 Virtual Memory Manager 4 GB virtual per process address space  Lower 2 GB - Private Page 0 reserved Code pages Heap  Upper 2 GB - System System code Shared dlls System cache

 Demand Technology, Inc. 32-bit VM Constraints 6 Virtual Memory Manager  All address spaces share the virtual addresses from the same upper 2 GB of the System area  All virtual addresses in the lower 2 GB User area are unique  Shared memory in the System range can be used for IPC

 Demand Technology, Inc. 32-bit VM Constraints 7 Virtual Memory map Paged  Memory mapped files  Shared DLLs  File Server  IIS (html, jpg, gif)  Page tables; kernel thread stacks nonPaged:  I/O buffers used by device drivers  TCP Session data

 Demand Technology, Inc. 32-bit VM Constraints 8 Virtual Memory Manager Commit Limit  A hard limit on the number of Virtual Memory pages that the system will allocate  Sizeof (RAM) + paging file(s)  Memory allocations start to fail as % Committed Bytes in Use  100%  For smaller systems (< 2 GB RAM), Committed Bytes tends to become exhausted prior to running out of VM

 Demand Technology, Inc. 32-bit VM Constraints 9 Monitoring Committed Bytes

 Demand Technology, Inc. 32-bit VM Constraints 10 Monitoring Process\(*)Virtual Bytes

 Demand Technology, Inc. 32-bit VM Constraints 11 Virtual Memory Manager  Commit Limit But page files are extendible! Should you or shouldn’t you?

 Demand Technology, Inc. 32-bit VM Constraints 12 Paging file extension

 Demand Technology, Inc. 32-bit VM Constraints 13 System memory pools No hard limits on the sizes of the system pools One reserved area of virtual memory can fill up faster than the others; AreaMax Size file cache960 MB PTEs660 MB Paged pool512 MB nonPaged Pools 256 MB

 Demand Technology, Inc. 32-bit VM Constraints 14 Monitoring the System Pools

 Demand Technology, Inc. 32-bit VM Constraints 15 Virtual Memory map Virtual memory map is dynamic, subject to adjustment by the OS One reserved area of virtual memory can fill up faster than the others When any one area fills, VMM routines can empty the file cache to acquire more vm for the paged/nonpaged pools Paged Pool nonPaged Pool PTEs Cache

 Demand Technology, Inc. 32-bit VM Constraints 16 Default Paged and Nonpaged pool size can be overridden:

 Demand Technology, Inc. 32-bit VM Constraints 17 Default Paged and Nonpaged pool sizes can be overridden: Dynamic adjustment of system virtual memory when one pool is exhausted. Workloads most vulnerable:  Terminal server with an excess of processes & threads  Servers booted with the /3 GB option NonPagedPoolSize, NonPagedPoolSize, and SystemPages can be set explicitly.  Max pool size is only available from the debugger!

 Demand Technology, Inc. 32-bit VM Constraints 19 Extended Virtual Addressing Boot.ini /3 GB switch  Virtual storage constraint relief for some applications  Squeezes the System VM into 1 GB Physical Address Extension (PAE)  Supports 37-bit real addresses on Xeon processors Address Windowing Extensions (AWE)  Permits processes to address real memory > 4 GB

 Demand Technology, Inc. 32-bit VM Constraints 20 Virtual Memory Manager Boot.ini /3 GB switch  Virtual storage constraint relief for some server applications SQL Server Exchange 2000 Etc.  /userva=SizeInMB subparameter, where SizeinMB can be any value between 2048 and 3072

 Demand Technology, Inc. 32-bit VM Constraints 21 Exchange 2003 memory tuning Exchange Server 2003 recommendations /3 GB /Userva = 3030 SystemPages = -1 HeapDeCommitFreeBlockThreshold = 0x msExchESEParamCacheSizeMax = (1.2 GB) See KB #815372

 Demand Technology, Inc. 32-bit VM Constraints 22 Virtual Memory constraints Try the /3 GB switch  1 GB system area is not enough virtual memory for some applications Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool – Where Session data from TCP connections is stored  Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e.g., SQL Server, MS Exchange database – store.exe) using Microsoft Cluster Server (MCS) During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time

 Demand Technology, Inc. 32-bit VM Constraints 23 Physical Address Extension PAE is supported on most recent Intel servers  Physical addresses are 37-bits  Up to 128 GB of RAM can be installed New format Page Table Entries (PTEs) are 64-bits Process address spaces are still limited to 4 GB. How can they exploit 64 GB of RAM?  Expand sideways, using multiple address spaces  Large Memory Enabled (LME) device drivers  Address Windowing Extensions (AWE) Boot.ini /pae switch  Limitation: system addresses can be no higher than 16 GB

 Demand Technology, Inc. 32-bit VM Constraints 24 Intel x86 in PAE mode

 Demand Technology, Inc. 32-bit VM Constraints 25 Physical Address Extension Expand sideways, using multiple address spaces  e.g., MS SQL Server

 Demand Technology, Inc. 32-bit VM Constraints 26 IIS 6.0 (expand sideways)

 Demand Technology, Inc. 32-bit VM Constraints 27 Configuring IIS 6.0 Application Pools

 Demand Technology, Inc. 32-bit VM Constraints 28 Address Windowing Extensions (AWE) API that allows processes to address real memory locations outside their 4 GB virtual addressing range  Create and manage memory Overlays  Used in conjunction with PAE AllocateUserPhysicalPages MapUserPhysicalPages Virtual Alloc

 Demand Technology, Inc. 32-bit VM Constraints 29 AWE Note: Frequent unmapping and remapping of Physical Memory blocks is expensive!

 Demand Technology, Inc. 32-bit VM Constraints 30 AWE support MS SQL Server 2000: /3 GB, PAE, & AWE  Multiple process instances  awe enabled = 1  AWE activity counters in the Buffer Manager object Oracle: /3 GB, PAE, & AWE  AWE_WINDOW_MEMORY  Also Set max server Oraclememory SAS: /3 GB, PAE  Work library can be placed in PAE memory

 Demand Technology, Inc. 32-bit VM Constraints 31 System Virtual Memory tuning HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management settings NonpagedPoolSizeDefaults based on the size of RAM, up to 256 MB. Can be set explicitly. -1 extends NonpagedPoolSize to its maximum PagedPoolSizeDefaults based on the size of RAM, Up to 650 MB. Can be set explicitly; -1 extends PagedPoolSize to its maximum SystemPagesDefaults based on the size of RAM.Can be set explicitly; -1 extends SystemPages to its maximum LargeSystemCache960 MBZero value allocates minimum sized file cache

 Demand Technology, Inc. 32-bit VM Constraints 32 Default Paged and Nonpaged pool size can be overridden:

 Demand Technology, Inc. 32-bit VM Constraints 33 Default Paged and Nonpaged pool size can be overridden (Win 2003): System Pages = x’ffff ffff’ 16 or -1  Maximizes the number of PTEs that can be built  Potentially useful with /PAE or Terminal Services PagedPoolSize = x’ffff ffff’ 16 or -1  Allows the OS maximum flexibility to determine the size of the Paged Pool  Or PagedPoolSize and NonPagedPoolSize can be set explicitly LargeSystemCache controls the maximum virtual address range of the file cache  512 MB – 1 GB

 Demand Technology, Inc. 32-bit VM Constraints 34 !vm Debugger command runs the Page Frame Number (PFN) database Available Pages: 4920 ( Kb) ResAvail Pages: 358 ( 1432 Kb) Locked IO Pages: 251 ( 1004 Kb) Free System PTEs: ( Kb) Free NP PTEs: ( Kb) Free Special NP: 0 ( 0 Kb) Modified Pages: 596 ( 2384 Kb) Modified PF Pages: 660 ( 2640 Kb) NonPagedPool Usage: 2750 ( Kb) NonPagedPool Max: ( Kb) PagedPool 0 Usage: 3544 ( Kb) PagedPool 1 Usage: 1359 ( 5436 Kb) PagedPool 2 Usage: 1340 ( 5360 Kb) PagedPool Usage: 6243 ( Kb) PagedPool Maximum: ( Kb) Shared Commit: 6842 ( Kb) Special Pool: 0 ( 0 Kb) Shared Process: 3688 ( Kb) PagedPool Commit: 6398 ( Kb) Driver Commit: 1630 ( 6520 Kb) Committed pages: ( Kb) Commit limit: ( Kb)

 Demand Technology, Inc. 32-bit VM Constraints 35 !poolused Debugger command accounts for all system pool allocations (plus pooltags.txt documentation) lkd> !poolused 2 Sorting by NonPaged Pool Consumed Pool Used: NonPaged Paged Tag Allocs Used Allocs Used LSwi NV File MmCm LSwr Devi Thre PcNw Irp

 Demand Technology, Inc. 32-bit VM Constraints 36 Poolmon utility (from DDK)

 Demand Technology, Inc. 32-bit VM Constraints bit Virtual Memory Relief

 Demand Technology, Inc. 32-bit VM Constraints bit Relief Windows supports two flavors of 64-bit hardware  IA-64 (Itanium)  AMD64 and Intel x64 (aka EMT) Runs 32-bit applications with almost no performance penalty 32-bit applications can allocate their entire 4 GB virtual memory range

 Demand Technology, Inc. 32-bit VM Constraints 39 Questions ?