1.  Demand Technology Software, Inc. Memory Leaks Demand Technology 1020 Eighth Avenue South, Suite 6, Naples, FL 34102 phone: (941) 261-8945 fax: (941) 261-5456 e-mail: markf@demandtech.com http://www.demandtech.com

2.  Demand Technology, Inc. Memory Leaks 2 Outline Topics  Virtual Memory concepts  Virtual Memory constraints in 32-bit Windows  Leaking processes  Page faults/sec

3.  Demand Technology, Inc. Memory Leaks 3 Virtual Memory Manager 4 GB virtual per process address space  Lower 2 GB - Private Page 0 reserved Code pages Heap  Upper 2 GB - System System code Shared dlls System cache User System 0000 7fff ffff 16 ffff ffff 16

4.  Demand Technology, Inc. Memory Leaks 4 User Virtual Memory Manager  All address spaces share the virtual addresses from the same upper 2 GB of the System area  All virtual addresses in the lower 2 GB User area are unique  Shared memory in the System range can be used for IPC User System 0000 7fff ffff 16 ffff ffff 16 Shared

5.  Demand Technology, Inc. Memory Leaks 5 Virtual Memory Manager Boot.ini /3 GB switch  Virtual storage constraint relief for some applications SQL Server Exchange 2000 Etc.

6.  Demand Technology, Inc. Memory Leaks 6 AWE Windows 2000 - Address Windowing Extensions  Address RAM > 4 GB limit -Intel Pentium III, IV requires PSE36 Driver Real addresses only Not backed on paging files – PTEs remain 32-bit

7.  Demand Technology, Inc. Memory Leaks 7 Win64 Virtual Memory

8.  Demand Technology, Inc. Memory Leaks 8 Virtual Memory Manager Real Memory allocation Counters Available Bytes Pool non-paged Bytes, Pool Paged Resident Bytes, System Cache Resident Bytes, System Code Resident Bytes, System Driver Resident Bytes  Instantaneous Counters (all reported as Bytes)  Cache Bytes is actually the pageable System Working Set = Pool Paged Resident Bytes + System Cache Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes

9.  Demand Technology, Inc. Memory Leaks 9 Virtual Memory Manager Memory allocation Counters Available Bytes Available KBytes Available MBytes Pool non-paged Bytes, Pool Paged Resident Bytes, System Cache Resident Bytes, System Code Resident Bytes, System Driver Resident Bytes

10.  Demand Technology, Inc. Memory Leaks 10 What is the difference between Pool Paged Bytes and Pool Nonpaged Bytes? Which applications would use which VM pool?

11.  Demand Technology, Inc. Memory Leaks 11 Virtual Memory map

12.  Demand Technology, Inc. Memory Leaks 12 Virtual Memory map Paged  Memory mapped files  Shared DLLs  File Server  IIS (html, jpg, gif)  Page tables nonPaged:  I/O buffers used by device drivers  TCP Session data

13.  Demand Technology, Inc. Memory Leaks 13 Virtual Memory map Virtual memory map is static, created once during OS initialization When any one area is full, memory allocations to that pool will fail One reserved area of virtual memory can fill up faster than the others;

14.  Demand Technology, Inc. Memory Leaks 14 Nonpaged pool size (ref. Q126402): MinimumNonPagedPoolSize = 256K MinAdditionNonPagedPoolPerMb = 32K DefaultMaximumNonPagedPool = 1 MB MaxAdditionNonPagedPoolPerMb = 400K PTE_PER_PAGE = 1024 PAGE_SIZE=4096 NonPagedPoolSize = MinimumNonPagedPoolSize + ((Physical MB - 4) * MinAdditionNonPagedPoolPerMB) If NonPagedPoolSize > 128 MB THEN NonPagedPoolSize = 128 MB

15.  Demand Technology, Inc. Memory Leaks 15 Paged pool size: Size = (2 * MaximumNonPagedPoolSize) / PAGESIZE Size = (Size + (PTE_PER_PAGE - 1)) / PTE_PER_PAGE PagedPoolSize = Size * PAGESIZE * PTE_PER_PAGE If PagedPoolSize > 192 MB THEN PagePoolSize = 192 MB Note: PagedPoolSize  sizeof (RAM) for systems < 192 MB

16.  Demand Technology, Inc. Memory Leaks 16 Default Paged and Nonpaged pool size can be overridden:

17.  Demand Technology, Inc. Memory Leaks 17 Virtual Memory constraints Virtual Memory constraints in 32-bit Windows tend to appear when sizeof (RAM)  4 GB  2 GB private area is not enough virtual memory for some applications e.g., SQL Server, Exchange database (store.exe) Due to fragmentation, it is typically not possible to allocate all 2 GB  2 GB system area is not enough virtual memory for some applications File cache for a conventional IIS-managed web site with many static.htm, jpg, gif, etc., files to retrieve

18.  Demand Technology, Inc. Memory Leaks 18 Virtual Memory constraints Virtual Memory constraints in 32-bit Windows tend to appear when sizeof (RAM)  4 GB  Ample RAM exists, but it is not possible for your applications to access it due to virtual memory addressing limitations  Large number of Available Bytes

19.  Demand Technology, Inc. Memory Leaks 19 Virtual Memory constraints So try the /3 GB switch  1 GB system area is not enough virtual memory for some applications Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool – Where Session data from TCP connections is stored  Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e.g., SQL Server, MS Exchange database – store.exe) using Microsoft Cluster Server (MCS) During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time

20.  Demand Technology, Inc. Memory Leaks 20 Exchange 2000 memory tuning Exchange default memory allocation parameters are “self-tuning,” but may not be optimal on servers with > 1 GB RAM  Adjust HKLM\SYSTEM\CurrentControlSet\Services\SMTPSVC\Queuin g MsgHandleThreshold & MsgHandleAsyncThreshold HKLM\SYSTEM\CurrentControlSet\Services\Inetinfo\Parameters FileCacheMaxHandles

21.  Demand Technology, Inc. Memory Leaks 21 Exchange 2000 memory tuning Store DB cache  Store cache normally expands to fill RAM  But, maximum cache = 900 MB  This value can be adjusted using ADSI Edit tool: msExchESEParamCacheSizeMax msExchESEParamCacheSizeMin  Also, consider adjusting msExchESEParamLogBuffers attribute for active, back-end servers

22.  Demand Technology, Inc. Memory Leaks 22 Detecting memory leaks Processes that allocate virtual memory, but later “forget” to free it. MS says leaks won’t happen in.Net managed code due to automatic garbage collection But, meanwhile…, Where to look depends on whether process or system addresses are being allocated:  Per Process: Virtual Bytes, Private Bytes, Pool Paged Bytes, Handle Count  System level: Memory Pool Paged Bytes, Pool Nonpaged Bytes and the Objects Object

23.  Demand Technology, Inc. Memory Leaks 23 Detecting memory leaks Look for a steady increase or a sharp spike in process Virtual Bytes, or the System’s Pool Paged Bytes.  If RAM is not full, the leak may also be manifest in the Memory allocation counters and result in increased paging, if RAM fills up. For example:

24.  Demand Technology, Inc. Memory Leaks 24 Detecting memory leaks (1)

25.  Demand Technology, Inc. Memory Leaks 25 Detecting memory leaks (2)

26.  Demand Technology, Inc. Memory Leaks 26 Detecting memory leaks (2)

27.  Demand Technology, Inc. Memory Leaks 27 Detecting memory leaks (2)

28.  Demand Technology, Inc. Memory Leaks 28 Detecting memory leaks (2)

29.  Demand Technology, Inc. Memory Leaks 29 Lab exercise. Open SUPPORT.200205080300.EXCHG2K.sum. smf, SUPPORT.200205090300.EXCHG2K.sum. smf, and SUPPORT.200205100300.EXCHG2K.sum. smf and investigate virtual memory usage. Can you find the application that is leaking?

30.  Demand Technology, Inc. Memory Leaks 30 Detecting Memory Leaks (3)

31.  Demand Technology, Inc. Memory Leaks 31 Detecting Memory Leaks (3)

32.  Demand Technology, Inc. Memory Leaks 32 Detecting Memory Leaks (3)

33.  Demand Technology, Inc. Memory Leaks 33 Detecting Memory Leaks (3)

34.  Demand Technology, Inc. Memory Leaks 34 Page replacement Theoretically, Page Faults/sec  Cache faults/sec + Transition faults/sec + Demand Zero Faults/sec + Pages Read/sec But empirically…

35.  Demand Technology, Inc. Memory Leaks 35

36.  Demand Technology, Inc. Memory Leaks 36 Page replacement Empirically, Page Faults/sec  Transition faults/sec + Demand Zero Faults/sec + Pages Read/sec

37.  Demand Technology, Inc. Memory Leaks 37 Questions ?