1. SQL Server Memory Internals 김 종구 고객기술지원부 한국마이크로소프트

2. 목 차 몸풀기 – Windows Memory Management Basic : Virtual Memory & AWE 실 전 – SQL Server Memory Internals 마무리 – SQL PASS 2003 PSS Service Center Labs : Troubleshooting Memory

3. 몸풀기 Windows Memory Management Basic : Virtual Memory & AWE

4. 사전지식 Inside Microsoft Windows 2000, 3rd Ed. Chap.7 Memory Management (esp. pp. 389-402) Programming Application for Microsoft Windows, 4 th Ed. Chap. 13 Windows Memory Architecture

6. 메모리 공간 (Address Space) Physical Memory 공간 - 하드웨어에 의해 지원됨 - 0 ~ MAXsys 까지의 주소 Logical Memory 공간 - Process 가 보는 자신의 주소 - 0 ~ MAXprog 까지의 주소 Program 은 Logical Address 만 볼 수 있음. - Physical Address 는 볼 수 없음.

7. 동적 Program 주소변경 Program 은 실행 시 주소가 동적으로 변경

8. SWAPPING Process 는 Memory 에서 저장장소 (Backing Storage) 로 잠시 swap 되었다가 실행 시 다시 메모리로 Load 될 수 있음.

9. Virtual Memory(1) User Logical Memory 를 Physical Memory 와 분리 - 실행 시 program 의 일부만이 메모리에 있다. - 따라서 Logical Address Space 는 Physical Address Space 보다 훨씬 클 수 있다. - Page 가 SWAP 될 수 있어야 한다.

10. Virtual Memory (2) 사용자에게 Memory 의 Physical 한 측면을 모두 감춤. - Memory 는 2 n Byte 만큼의 Virtual Address Space 를 가짐. - Virtual Address Space 의 일부분만이 Physical Memory 에 존재함.

11. Paging 원리 - Process 의 Physical Address Space 는 연속적이지 않을 수 있음. - Physical Memory 를 4K 단위의 frame 으로 나눔. - Logical Memory 를 4K 단위의 page 로 나눔. - n page 의 Program 을 수행하려면, n 개의 frame 이 필요함. - 모든 free frame 에 대한 추적이 필요함. - page 와 frame 이 1:1 로 mapping 됨. - page 는 logical address 에서 연속적이나 physical address 의 frame 은 임의의 위치에 존재함.

12. Page 와 frame 의 mapping

13. Virtual Address Translation Page 테이블 통해 virtual page 와 physical frame 을 mapping 함.

14. Page Fault Memory 상에 없는 Page 를 참조할 때 발생

16. Heaps(1) Process 가 생성 시 1MB 가 기본적으로 생성됨. 여러 개의 작은 데이터 block 을 할당하는데 유용함. ( 예 - linked list, tree 관리 ) Allocation granularity(64KB) 및 page boundary(4KB) 를 무시하고 필요한 만큼만 메모리를 할당함. 하지만 할당 및 해제가 느린 단점이 있음. Physical storage 에 대한 commit 과 decommit 에 대한 제어를 잃어버림.

17. Heap(2) A Process’s Default Heap - Windows Function 이 이 공간을 사용함 예 ) application 의 ANSI 문자열을 UNICODE 문자열로 변환 시 변환 문자 저장 Thread 들의 Heap allocation 은 serialize 됨 - 한번에 하나의 thread 만이 Heap memory 를 allocation 할 수 있음. - corruption 방지 추가적인 Heap 생성이 가능 (HeapCreate)

18. Shared Memory Process 들과 OS 사이에 메모리를 공유하는 것 예 ) 두 개의 process 가 동일한 DLL 을 공유하는 경우

19. Address Windowing Extensions (AWE) 32bit process 에게 제한된 주소 공간 (2G /3G) 을 초과하는 physical memory 를 할당하고 접근할 수 있도록 하는 방법 3 단계를 통해 AWE 를 이용 - 사용할 physical memory 할당 - physical memory 를 접근할 수 있는 window 역할을 할 virtual address space 지역 생성 - 생성된 window 를 통해 physical memory 접근

20. Address Windowing Extensions (AWE)

21. 실 전 SQL Server Memory Internals - 이 세션은 PASS 2003 에서 SQL Server Escalation Engineer 인 Ken Henderson 에 의해 발표된 내용입니다.

24. 329/SQL Server Memory Internals Ken Henderson/Author Keith Elmore/Microsoft* 2003 PASS Community Summit November 11-14, 2003 *This presentation is based on internal training developed by Keith Elmore of Microsoft

25. Goals Describe the SQL Server memory management architecture Provide a brief overview of the implementation of this architecture. Teach you how to figure out how to solve memory-related problems

26. Agenda Windows memory management review Memory usage and SQL Server processes Memory usage by non-SQL Server processes Monitoring memory usage Troubleshooting memory-related errors

27. Memory Management Review

28. Virtual vs. physical memory Windows provides virtual memory services Committed virtual memory is backed by some type of physical storage – usually the page file or physical memory Windows handles translations between virtual memory and physical storage 32-bit processes have a 4GB virtual memory address space divided between kernel- and user-accessible areas

29. Address Space Application Operating System 0xFFFFFFFF 0x00000000 0x80000000 0xC0000000 Application or OS (/3GB or /USERVA switch)

30. Reserved vs. Committed Free – not used Reserved – address space set aside for later use; no physical storage used Committed – physical storage is being used to store something

31. Allocation Granularity Determines the smallest memory reservation Depends on the operating system Is 64KB on 32-bit Windows

32. System Page Size Determines the smallest memory allocation Depends on the processor architecture Is 4KB on I386

33. xp_sysinfo Custom xproc from my last book Uses Win32 GetSystemInfo API to retrieve useful system info Returns the system page size, allocation granularity, etc. exec xp_sysinfo PageSize AllocationGranularity NumberOfProcessors ProcessorType -------- --------------------- ------------------ ------------- 4096 65536 2 Intel

34. Types of Win32 memory Virtual memory – a range of addresses that can be committed or reserved and be noncontiguous Heap – a (typically) small contiguous memory region Shared memory – can share memory between processes and allow files to be treated as memory

35. Virtual Memory VirtualAlloc -- only Win32 allocator that differentiates between reserving and committing Reserve a memory region (multiples of 64 KB) Commit some or all of a previously reserved region (multiples of 4 KB) Can reserve and commit separately or at the same time

36. Heaps HeapAlloc - used to allocate (typically) small chunks from the heap manager, which manages one or more regions that it allocated from virtual memory Each process has a default process heap (GetProcessHeap) Applications can create their own heaps (HeapCreate)

37. Shared Memory Two uses: Allows multiple processes to share memory Allows a file to be treated as memory CreateFileMapping and MapViewOfFile set it up Used by SQL Server’s shared memory NetLib to avoid the network stack

38. Other allocators C++ new operator COM IMalloc, CoTaskMemAlloc All allocators eventually resolve to VirtualAlloc

39. C++ new operator The default Microsoft C++ implementation simply calls HeapAlloc A class can overload new to do whatever it requires This operator is almost never used within SQL Server

40. IMalloc / CoTaskMemAlloc IMalloc is a standard interface for COM- based code to allocate memory CoTaskMemAlloc is semantically identical to IMalloc::Alloc SQL Server uses an interface derived from IMalloc to manage internal memory allocations

42. Review 2-3 GB memory available to application Memory can be free, reserved or committed All Win32 allocators eventually end up using Virtual* APIs to do their work

43. SQL Server Buffer Pool

44. Buffer Pool Size SQL Server calculates max buffer pool size and reserves it at startup The computation is based on physical memory Lowering max server memory does not change the amount of memory reserved To change the reservation size, increase MemToLeave (and shrink the maximum BPool size) via the -g parameter

45. Buffer Pool Allocations Up to 32 different regions can be allocated to set up the buffer pool The BPool maintains parallel arrays to track the start of each region and the number of pages in it, respectively SQL Server maintains a bitmap that tracks the committed pages in the BPool SQL Server maintains an array of structures (“BUFs”) that track allocated pages

46. Finding the BPool The BPool is exposed in a simple global variable that you can find with WinDbg Demo: Finding BPool

47. Buffer Pool & AWE Windows 2000 and later Involves mapping/unmapping a “view” of physical memory pages into the process virtual address space Limited amount of “visible” buffers at any given time (only what can be mapped into the process address space) May have to throw away procedure plans to clear space for mapping buffers

48. Review SQL Server dynamically adjusts the amount of memory it uses There is limited virtual address space left after the buffer pool reserves its maximum AWE helps some, but we’re still limited by virtual address space

49. Memory Allocation by SQL Server Processes

51. Memory Managers Connection – e.g., PSS and SRVPROC Query Plan – compiled plans/execution plans Optimizer – metadata and tree structures Utility – e.g., SQLTRACE, log manager, etc. General – e.g., locks OS (Reserved) “Manager” “No Fail” Low Memory “Manager”

52. Direct Calls to VirtualAlloc/HeapAlloc Transaction Log Backup Profiler (7.0) NetLibs

53. Callbacks to Allocate Memory srv_alloc – (typically used by xprocs) callback routine that allocates from general pool or from OS buffers, depending on size MSXML

54. Lazywriter Two purposes: Tries to keep a minimum number of free buffers (freeing dirty buffers requires I/O) Keeps enough physical memory free on the machine to avoid paging Sweeps across BUFs to “age” them Many things that it can’t remove Demo: Watching Lazywriter in action

55. Review Five core memory managers OS memory “manager” for allocations greater than 8 KB; low memory mgr for emergencies Some places still do direct VirtualAlloc srv_alloc -- callback into memory manager Lazywriter’s role in keeping memory available

56. Memory and External Components

57. Linked Servers Typically run in process Might be able to run out-of-process Even if run out-of-process, there are still some memory requirements to load the marshalling stubs

58. COM Objects Default sp_OACreate syntax creates objects in process Can force out of process even if app changes can’t be made (Q198891) Marshalling required if out of process All in process allocations use address space from MemToLeave

59. Extended Stored Procs Address space used for the DLL Memory allocations may come from buffer pool (srv_alloc = 8 KB, standard HeapAlloc or VirtualAlloc) Can be run from another server (Q243428)

60. Review External consumers of SQL Server memory: Linked servers COM objects (sp_OA) Extended stored procs The first two always use MemToLeave; the third may use buffer pool if using srv_alloc

61. Monitoring Memory Usage

62. Performance Counters Process Object Virtual Bytes – reserved virtual memory Private Bytes – committed virtual memory SQL Server:Buffer Partition Free list statistics SQL Server:Buffer Manager Stolen, Target, Total SQL Server:Memory Manager Memory manager statistics, etc.

63. DBCC MEMORYSTATUS Documented in the KB (Q271624) Same info is produced for certain errors PSS may ask you to run this in certain situations

64. vmmap, vmstat, tlist Command line tools vmmap and vmstat are variants of Richter-based internal tools that PSS may ask you to run vmmap dumps a row describing the attributes of each block of virtual memory vmstat displays summary info on the count, min, max and avg sizes of the different types tlist shows either a list of processes or details about threads/modules in a specific process

65. Review Performance Counters DBCC MEMORYSTATUS vmmap, vmstat, tlist

66. Troubleshooting

67. WARNING: Clearing procedure cache to free contiguous memory Allocation from OS Memory manager Normally out of address space in MemToLeave Start by capturing PerfMon, DBCC MEMORYSTATUS, and the errorlog May need a userdump

68. WARNING: Failed to [reserve | commit] contiguous memory Shortage of contiguous address space in MemToLeave (or physical memory on server)

69. Error 17803 Memory allocation from ODS component Issue may be in BPool or MemToLeave—need more information to troubleshoot further May require a userdump to effectively troubleshoot

70. Error 17802/Unable to spawn xxx thread Address space is exhausted or heavily fragmented What is loaded in the process? Are there leaks? May require a userdump to effectively troubleshoot

71. Error 701 -y701 is useful here to generate a stack dump when the error occurs (dbcc dumptrigger also works) Check sp_configure settings (e.g., Open Objects) Demo: -y and dumptrigger

72. Error 7399 Shortage of memory in the MemToLeave area Could be load related (OLE-DB provider is using lots of memory) and require a larger –g value May be a leak in the provider

73. Error 8645 Caused by many users running queries with sort or hash operators Goal is to find who is consuming all the workspace memory May be an indication that these queries are picking a bad plan or need some optimization

74. Further Reading Inside Windows 2000 (MS Press, 2000) by David Solomon and Mark Russinovich Programming Applications for Windows (MS Press, 1999) by Jeffrey Richter Inside SQL Server 2000 (MS Press, 2000) by Kalen Delaney The Guru’s Guide to SQL Server Architecture and Internals (Addison-Wesley, 2003) by Ken Henderson The BOL and various whitepapers on www.microsoft.com.

75. Updates Server: privatenews.microsoft.com Account name: privatenews\servicecenter Password: PASS

76. 마무리 SQL PASS 2003 PSS Service Center Labs

77. Troubleshooting memory By using 4 Memory Senarios

78. Important Resources SQL Server Books Online “Monitoring with SQL Profiler” SQL Server Books Online “Memory Architecture” KB Article 293878 “INF: General Performance Troubleshooting Using SQL Profiler Traces” KB Article 230785 “INF: SQL Server 7.0 and SQL Server 2000 Logging and Data Storage KB Article 321363 “INF: SQL Server Memory Usage” KB Article 243428 “HOW TO: Move Extended Stored Procedures Out of Process” KB Article 309256 “HOW TO: Troubleshoot SQL Server Error 8645”

79. © 2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.