Embedded System Lab. 오명훈 Memory Resource Management in VMware ESX Server Carl A. Waldspurger VMware, Inc. Palo Alto, CA USA

오 명 훈오 명 훈 Embedded System Lab. TABLE OF CONTENTS Introduction Reclamation Mechanism Sharing Memory Shares vs Working sets Dynamic Reallocation I/O Page Remapping Reference

오 명 훈오 명 훈 Embedded System Lab. Introduction ESX Server manages system hardware directly, providing significantly higher I/O performance and complete control over resource management This paper introduces several novel mechanisms and policies that ESX Server 1.5 uses to manage memory

오 명 훈오 명 훈 Embedded System Lab. Reclamation Mechanism Memory Overcommit Page replacement Issues  meta-level page replacement policy, double paging Ballooning  small balloon module is loaded into the guest OS -inflate(alloc), deflate(free), get_free_page() 40 client dbench due to VM data structures

오 명 훈오 명 훈 Embedded System Lab. Sharing Memory Eliminating redundant copies of pages Content-Based Page Sharing  hashing, COW  0.5% space overhead Scanning only during otherwise-wasted idle cycles The CPU overhead due to page sharing was negligible, 1.6% lower to 1.8% higher (increase hit rate in physically-indexed caches) single chain 16bytes

오 명 훈오 명 훈 Embedded System Lab. Shares vs Working Sets The working set of a VM is the set of memory pages which are being “actively” accessed by the VM Shared based allocation (relative resource rights) Measuring, Reclaiming Idle Memory - sampling  minor page fault count Multiple sampling rate ESX VM Shares-per page ratio Default : 0.75 Default : 100pages 30sec

오 명 훈오 명 훈 Embedded System Lab. Shares vs Working Sets 2 Zero Page Thread

오 명 훈오 명 훈 Embedded System Lab. Dynamic Reallocation Most operating systems attempt to maintain a minimum amount of free memory Prevent rapid state fluctuations

오 명 훈오 명 훈 Embedded System Lab. Dynamic Reallocation 2 Only 1GB was available for executing VMS The aggregate VM workload was configured to use a total of 1472 MB  1200MB (result of sharing) Nearly all time is spent in the high and soft states( Shared ↓  Balloon ↑ ) Windows zero min size Long-running query

오 명 훈오 명 훈 Embedded System Lab. I/O Page Remapping Support Remapping (Highmem  Lowmem)  increased latency, reduced throughput, increased CPU load specified threshold It may be desirable to remap some low pages into high memory ( simple random replacement ) Hot page HIGHMEM DMA Repeated I/O operation

오 명 훈오 명 훈 Embedded System Lab. Reference P. Barham et al., "Xen and the Art of Virtualization", SOSP, 2003 Professional Linux Kernel Architecture book