1. Kernel Memory Chris Gill, David Ferry, Brian Kocoloski
CSE 422S - Operating Systems Organization
Washington University in St. Louis
St. Louis, MO 63130

2. Kernel vs User Virtual Memory
Kernel Virtual Memory
Physical Memory
Mappings created at runtime via memory allocation (e.g., mmap()) and page fault handling
Mappings created at system boot time
CSE 422S – Operating Systems Organization

3. Traditional Memory Mapping Strategy in Linux (on x86_64)
Virtual Memory
Mappings created at system boot time
Kernel address space
Physical Memory
CSE 422S – Operating Systems Organization

4. Traditional Memory Mapping Strategy in Linux (on x86_64)
Virtual Memory
Mappings created at system boot time
User address space
Mappings created at runtime via memory allocation (e.g., mmap()) and page fault handling
Kernel address space
Physical Memory
CSE 422S – Operating Systems Organization

5. Traditional Memory Mapping Strategy in Linux (on x86_64)
Virtual Memory
Green VA are free
Blue VA are allocated
But, both area always mapped in the kernel’s page tables
User address space
Kernel address space
Physical Memory
CSE 422S – Operating Systems Organization

6. Kernel vs User Virtual Memory
Given a physical address, how do I calculate the kernel virtual address to access it?
__va(unsigned long pa);
On arm 7 (R Pi 3): ((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
PHYS_OFFSET: first address of physical memory
PAGE_OFFSET: (On our 32-bit R Pis) 0x
The point is, reverse PA->VA translation can be done in O(1) time
(You do this in studio 12)
Given a user virtual address, how would the kernel calculate its physical address?
Given a physical address, how do I calculate the user virtual address to access it?
CSE 422S – Operating Systems Organization

7. CSE 422S – Operating Systems Organization
Memory Pages
All physical memory is divided into pages
Pages are mapped to virtual memory
Virtual addresses require translation
Virtual Address Translation:
Process A
Process B
Virtual Address
Page #
Offset
4KB
4KB
4KB
4KB
4KB
4KB
Physical Page
Phys.
Addr
CSE 422S – Operating Systems Organization

8. Dynamic Management of Kernel Memory
Page level management
Use alloc_pages() and free_pages() etc.
Object level management
Use kmalloc() and kfree() etc. (virtually and physically contiguous)
Use vmalloc() and vfree() etc. (only virtually contiguous )
Slab allocation
Use kmem_cache_create() and then kmem_cache_alloc() and kmem_cache_free() etc. for each distinct type of memory object
CSE 422S – Operating Systems Organization

9. CSE 422S – Operating Systems Organization
Vmalloc vs Kmalloc
Kmalloc: virtually and physically contiguous
Vmalloc: only virtually contiguous
What does virtually contiguous mean?
int * ar = kmalloc(sizeof(int) * 100, GFP_KERNEL)
printf(“%p”, ar);
If value of ar is X, then the virtual address range [X, X+sizeof(int)*100) stores the array contents
CSE 422S – Operating Systems Organization

10. CSE 422S – Operating Systems Organization
Vmalloc vs Kmalloc
Kmalloc: virtually and physically contiguous
Vmalloc: only virtually contiguous
What does physically contiguous mean?
int * ar = kmalloc(sizeof(int) * 100, GFP_KERNEL)
printf(“%p”, ar);
If VA of ar is X, what is the physical address (PA)?
X+Y, where Y is some known constant value (recall slides 2 through 5)
CSE 422S – Operating Systems Organization

11. CSE 422S – Operating Systems Organization
Vmalloc vs Kmalloc
kmalloc() returns virtually and physically contiguous addresses
i.e.,
Assume we want to allocate S bytes of memory
Assume X is the return value (virtual address) from a call to kmalloc(S)
Assume pa(X) = Y
Then, for all every byte x from 0 to S, PA(x) = Y+x
CSE 422S – Operating Systems Organization

12. Traditional Memory Mapping Strategy in Linux (on x86_64)
Virtual Memory
Assume green VA are free
Assume blue VA are allocated
Kernel address space
Physical Memory
Calls to kmalloc() effectively search through green
VA space to find free virtual address space
CSE 422S – Operating Systems Organization

13. CSE 422S – Operating Systems Organization
Vmalloc vs Kmalloc
vmalloc() returns only virtually contiguous addresses
i.e
Assume we want to allocate S bytes of memory
Assume X is the return value (virtual address) from a call to vmalloc(S)
Assume pa(X) = Y
Then, for all every byte x from 0 to PAGE_SIZE-1, PA(X+x) = Y+x
But what about byte x=PAGE_SIZE?
Assume PA(X+PAGE_SIZE) = Z
Could be that Z=Y+PAGE_SIZE
But, could be a completely different physical address Z
CSE 422S – Operating Systems Organization

14. Traditional Memory Mapping Strategy in Linux (on x86_64)
Virtual Memory
Assume blue VA are allocated
Calls to vmalloc() effectively allocate new, previously unused virtual address space, and map it to possibly discontiguous memory
Kernel address space
Physical Memory
New vmalloc()’d memory
CSE 422S – Operating Systems Organization

15. CSE 422S – Operating Systems Organization
Page Level Management
(Pre-)allocating larger chunks of memory
E.g., 2n contiguous pages: alloc_pages(GFP_KERNEL, n)
Each page is fixed size (usually 4Kb or 8Kb) with enough room for hundreds of individual objects
Can request a page with filled-in values
Use get_zeroed_page() to hide previous contents
Must free pages once finished using them
Use free_page() if a single page was allocated
Use free_pages() if multiple were allocated at once
CSE 422S – Operating Systems Organization

16. Object Level Management
Size of each allocation is in bytes
I.e., the size of the object being allocated
Rather than the number of fixed-sized pages
Kernel version of malloc: kmalloc() and kfree()
Action modifiers (can/cannot sleep, etc.)
Zone modifiers (DMA, DMA32, HIGH, NORMAL)
Type flags: combine action and zone modifiers
Allocated memory is physically contiguous
Virtually contiguous memory
Though not necessarily physically contiguous
Use vmalloc() and vfree() etc.
CSE 422S – Operating Systems Organization

17. Kernel Memory Trade-Offs
Page level management
Coarse granularity, low performance overhead
Object level management
Finer granularity, higher performance overhead
By default, prefer the kmalloc() version over the vmalloc() version, for performance
Slab allocation
Optimizes performance if allocating and de-allocating large numbers of the same kind of object
CSE 422S – Operating Systems Organization

18. Other Kernel Memory Options
Static allocation on the stack
Can be risky, as stacks are limited to 1 or 2 pages
In general, minimize stack usage by design
By default, use the kernel allocation techniques previously discussed instead of stack allocation
Per-CPU allocation
Think of this as “core-specific storage”
Can improve cache performance, reduce contention
Must disable kernel preemption (per-CPU interface does)
Must “alloc” and “put” per-CPU data (like alloc/free)
CSE 422S – Operating Systems Organization