Virtual Memory and Paging Brian Kocoloski CSE 422S - Operating Systems Organization Washington University in St. Louis St. Louis, MO 63130

Resource Virtualization What resources do you use when you write programs? The processor(s) Memory Files Data (global variables, things like hardcoded strings) What do you not have to worry about generally? Existence of other processes State of hardware resources CSE 422S – Operating Systems Organization

Resource Virtualization The operating system virtualizes physical hardware resources into virtual resources in the form of processes Physical Memory Physical Processors CSE 422S – Operating Systems Organization

Resource Virtualization Process 1 Process 2 Processor Virtualization: Two (or more) processes may share the same set of physical CPUs How to share them? Context switching + CPU scheduling Memory Virtualization: Translate private memory addresses to physical addresses Virtual memory Physical Memory Physical Processors CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Virtual Memory What is the purpose of virtual memory? Give processes illusion of full dedicated access to entire system memory Every process’ virtual address space is unique and independent from all other processes Similar to how a process thinks it has dedicated access to processors Preserve physical resources until they are actually needed Many programs don’t need all of their code/data in memory at once (untaken branches, unused data, etc.) Allocating virtual memory != allocating physical memory Unify the differences in physical memory devices behind a common abstraction Simplifies application programming Similar to how ’processes’ and ‘threads’ exist on all different architectures despite core-level differences CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Memory Pages All physical memory is divided into pages Page sizes are determined by the architecture E.g. on both x86 and ARM, pages are 4KB in size Total amount of physical memory: X bytes Total number of pages: X/4KB Physical Address: 0x0 0x1000 0x2000 0x3000 4KB 4KB 4KB 4KB 4KB 4KB … 4KB CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Memory Pages On all modern architectures, memory is accessed virtually via virtual addresses Virtual addresses require translation Virtual Address Translation: Process A Process B Virtual Address 4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB Virtual Memory Page # Offset 4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB Physical Memory Physical Page Phys. Addr + CSE 422S – Operating Systems Organization

/proc/<pid>/maps The valid virtual addresses in a process $ cat /proc/self/maps Executable code Read-only static data Read/write static data CSE 422S – Operating Systems Organization

/proc/<pid>/maps The valid virtual addresses in a process $ cat /proc/self/maps CSE 422S – Operating Systems Organization

/proc/<pid>/maps The valid virtual addresses in a process heap: 0x00726000 – 0x00747000 0x21000 in length, or 132KB stack: 0x7e7f6000 – 0x7e817000 0x21000 in length, also 132KB CSE 422S – Operating Systems Organization

Memory Allocation/Mapping In Linux, there are two primary system calls that perform memory allocation brk() – expand the process’ heap mmap() – similar to heap, but can have “holes” in the address space, and do other things like memory map files CSE 422S – Operating Systems Organization

/proc/<pid>/maps The valid virtual addresses in a process $ cat /proc/self/maps Anonymous Memory (mmap) CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Paging malloc(…) 0x001b… User space Virtual address (x = 0x001b …) # of bytes Allocate new virtual memory area (VA) Allocate physical memory pages (PA) Update page tables to map VA -> PA Kernel space Page Tables VA (x) PA (y) Hardware CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Paging write(“Blah”, 0x001b…); malloc(…) 0x001b… User space Virtual address (x = 0x001b …) # of bytes Allocate new virtual memory area (VA) Allocate physical memory pages (PA) Update page tables to map VA -> PA Address 0x001b Kernel space Page Tables MMU Hardware CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Paging write(“Blah”, 0x001b…); malloc(…) 0x001b… User space Virtual address # of bytes Allocate new virtual memory area (VA) Allocate physical memory pages (PA) Update page tables to map VA -> PA Address 0x001b Kernel space Page Tables Physical Memory MMU Address 0x001b Hardware CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Demand Paging malloc(…) 0x001b… User space Virtual address # of bytes Allocate new virtual memory area (VA) DO NOT allocate physical memory DO NOT update page tables X Kernel space Page Tables X Hardware CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Demand Paging write(“Blah”, 0x001b…); malloc(…) 0x001b… User space Virtual address # of bytes Allocate new virtual memory area (VA) DO NOT allocate physical memory DO NOT update page tables X Kernel space Page Tables Page Tables X MMU Address 0x001b Hardware CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Demand Paging write(“Blah”, 0x001b…); malloc(…) 0x001b… User space Virtual address PAGE FAULT MMU cannot find VA (0x001b ...) in the page tables # of bytes Allocate new virtual memory area (VA) DO NOT allocate physical memory DO NOT update page tables X Kernel space Page Tables Page Tables X MMU Address 0x001b Hardware CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Demand Paging write(“Blah”, 0x001b…); User space Query list of vm_area_struct for the process If no VMA exists, invalid If VMA exists, but no write permission, invalid Else, OK Page fault handler Can user access VA 0x001b …? Kernel space Page Tables Page Tables MMU Address 0x001b Hardware CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Demand Paging write(“Blah”, 0x001b…); Segmentation fault User space (1) Allocate physical pages and map them in the page tables (2) Reissue instruction Page fault handler Can user access VA 0x001b …? No Kernel space Page Tables Page Tables Yes write(“Blah”, 0x001b…); Physical Memory MMU Address 0x001b Hardware CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Address Translation A virtual address is broken up into 2 pieces: p: page number d: page offset 64 bit virtual address space 4096 (4KB) byte page frame 12 bits to reference bytes within page frame (2^12 = 4096) That leaves 52 bits for p Virtual Address Translation: Virtual Address Page # Offset Physical Page Phys. Addr + CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Address Translation Q: With 52 bits for the page number, how many different page numbers can we address? In other words, how big must our translation table be? Virtual Address Translation: Virtual Address Page number (52 bits) Translation Table Physical Page CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Address Translation Q: With 52 bits for the page number, how many different page numbers can we address? A: 2^52 Q: How to design a lookup table that can efficiently perform translations of up to 2^52 different virtual page numbers? Solution: multi-level page tables Virtual Address Translation: Virtual Address Page number (52 bits) Translation Table Physical Page CSE 422S – Operating Systems Organization

Multi-Level Paging … Virtual Address Translation: Virtual Address Top level lookup directory Physical Page Second level lookup directory Third level lookup directory … CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Multi-Level Paging Goal: We want each page table to fit into one page (4096 bytes) of memory Proposed scheme: Page Frame: 12 bits (2^12 = 4KB) Top level page number: 9 bits 2nd level page number: 9 bits 3rd level page number: 9 bits 4th level page number: 9 bits Unused: 16 bits Note: These page numbers are really offsets into other page tables! CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example Example 0x0000 cd00 2240 6c90 1010 1011 0000 0000 0010 0010 0100 0000 0110 1010 1001 0000 ( c d 0 0 2 2 4 0 6 c 9 0 ) Top level page number (9 bits) 1 0101 0110 : 342 2nd level page number (9 bits) 0 0000 0000 : 0 3rd level page number (9 bits) 1 0001 0010 : 274 4th level page number (9 bits) 0 0000 0110 : 6 Page offset (12 bits) 1010 1001 0000 : 2704 CSE 422S – Operating Systems Organization

Control Register (x86: CR3; Example CPU instruction: WR R1, 0x0000 cd00 2240 6c90 Step 1: Get Top-level page table address Control Register (x86: CR3; ARM: CP15 c2) MMU WR 0x0000 cd00 2240 6c90 Top level page table address (a) CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example 0x0000 cd00 2240 6c90 1010 1011 0000 0000 0010 0010 0100 0000 0110 1010 1001 0000 ( 342 0 274 6 2704 ) a Index MetaData (r/w/x) (present) Page Address for next level of page table … 342 Present: 0 511 Hardware sees present bit set to 0: it CANNOT complete the VA -> PA translation So it raises an exception called a PAGE_FAULT CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Page Fault Handler Hardware tells the kernel that a process tried to access a virtual address that either Has not associated physical address in the page table Or, accessed the memory in an invalid fashion (i.e., tried to write to read-only memory) Kernel asks: (1) Is this a valid virtual address for the process to be accessing? i.e., if it’s a read, is the memory readable? If write, writable? If execution, executable? (2) If not, raise SIGSEGV – the dreaded Segmentation Fault (3) Else, update the page tables so the hardware knows how to complete the translation CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example 0x0000 cd00 2240 6c90 1010 1011 0000 0000 0010 0010 0100 0000 0110 1010 1001 0000 ( 342 0 274 6 2704 ) a Index MetaData (r/w/x) (present) Page Address for next level of page table … 342 Present: 0 511 Hardware sees present bit set to 0: it CANNOT complete the VA -> PA translation So it raises an exception called a PAGE_FAULT CSE 422S – Operating Systems Organization

Kernel Page fault handler Example 0x0000 cd00 2240 6c90 1010 1011 0000 0000 0010 0010 0100 0000 0110 1010 1001 0000 ( 342 0 274 6 2704 ) a idx P Page Address … 342 1 b 511 Kernel Page fault handler Check if access is valid Allocate a new page, say p Determine physical address of, say b Store b in page address of page table Set present bit to 1 Hardware sees present bit set to 0: it CANNOT complete the VA -> PA translation So it raises an exception called a PAGE_FAULT CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example 0x0000 cd00 2240 6c90 1010 1011 0000 0000 0010 0010 0100 0000 0110 1010 1001 0000 ( 342 0 274 6 2704 ) a idx P Page Address … 342 1 b 511 b idx P Page Address … 511 Hardware sees present bit set to 0: it CANNOT complete the VA -> PA translation So it raises an exception called a PAGE_FAULT CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example 0x0000 cd00 2240 6c90 1010 1011 0000 0000 0010 0010 0100 0000 0110 1010 1001 0000 ( 342 0 274 6 2704 ) a idx P Page Address … 342 1 b 511 b idx P Page Address 1 c … 511 c idx P Page Address … 274 511 Hardware sees present bit set to 0: it CANNOT complete the VA -> PA translation So it raises an exception called a PAGE_FAULT CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example 0x0000 cd00 2240 6c90 1010 1011 0000 0000 0010 0010 0100 0000 0110 1010 1001 0000 ( 342 0 274 6 2704 ) Index Page … 342 b 511 Grab the 2704’th byte in this page Physical Page (4KB) a Index Page c … 511 Index Page … 274 d 511 Index Page … 6 e 511 CSE 422S – Operating Systems Organization

Kernel vs User Virtual Memory When page tables are filled in on-demand like this, this is referred to as demand paging By default, Linux uses demand paging for all user processes Keeps memory allocation (virtual, i.e., malloc()) fast Doesn’t allocate physical memory until process tries to access it However, kernel virtual memory is always pre-paged When the system boots, all physical memory is mapped into the kernel virtual address space CSE 422S – Operating Systems Organization

Kernel vs User Virtual Memory Kernel Virtual Memory Physical Memory Mappings created at runtime via memory allocation (e.g., mmap()) and page fault handling Mappings created at system boot time CSE 422S – Operating Systems Organization

Kernel vs User Virtual Memory Given a physical address, how do I calculate the kernel virtual address to access it? __va(unsigned long pa); On arm 7 (R Pi 3): ((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET)) PHYS_OFFSET: first address of physical memory PAGE_OFFSET: (On our 32-bit R Pis) 0x80000000 The point is, reverse PA->VA translation can be done in O(1) time (You did this in studio 12) Given a user virtual address, how would the kernel calculate its physical address? Given a physical address, how do I calculate the user virtual address to access it? CSE 422S – Operating Systems Organization