Virtual Memory: Systems

Virtual Memory: Systems Simple memory system example Case study: Core i7/Linux memory system Memory mapping

Review of Symbols Basic Parameters N = 2n : Number of addresses in virtual address space M = 2m : Number of addresses in physical address space P = 2p : Page size (bytes) Components of the virtual address (VA) TLBI: TLB index TLBT: TLB tag VPO: Virtual page offset VPN: Virtual page number Components of the physical address (PA) PPO: Physical page offset (same as VPO) PPN: Physical page number CO: Byte offset within cache line CI: Cache index CT: Cache tag

Simple Memory System Example Addressing 14-bit virtual addresses 12-bit physical address Page size = 64 bytes 13 12 11 10 9 8 7 6 5 4 3 2 1 VPN VPO Virtual Page Number Virtual Page Offset 11 10 9 8 7 6 5 4 3 2 1 PPN PPO Physical Page Number Physical Page Offset

1. Simple Memory System TLB 16 entries 4-way associative TLBT TLBI 13 12 11 10 9 8 7 6 5 4 3 2 1 VPN VPO Set Tag PPN Valid Tag PPN Valid Tag PPN Valid Tag PPN Valid 03 – 09 0D 1 00 – 07 02 1 1 03 2D 1 02 – 04 – 0A – 2 02 – 08 – 06 – 03 – 3 07 – 03 0D 1 0A 34 1 02 –

2. Simple Memory System Page Table Only show first 16 entries (out of 256) VPN PPN Valid VPN PPN Valid 00 28 1 08 13 1 01 – 09 17 1 02 33 1 0A 09 1 03 02 1 0B – 04 – 0C – 05 16 1 0D 2D 1 06 – 0E 11 1 07 – 0F 0D 1

3. Simple Memory System Cache 16 lines, 4-byte block size Physically addressed Direct mapped CT CI CO 11 10 9 8 7 6 5 4 3 2 1 PPN PPO Idx Tag Valid B0 B1 B2 B3 Idx Tag Valid B0 B1 B2 B3 19 1 99 11 23 11 8 24 1 3A 00 51 89 1 15 – – – – 9 2D – – – – 2 1B 1 00 02 04 08 A 2D 1 93 15 DA 3B 3 36 – – – – B 0B – – – – 4 32 1 43 6D 8F 09 C 12 – – – – 5 0D 1 36 72 F0 1D D 16 1 04 96 34 15 6 31 – – – – E 13 1 83 77 1B D3 7 16 1 11 C2 DF 03 F 14 – – – –

Address Translation Example #1 Virtual Address: 0x03D4 VPN ___ TLBI ___ TLBT ____ TLB Hit? __ Page Fault? __ PPN: ____ Physical Address CO ___ CI___ CT ____ Hit? __ Byte: ____ TLBT TLBI 13 12 11 10 9 8 7 6 5 4 3 2 1 1 1 1 1 1 1 VPN VPO 0x0F 0x3 0x03 Y N 0x0D CT CI CO 11 10 9 8 7 6 5 4 3 2 1 1 PPN PPO 0x5 0x0D Y 0x36

Address Translation Example #2 Virtual Address: 0x0020 VPN ___ TLBI ___ TLBT ____ TLB Hit? __ Page Fault? __ PPN: ____ Physical Address CO___ CI___ CT ____ Hit? __ Byte: ____ TLBT TLBI 13 12 11 10 9 8 7 6 5 4 3 2 1 1 VPN VPO 0x00 0x00 N N 0x28 CT CI CO 11 10 9 8 7 6 5 4 3 2 1 1 PPN PPO 0x8 0x28 N Mem

Address Translation Example #3 Virtual Address: 0x0020 VPN ___ TLBI ___ TLBT ____ TLB Hit? __ Page Fault? __ PPN: ____ Physical Address CO___ CI___ CT ____ Hit? __ Byte: ____ TLBT TLBI 13 12 11 10 9 8 7 6 5 4 3 2 1 1 VPN VPO 0x00 0x00 N N 0x28 CT CI CO 11 10 9 8 7 6 5 4 3 2 1 1 PPN PPO 0x8 0x28 N Mem

Virtual Memory: Systems Simple memory system example Case study: Core i7/Linux memory system Memory mapping

Intel Core i7 Memory System Processor package Core x4 Registers Instruction fetch MMU (addr translation) L1 d-cache 32 KB, 8-way L1 i-cache 32 KB, 8-way L1 d-TLB 64 entries, 4-way L1 i-TLB 128 entries, 4-way L2 unified cache 256 KB, 8-way L2 unified TLB 512 entries, 4-way To other cores QuickPath interconnect 4 links @ 25.6 GB/s each To I/O bridge L3 unified cache 8 MB, 16-way (shared by all cores) DDR3 Memory controller 3 x 64 bit @ 10.66 GB/s 32 GB/s total (shared by all cores) Main memory

Review of Symbols Basic Parameters N = 2n : Number of addresses in virtual address space M = 2m : Number of addresses in physical address space P = 2p : Page size (bytes) Components of the virtual address (VA) TLBI: TLB index TLBT: TLB tag VPO: Virtual page offset VPN: Virtual page number Components of the physical address (PA) PPO: Physical page offset (same as VPO) PPN: Physical page number CO: Byte offset within cache line CI: Cache index CT: Cache tag

End-to-end Core i7 Address Translation CPU 32/64 L2, L3, and main memory Result Virtual address (VA) 36 12 VPN VPO L1 miss L1 hit 32 4 TLBT TLBI L1 d-cache (64 sets, 8 lines/set) TLB hit TLB miss ... ... L1 TLB (16 sets, 4 entries/set) 9 9 9 9 40 12 40 6 6 VPN1 VPN2 VPN3 VPN4 PPN PPO CT CI CO Physical address (PA) CR3 PTE PTE PTE PTE Page tables

Core i7 Level 1-3 Page Table Entries 63 62 52 51 12 11 9 8 7 6 5 4 3 2 1 XD Unused Page table physical base address Unused G PS A CD WT U/S R/W P=1 Available for OS (page table location on disk) P=0 Each entry references a 4K child page table. Significant fields: P: Child page table present in physical memory (1) or not (0). R/W: Read-only or read-write access access permission for all reachable pages. U/S: user or supervisor (kernel) mode access permission for all reachable pages. WT: Write-through or write-back cache policy for the child page table. CD: Caching disabled or enabled for the child page table A: Reference bit (set by MMU on reads and writes, cleared by software). PS: Page size either 4 KB or 4 MB (defined for Level 1 PTEs only). Page table physical base address: 40 most significant bits of physical page table address (forces page tables to be 4KB aligned) XD: Disable or enable instruction fetches from all pages reachable from this PTE.

Core i7 Level 4 Page Table Entries 63 62 52 51 12 11 9 8 7 6 5 4 3 2 1 XD Unused Page physical base address Unused G D A CD WT U/S R/W P=1 Available for OS (page location on disk) P=0 Each entry references a 4K child page. Significant fields: P: Child page is present in memory (1) or not (0) R/W: Read-only or read-write access permission for child page U/S: User or supervisor mode access WT: Write-through or write-back cache policy for this page CD: Caching disabled or enabled for this page A: Reference bit (set by MMU on reads and writes, cleared by software) D: Dirty bit (set by MMU on writes, cleared by software) Page physical base address: 40 most significant bits of physical page address (forces pages to be 4KB aligned) XD: Disable or enable instruction fetches from this page.

Core i7 Page Table Translation 9 9 9 9 12 Virtual address VPN 1 VPN 2 VPN 3 VPN 4 VPO L1 PT Page global directory L2 PT Page upper directory L3 PT Page middle directory L4 PT Page table 40 40 40 40 CR3 / / / / Physical address of L1 PT Offset into physical and virtual page / 12 L1 PTE L2 PTE L3 PTE L4 PTE Physical address of page 512 GB region per entry 1 GB region per entry 2 MB region per entry 4 KB region per entry 40 / 40 12 Physical address PPN PPO

Cute Trick for Speeding Up L1 Access CT Tag Check 40 6 6 Physical address (PA) CT CI CO PPN PPO Address Translation No Change Virtual address (VA) CI L1 Cache VPN VPO Observation Bits that determine CI identical in virtual and physical address Can index into cache while address translation taking place Generally we hit in TLB, so PPN bits (CT bits) available next “Virtually indexed, physically tagged” Cache carefully sized to make this possible 36 12

Virtual Address Space of a Linux Process Process-specific data structs (ptables, task and mm structs, kernel stack) Different for each process Kernel virtual memory Physical memory Identical for each process Kernel code and data User stack %rsp Memory mapped region for shared libraries Process virtual memory brk Runtime heap (malloc) Uninitialized data (.bss) Initialized data (.data) Program text (.text) 0x00400000

Linux Organizes VM as Collection of “Areas” Process virtual memory vm_area_struct task_struct mm_struct vm_end vm_start mm pgd vm_prot vm_flags mmap Shared libraries vm_next vm_end pgd: Page global directory address Points to L1 page table vm_prot: Read/write permissions for this area vm_flags Pages shared with other processes or private to this process vm_start Data vm_prot vm_flags Text vm_next vm_end vm_start vm_prot vm_flags vm_next

Linux Page Fault Handling Process virtual memory vm_area_struct vm_end vm_start vm_prot vm_flags shared libraries vm_next read 1 Segmentation fault: accessing a non-existing page vm_end vm_start read 3 data vm_prot Normal page fault vm_flags text vm_next write 2 Protection exception: e.g., violating permission by writing to a read-only page (Linux reports as Segmentation fault) vm_end vm_start vm_prot vm_flags vm_next

Virtual Memory: Systems Simple memory system example Case study: Core i7/Linux memory system Memory mapping

Memory Mapping VM areas initialized by associating them with disk objects. Process is known as memory mapping. Area can be backed by (i.e., get its initial values from) : Regular file on disk (e.g., an executable object file) Initial page bytes come from a section of a file Anonymous file (e.g., nothing) First fault will allocate a physical page full of 0's (demand-zero page) Once the page is written to (dirtied), it is like any other page Dirty pages are copied back and forth between memory and a special swap file.

Sharing Revisited: Shared Objects Process 1 virtual memory Physical memory Process 2 virtual memory Process 1 maps the shared object. Shared object

Sharing Revisited: Shared Objects Process 1 virtual memory Physical memory Process 2 virtual memory Process 2 maps the shared object. Notice how the virtual addresses can be different. Shared object

Sharing Revisited: Private Copy-on-write (COW) Objects Process 1 virtual memory Physical memory Process 2 virtual memory Two processes mapping a private copy-on-write (COW) object. Area flagged as private copy-on-write PTEs in private areas are flagged as read-only Private copy-on-write area Private copy-on-write object

Sharing Revisited: Private Copy-on-write (COW) Objects Process 1 virtual memory Physical memory Process 2 virtual memory Instruction writing to private page triggers protection fault. Handler creates new R/W page. Instruction restarts upon handler return. Copying deferred as long as possible! Copy-on-write Write to private copy-on-write page Private copy-on-write object

User-Level Memory Mapping void *mmap(void *start, int len, int prot, int flags, int fd, int offset) Map len bytes starting at offset offset of the file specified by file description fd, preferably at address start start: may be 0 for “pick an address” prot: PROT_READ, PROT_WRITE, ... flags: MAP_ANON, MAP_PRIVATE, MAP_SHARED, ... Return a pointer to start of mapped area (may not be start)

User-Level Memory Mapping void *mmap(void *start, int len, int prot, int flags, int fd, int offset) len bytes start (or address chosen by kernel) len bytes offset (bytes) Disk file specified by file descriptor fd Process virtual memory

Example: Using mmap to Copy Files Copying a file to stdout without transferring data to user space . #include "csapp.h" void mmapcopy(int fd, int size) { /* Ptr to memory mapped area */ char *bufp; bufp = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0); write(1, bufp, size); return; } /* mmapcopy driver */ int main(int argc, char **argv) { struct stat stat; int fd; /* Check for required cmd line arg */ if (argc != 2) { printf("usage: %s <filename>\n", argv[0]); exit(0); } /* Copy input file to stdout */ fd = open(argv[1], O_RDONLY, 0); fstat(fd, &stat); mmapcopy(fd, stat.st_size); mmapcopy.c mmapcopy.c