Virtual Memory and Interrupts Constructive Computer Architecture Virtual Memory and Interrupts Arvind Computer Science & Artificial Intelligence Lab. Massachusetts Institute of Technology November 13, 2015 http://csg.csail.mit.edu/6.175

Caching vs. Demand Paging secondary memory primary memory primary memory CPU CPU cache Caching Demand paging cache slot page frame cache line (~32 bytes) page (~4K bytes) cache miss rate (1% to 20%) page miss rate (<0.001%) cache hit (~1 cycle) page hit (~100 cycles) cache miss (~100 cycles) page miss (~5M cycles) miss is handled in hardware miss is handled mostly in software November 9, 2015 http://csg.csail.mit.edu/6.175

Address Translation: putting it all together Virtual Address hardware hardware or software software TLB Lookup miss hit Page Table Walk Protection Check the page is Ï memory Î memory denied Need to restart instruction. Soft and hard page faults. permitted Page Fault (OS loads page) Protection Fault Update TLB Physical Address (to cache) Where? SEGFAULT Resume the instruction November 13, 2015 http://csg.csail.mit.edu/6.175

Address Translation in Pipeline Machines PC D E M W Inst TLB Inst. Cache Decode Data TLB Data Cache + TLB miss? Page Fault? Protection violation? TLB miss? Page Fault? Protection violation? Software handlers need a restartable exception on page fault or protection violation Handling a TLB miss needs a hardware or software mechanism to refill TLB Methods to overcome the additional latency of a TLB: slow down the clock pipeline the TLB and cache access virtual address caches parallel TLB/cache access Restartable exception is going back to the same LD/ST instruction. If we have a divide by zero exception, we don’t go back to the instruction that caused the exception.   November 13, 2015 http://csg.csail.mit.edu/6.175

Physical vs Virtual Address Caches? CPU Physical Cache TLB Primary Memory VA PA Alternative: place the cache before the TLB CPU VA (StrongARM) Virtual Cache PA TLB Primary Memory One cycle in case of a hit (+) cache needs to be flushed on a context switch unless address space identifiers (ASIDs) included in tags (-) aliasing problems due to the sharing of pages (-) November 13, 2015 http://csg.csail.mit.edu/6.175

Aliasing in Virtual-Address Caches Page Table Tag Data VA1 Data Pages VA1 VA2 1st Copy of Data at PA 2nd Copy of Data at PA PA VA2 Virtual cache can have two copies of same physical data. Writes to one copy not visible to reads of other! Two virtual pages share one physical page Two processes sharing the same file, Map the same memory segment to different Parts of their address space. General Solution: Disallow aliases to coexist in cache Software (i.e., OS) solution for direct-mapped cache VAs of shared pages must agree in cache index bits; this ensures all VAs accessing same PA will conflict in direct-mapped cache (early SPARCs) November 13, 2015 http://csg.csail.mit.edu/6.175

Concurrent Access to TLB & Cache VPN L b TLB Direct-map Cache 2L blocks 2b-byte block PPN Page Offset = hit? Data Physical Tag Tag VA PA Virtual Index k Tag size = {PA} – L – b What if you want to build a large cache where L + b > k? Index L is available without consulting the TLB cache and TLB accesses can begin simultaneously Tag comparison is made after both accesses are completed Cases: L + b = k L + b < k L + b > k what happens here? Partially VA cache! November 13, 2015 http://csg.csail.mit.edu/6.175

Virtual-Index Physical-Tag Caches: Associative Organization VPN L = k-b b TLB Direct-map 2L blocks PPN Page Offset = hit? Data Phy. Tag VA PA Virtual Index k W ways After the PPN is known, W physical tags are compared Allows cache size to be greater than 2L+b bytes November 13, 2015 http://csg.csail.mit.edu/6.175

Exception handling in a pipeline machine November 13, 2015 http://csg.csail.mit.edu/6.175

Exception Handling Commit Point PC D E M W Inst. Mem Decode Data Mem + Kill D Stage Kill F Stage Kill E Stage Select Handler PC Kill Writeback Illegal Opcode Overflow Data address Exceptions PC address Exception Ex D Ex E Ex M Cause PC D PC E PC M EPC External Interrupts 1. An instruction may cause multiple exceptions; which one should we process? from the earliest stage 2. When multiple instructions are causing exceptions; which one should we process first? from the oldest instruction November 13, 2015 http://csg.csail.mit.edu/6.175

Interrupt processing Internal interrupts can happen at any stage but cause a redirection only at Commit External interrupts are considered only at Commit If an instruction causes an interrupt then the external interrupt, if present, is given a priority and the instruction is executed again Some instructions, like Store, cannot be undone once launched. So an instruction is considered to have completed before an external interrupt is taken November 13, 2015 http://csg.csail.mit.edu/6.175

Exception Handling When instruction x in stagei raises an exception, its cause is recorded and passed down the pipeline For a given instruction, exceptions from the later stages of the pipeline do not override cause of exception from the earlier stages If an exception is present at commit: Cause and EPC registers are set, and pc is redirected to the handler PC Epoch mechanism takes care of redirecting the pc There is also a status register. (See my notes on Slide 14.) November 4, 2015 http://csg.csail.mit.edu/6.175

Killing vs Poisoning PC Inst Memory Decode Register File Execute Data f2d Epoch m2c d2e Next Addr Pred scoreboard f12f2 e2m external interrupts considered at Commit one_cyc.bsv is pretty straightforward. pipeline.bsv is not to bad either (shared most of the code with one_cyc in common_lib). Try diff one_cyc.bsv and pipeline.bsv against your reference release. The changes are very surgical. On pipeline.bsv there was an interest scheduling issue which Andy and I discussed by email in August. >>>excerpt The most interesting issue I ran into were in redirecting the PC after an exception in the pipelined version. I would have liked to redirect the PC in the WB-rule (updating pc_ehr[4] *after* all of the other pc_ehr writes). This ordering conflicts with the bypassing-RFile which insists that the WB-rule happens before the Rfetch-rule.   I choose to solve it by redirecting the PC in the memory stage after checking for load/store address exceptions. wrong path insts are dropped wrong path insts are poisoned This affects whether an instruction is removed from sb in case of an interrupt November 4, 2015 http://csg.csail.mit.edu/6.175 13

Interrupt processing at Execute Incoming Interrupt no yes -if (mem type) issue Ld/St -if (mispred) redirect -pass eInst to M stage -pass eInst to M stage unmodified eInst will contain information about any newly detected interrupts at Execute I didn’t do interrupt, but a superset of plumbing exist. Interrupt is no harder than syscall. Also, I don’t have the context for the next 3 slides to understand it. November 13, 2015 http://csg.csail.mit.edu/6.175

Interrupt processing at Memory stage Incoming Interrupt no yes Memory Interrupt? -pass eInst to Commit unmodified no yes -pass eInst with modified data to Commit -pass new Cause to Commit November 13, 2015 http://csg.csail.mit.edu/6.175

Interrupt processing at Commit External Interrupt? no yes Incoming interrupt Incoming interrupt no yes no yes commit; sb.rm; EPC<= ppc; causeR <= Ext; mode <= privilege; Redirect -commit -sb.rm EPC<= pc; causeR <= inCause; if (inCause after Reg Fetch) sb.rm; mode <= privilege; Redirect EPC<= pc; causeR <= Ext; if (inCause after Reg Fetch) sb.rm; mode <= privilege; Redirect November 13, 2015 http://csg.csail.mit.edu/6.175

Final comment There is generally a lot of machinery associated with a plethora of exceptions in ISAs Precise exceptions are difficult to implement correctly in pipelined machines Performance is usually not the issue and therefore sometimes exceptions are implemented using microcode November 4, 2015 http://csg.csail.mit.edu/6.175

RISC-V Virtual Memory Privileged ISA v. 1.9.1 November 13, 2015 http://csg.csail.mit.edu/6.175

RISC-V Privilege Levels Separation between low-level access to the hardware and high-level user programs Machine-mode (M) – all addresses are physical addresses, has access to all addresses including memory-mapped IO devices Supervisor-mode (S) – addresses are typically virtual addresses, can switch page-table in use (sptbr) User-mode (U) – addresses are virtual, access to devices only through systemcalls November 13, 2015 http://csg.csail.mit.edu/6.175

RISC-V Memory Maps Machine-Mode Physical Addresses DRAM Boot ROM MMIO Debug Unit Machine-Mode Physical Addresses DRAM User-Mode Virtual Addresses Only part of the address space is DRAM Demand Paging makes the entire address space look like DRAM November 13, 2015 http://csg.csail.mit.edu/6.175

RISC-V Paged Virtual Memory Different modes for different systems: Sv32: 32-bit VA, 34-bit PA 4 GB virtual address space 16 GB physical address space 2-layer page table Sv39: 39-bit VA, 50-bit PA 512 GB virtual address space 1 PB physical address space 3-layer page table Requires RV64, 64-bit ISA November 13, 2015 http://csg.csail.mit.edu/6.175

Sv32 Addresses Virtual Addresses: Physical Addresses: VPN[0] page offset VPN[1] 12 bits 10 bits 1st level Page Table Index 2st level Page Table Index VPN[0] page offset PPN[1] 12 bits 10 bits Both come from 2nd level Page Table Entry November 13, 2015 http://csg.csail.mit.edu/6.175

Sv32 Page Table Entries PPN[0] SW Reserved PPN[1] D A G U X W R V 2 bits 10 bits 12 bits Dirty – This page has been written to Accessed - This page has been accessed Global – Mapping exists in all virtual address spaces User – User-mode programs can access this page eXecute – This page can be executed Write – This page can be written to Read – This page can be read from Valid – This page valid and in memory November 13, 2015 http://csg.csail.mit.edu/6.175

Sv32 Page Table Entries PPN[0] SW Reserved PPN[1] D A G U X W R V 2 bits 10 bits 12 bits If V = 1, but X, W, R == 0, PPN[] points to the 2nd level page table If V = 0, page is either invalid or in disk. If in disk, the OS can reuse bits in the PTE to store the disk address (or part of it). Disk Address G U X W R 26 bits November 13, 2015 http://csg.csail.mit.edu/6.175

RISC-V Pipeline with VM PC D E M W Inst TLB Inst. Cache Decode Data TLB Data Cache + translation or fault translation or fault miss miss Page Table Walker Page Table Walker memory accesses memory accesses On a fault, an exception is raised and the OS takes over November 13, 2015 http://csg.csail.mit.edu/6.175

RISC-V VM Instructions SFence.VM Privileged instruction to synchronize TLB translation. Ensures that stores to data cache are seen by hardware page table walker CSRs – Control and Status Registers Privileged registers for processor configuration sptbr – Page Table Base Register mstatus.vm – Virtual Memory mode (e.g. Sv32) mstatus.mxr, mstatus.pum, mstatus.mprv – Fields for modifying privileges for memory accesses to emulate accesses at low privilege levels November 13, 2015 http://csg.csail.mit.edu/6.175