1. 1 Lecture 12 Virtual Memory Peng Liu liupeng@zju.edu.cn

2. 2 Last time in Lecture 11 Cache –The memory system has a significant effect on program execution time. –The number of memory-stall cycles depends on both the miss rate and the miss penalty. »To reduce the miss rate, the use of associative placement schemes »To reduce the miss penalty by allowing a larger secondary cache to handle misses to the primary cache.

3. 3 Associative Cache Example Review

4. 4 Direct-Mapped Cache TagData Block V = Block Offset TagIndex t k b t HIT Data Word or Byte 2 k lines Review

5. 5 2-Way Set-Associative Cache TagData Block V = Block Offset TagIndex t k b HIT TagData Block V Data Word or Byte = t Review

6. 6 Fully Associative Cache TagData Block V = Block Offset Tag t b HIT Data Word or Byte = = t Review

7. 7 Tag & Index with Set-Associative Caches Assume a 2 n -byte cache with 2 m -byte blocks that is 2 a way set- associative –Which bits of the address are the tag or the index? –m least significant bits are byte select within the block Basic idea –The cache contains 2 n /2 m =2 n-m blocks –Each cache way contains 2 n-m /2 a =2 n-m-a blocks –Cache index: (n-m-a) bits after the byte select Same index used with all cache ways … Observation –For fixed size, length of tags increases with the associativity –Associative caches incur more overhead for tags Review

8. 8 Replacement Methods Which line do you replace on a miss? Direct Mapped –Easy, you have only one choice –Replace the line at the index you need N-way Set Associative –Need to choose which way to replace –Random (choose one at random) –Least Recently Used (LRU) (the one used least recently) Often difficult to calculate, so people use approximations. Often they are really not recently used Review

9. 9 Replacement Policy In an associative cache, which block from a set should be evicted when the set becomes full? Random Least Recently Used (LRU) LRU cache state must be updated on every access true implementation only feasible for small sets (2-way) pseudo-LRU binary tree often used for 4-8 way First In, First Out (FIFO) a.k.a. Round-Robin used in highly associative caches This is a second-order effect. Why? Replacement only happens on misses Review

10. 10 Causes for Cache Misses Compulsory: first-reference to a block a.k.a. cold start misses - misses that would occur even with infinite cache Capacity: cache is too small to hold all data needed by the program - misses that would occur even under perfect replacement policy Conflict: misses that occur because of collisions due to block-placement strategy - misses that would not occur with full associativity Review

11. 11 Write Policy Choices Cache hit: –write through: write both cache & memory generally higher traffic but simplifies cache coherence –write back: write cache only (memory is written only when the entry is evicted) a dirty bit per block can further reduce the traffic Cache miss: –no write allocate: only write to main memory –write allocate (aka fetch on write): fetch into cache Common combinations: –write through and no write allocate –write back with write allocate Review

12. 12 Cache Design: Datapath + Control To CPU To Lower Level Memory To CPU To Lower Level Memory TagsBlocks Addr Din Dout Addr Din Dout State Machine Control Most design errors come from incorrect specification of state machine behavior! Common bugs: Stalls, Block replacement, Write buffer Review

13. 13 Virtual Memory

14. 14 Motivation #1: Large Address Space for Each Executing Program Each program thinks it has a ~2 32 byte address space of its own –May not use it all though Available main memory may be much smaller

15. 15 Motivation #2: Memory Management for Multiple Programs At an point in time, a computer may be running multiple programs –E.g., Firefox + Foxmail Questions: –How do we share memory between multiple programs? –How do we avoid address conflicts? –How do we protect programs Isolations and selective sharing

16. 16 Virtual Memory in a Nutshell Use hard disk (or Flash) as a large storage for data of all programs –Main memory (DRAM) is a cache for the disk –Managed jointly by hardware and the operating system (OS) Each running program has its own virtual address space –Address space as shown in previous figure –Protected from other programs Frequently-used portions of virtual address space copied to DRAM –DRAM = physical address space –Hardware + OS translate virtual addresses (VA) used by program to physical addresses (PA) used by the hardware –Translation enables relocation (DRAM disk) & protection

17. 17 Reminder: Memory Hierarchy Everything is a Cache for Something Else Access timeCapacityManaged by 1 cycle~500BSoftware/compiler 1-3 cycles~64KBhardware 5-10 cycles1-10MBhardware ~100 cycles~10GBSoftware/OS 10 6 -10 7 cycles~100GBSoftware/OS

18. 18 DRAM vs. SRAM as a “Cache” DRAM vs. disk is more extreme than SRAM vs. DRAM –Access latencies: DRAM ~10X slower than SRAM Disk ~100000X slower than DRAM –Importance of exploiting spatial locality First byte is ~100,000X slower than successive bytes on disk vs, ~4X improvement for page-mode vs. regular accesses to DRAM

19. 19 Impact of These Properties on Design Bottom line: –Design decision made for virtual memory driven by enormous cost of misses (disk accesses) Consider the following parameters for DRAM as a “cache” for the disk –Line size? Large, since disk better at transferring large blocks and minminzes miss rate –Associativity? High, to minminze miss rate –Write through or write back? Write back, since can’t afford to perform small writes to disk

20. 20 Terminology for Virtual Memory Virtual memory used DRAM as a cache for disk New terms –VM block is called a “page” The unit of data moving between disk and DRAM It is typically larger than a cache block (e.g., 4KB or 16KB) Virtual and physical address spaces can be divided up to virtual pages and physical pages (e.g., contiguous chunks of 4KB) –VM miss is called a “page fault” More on this later

21. 21 Locating an Object in a “Cache” SRAM Cache (L1,L2, etc) –Tag stored with cache line –Maps from cache block to a memory address –No tag for blocks not in cache If not in cache, then it is in main memory –Hardware retrieves and manages tag information Can quickly match against multiple tags

22. 22 Locating an Object in a “Cache” (cont.) SRAM Cache (virtual memory) –Each allocated page of virtual memory has entry in page table –Mapping from virtual pages to physical pages One entry per page in the virtual address space –Page table entry even if page not in memory Specifies disk address –OS retrieve and manages page table information

23. 23 A System with Physical Memory Only Examples: –Most Cray machines, early PCs, nearly all embedded systems, etc Addresses generated by the CPU point directly to bytes in physical memory

24. 24 A System with Virtual Memory Examples: –Workstations, serves, modern PCs, etc. Address Translation: Hardware converts virtual addresses to physical addresses via an OS-managed lookup table (page table)

25. 25 Page Faults (Similar to “Cache Misses”) What if an object is on disk rather than in memory? –Page table entry indicates virtual address not in memory –OS exception handler invoked to move data from disk into memory OS has full control over placement Full-associativity to minimize future misses Before fault After fault

26. 26 Does VM Satisfy Original Motivations? Multiple active programs can share physical address space Address conflicts are resolved –All programs think their code is at 0x400000 Data from different programs can be protected Programs can share data or code when desired

27. 27 Answer: Yes, Using Separate Addresses Spaces Per Program Each program has its own virtual address space and own page table –Addresses 0x400000 from different programs can map to different locations or same location as desired –OS control how virtual pages as assigned to physical memory

28. 28 Bare Machine In a bare machine, the only kind of address is a physical address PC Inst. Cache D Decode EM Data Cache W + Main Memory (DRAM) Memory Controller Physical Address

29. 29 Dynamic Address Translation Motivation In the early machines, I/O operations were slow and each word transferred involved the CPU Higher throughput if CPU and I/O of 2 or more programs were overlapped. How?multiprogramming with DMA I/O devices, interrupts Location-independent programs Programming and storage management ease  need for a base register Protection Independent programs should not affect each other inadvertently  need for a bound register Multiprogramming drives requirement for resident supervisor software to manage context switches between multiple programs prog1 prog2 Physical Memory OS

30. 30 Translation: High-level View Fixed-size pages Physical page sometimes called as frame

31. 31 Translation: Process

32. 32 Address Translation & Protection Every instruction and data access needs address translation and protection checks A good VM design needs to be fast (~ one cycle) and space efficient Physical Address Virtual Address Address Translation Virtual Page No. (VPN)offset Physical Page No. (PPN)offset Protection Check Exception? Kernel/User Mode Read/Write

33. 33 Translation Process Explained Valid page –Check access rights (R,W,X) against access type Generate physical address if allowed Generate a protection fault (exception) if illegal access Invalid page –Page is not currently mapped and a page fault is generated Faults are handled by the operating system –Sometimes due to a program error => program terminated E.g. accessing out of the bounds of array –Sometimes due to “caching” => refill & restart Desired data or code available on disk Space allocated in DRAM, page copied from disk, page table updated Replacement may be needed

34. 34 VM: Replacement and Writes To reduce page fault rate, OS uses least-recently used (LRU) replacement –Reference bit (aka use bit) in PTE set to 1 on access to page –Periodically cleared to 0 by OS –A page with reference bit = 0 has not been used recently Disk writes take millions of cycles –Block at once, not individual locations –Write through is impractical –Use write-back –Dirty bit in PTE set when page is written

35. 35 VM: Issues with Unaligned Accesses Memory access might be aligned or unaligned What happens if unaligned address access straddles a page boundary? –What if one page is present and the other is not? –Or, what if neither is present? MIPS architecture disallows unaligned memory access Interesting legacy problem on 80x86 which does support unaligned access

36. 36 Fast Translation Using a TLB Address translation would appear to require extra memory references –One to access the PTE –Then the actual memory access But access to page tables has good locality –So use a fast hardware cache of PTEs within the processor –Called a Translation Look-aside Buffer (TLB) –Typical: 16-512 PTEs, 0.5-1 cycle for hit 10-100 cycles for miss, 0.01%-1% miss rate Misses could be handled by hardware or software

37. 37 Fast Translation Using a TLB

38. 38 Translation LookasideBuffers (TLB) Address translation is very expensive! In a two-level page table, each reference becomes several memory accesses Solution: Cache translations in TLB TLB hitSingle-Cycle Translation TLB miss Page-Table Walk to refill VPN offset V R W D tag PPN physical address PPN offset virtual address hit? (VPN = virtual page number) (PPN = physical page number)

39. 39 TLB Entries The TLB is a cache for page table entries (PTE) The data for a TLB entry ( == a PTE entry) –Physical page number (frame #) –Access rights (R/W bits) –Any other PTE information (dirty bit, LRU info, etc) The tags for a TLB entry –Physical page number Portion of it not used for indexing into the TLB –Valid bit –LRU bits If TLB is associative and LRU replacement is used

40. 40 TLB Misses If page is in memory –Load the PTE from memory and retry –Could be handled in hardware Can get complex for more complicated page table structures –Or in software Raise a special exception, with optimized handler This is what MIPS does using a special vectored interrupt If page is not in memory (page fault) –OS handles fetching the page and updating the page table –Then restart the faulting instruction

41. 41 TLB & Memory Hierarchies Once address is translated, it used to access memory hierarchy –A hierarchy of caches (L1, L2, etc)

42. 42 TLB and Cache Interaction Basic process –Use TLB to get PA –Use PA to access caches and DRAM Question: can you ever access the TLB and the cache in parallel?

43. 43 Page-Based Virtual-Memory Machine (Hardware Page-Table Walk) PC Inst. TLB Inst. Cache D Decode EM Data Cache W + Page Fault? Protection violation? Page Fault? Protection violation? Assumes page tables held in untranslated physical memory Data TLB Main Memory (DRAM) Memory Controller Physical Address Page-Table Base Register Virtual Address Physical Address Virtual Address Hardware Page Table Walker Miss?

44. 44 Virtual Memory Summary Use hard disk ( or Flash) as large storage for data of all programs –Main memory (DRAM) is a cache for the disk –Managed jointly by hardware and the operating system (OS) Each running program has its own virtual address space –Address space as shown in previous figure –Protected from other programs Frequently-used portions of virtual address space copied to DRAM –DRAM = physical address space –Hardware + OS translate virtual addresses (VA) used by program to physical addresses (PA) used by the hardware –Translation enables relocation & protection

45. 45 Acknowledgements These slides contain material from courses: –UCB CS152 –Stanford EE108B Read Book –Pages. 412-454