1Chapter 7 Memory Hierarchies Outline of Lectures on Memory Systems 1. Memory Hierarchies 2. Cache Memory 3. Virtual Memory 4. The future

2Chapter 7 Review: Major Components of a Computer Processor Control Datapath Memory Devices Input Output

3Chapter 7 Processor-Memory Performance Gap “Moore’s Law” µProc 55%/year (2X/1.5yr) DRAM 7%/year (2X/10yrs) Processor-Memory Performance Gap (grows 50%/year)

4Chapter 7 The “Memory Wall” Logic vs DRAM speed gap continues to grow Clocks per instruction Clocks per DRAM access

5Chapter 7 The Memory Hierarchy Goal Fact: Large memories are slow and fast memories are small How do we create a memory that gives the illusion of being large, cheap and fast (most of the time)? –With hierarchy –With parallelism

6Chapter 7 Second Level Cache (SRAM) A Typical Memory Hierarchy Control Datapath Secondary Memory (Disk) On-Chip Components RegFile Main Memory (DRAM) Data Cache Instr Cache ITLB DTLB eDRAM Speed (ck cycles): ½’s 1’s 10’s 100’s 1,000’s Size (bytes): 100’s K’s 10K’s M’s G’s to T’s Cost: highest lowest  By taking advantage of the principle of locality l Can present the user with as much memory as is available in the cheapest technology l at the speed offered by the fastest technology

7Chapter 7 Characteristics of the Memory Hierarchy Increasing distance from the processor in access time L1$ L2$ Main Memory Secondary Memory Processor (Relative) size of the memory at each level Inclusive– what is in L1$ is a subset of what is in L2$ is a subset of what is in MM that is a subset of is in SM 4-8 bytes (word) 1 to 4 blocks 1,024+ bytes (disk sector = page) 8-32 bytes (block)

8Chapter 7 Memory Hierarchy Technologies Caches use SRAM for speed and technology compatibility –Low density (6 transistor cells), high power, expensive, fast –Static: content will last “forever” (until power turned off) Main Memory uses DRAM for size (density) –High density (1 transistor cells), low power, cheap, slow –Dynamic: needs to be “refreshed” regularly (~ every 8 ms) »1% to 2% of the active cycles of the DRAM –Addresses divided into 2 halves (row and column) »RAS or Row Access Strobe triggering row decoder »CAS or Column Access Strobe triggering column selector Dout[15-0] SRAM 2M x 16 Din[15-0] Address Chip select Output enable Write enable 16 21

9Chapter 7 Memory Performance Metrics Latency: Time to access one word –Access time: time between the request and when the data is available (or written) –Cycle time: time between requests –Usually cycle time > access time –Typical read access times for SRAMs in 2004 are 2 to 4 ns for the fastest parts to 8 to 20ns for the typical largest parts Bandwidth: How much data from the memory can be supplied to the processor per unit time –width of the data channel * the rate at which it can be used Size: DRAM to SRAM ­ 4 to 8 Cost/Cycle time: SRAM to DRAM ­ 8 to 16

10Chapter 7 Classical RAM Organization (~Square) RowDecoderRowDecoder row address data bit or word RAM Cell Array word (row) line bit (data) lines Each intersection represents a 6-T SRAM cell or a 1-T DRAM cell Column Selector & I/O Circuits column address One memory row holds a block of data, so the column address selects the requested bit or word from that block

11Chapter 7 data bit Classical DRAM Organization (~Square Planes) RowDecoderRowDecoder row address Column Selector & I/O Circuits column address data bit word (row) line bit (data) lines Each intersection represents a 1-T DRAM cell The column address selects the requested bit from the row in each plane data word... RAM Cell Array

12Chapter 7 Classical DRAM Operation DRAM Organization: –N rows x N column x M-bit –Read or Write M-bit at a time –Each M-bit access requires a RAS / CAS cycle Row Address CAS RAS Col AddressRow AddressCol Address N rows N cols DRAM M bit planes Row Address Column Address M-bit Output 1 st M-bit Access 2 nd M-bit Access Cycle Time

13Chapter 7 N rows N cols DRAM Column Address M-bit Output M bit planes N x M SRAM Row Address Page Mode DRAM Operation Page Mode DRAM –N x M SRAM to save a row  After a row is read into the SRAM “register” l Only CAS is needed to access other M-bit words on that row l RAS remains asserted while CAS is toggled Row Address CAS RAS Col Address 1 st M-bit Access2 nd M-bit3 rd M-bit4 th M-bit Cycle Time

14Chapter 7 N rows N cols DRAM Column Address M-bit Output M bit planes N x M SRAM Row Address Synchronous DRAM (SDRAM) Operation  After a row is read into the SRAM register l Inputs CAS as the starting “burst” address along with a burst length l Transfers a burst of data from a series of sequential addresses within that row - A clock controls transfer of successive words in the burst – 300MHz in Row Address CAS RAS Col Address 1 st M-bit Access2 nd M-bit3 rd M-bit4 th M-bit Cycle Time Row Add

15Chapter 7 Other DRAM Architectures Double Data Rate SDRAMs – DDR-SDRAMs (and DDR-SRAMs) –Double data rate because they transfer data on both the rising and falling edge of the clock –Are the most widely used form of SDRAMs DDR2-SDRAMs

16Chapter 7 DRAM Memory Latency & Bandwidth Milestones In the time that the memory to processor bandwidth doubles, the memory latency improves by a factor of only 1.2 to 1.4 To deliver such high bandwidth, the internal DRAM has to be organized as interleaved memory banks DRAMPage DRAM FastPage DRAM Synch DRAM DDR SDRAM Module Width16b 32b64b Year Mb/chip Die size (mm 2 ) Pins/chip BWidth (MB/s) Latency (nsec) Patterson, CACM Vol 47, #10, 2004

17Chapter 7 The off-chip interconnect and memory architecture can affect overall system performance in dramatic ways Memory Systems that Support Caches CPU Cache Memory bus One word wide organization (one word wide bus and one word wide memory)  Assume 1. 1 clock cycle to send the address clock cycles for DRAM cycle time, 8 clock cycles access time 3. 1 clock cycle to return a word of data  Memory-Bus to Cache bandwidth number of bytes accessed from memory and transferred to cache/CPU per clock cycle 32-bit data & 32-bit addr per cycle on-chip

18Chapter 7 One Word Wide Memory Organization CPU Cache Memory bus on-chip If the block size is one word, then for a memory access due to a cache miss, the pipeline will have to stall the number of cycles required to return one data word from memory cycle to send address cycles to read DRAM cycle to return data total clock cycles miss penalty Number of bytes transferred per clock cycle (bandwidth) for a single miss

19Chapter 7 One Word Wide Memory Organization CPU Cache Memory bus on-chip If the block size is one word, then for a memory access due to a cache miss, the pipeline will have to stall the number of cycles required to return one data word from memory cycle to send address cycles to read DRAM cycle to return data total clock cycles miss penalty Number of bytes transferred per clock cycle (bandwidth) for a single miss is bytes per clock /27 = 0.148

20Chapter 7 One Word Wide Memory Organization, con’t CPU Cache Memory bus on-chip What if the block size is four words? cycle to send 1 st address cycles to read DRAM cycles to return last data word total clock cycles miss penalty Number of bytes transferred per clock cycle (bandwidth) for a single miss is bytes per clock 25 cycles 1 4 x 25 = (4 x 4)/102 = 0.157

21Chapter 7 One Word Wide Memory Organization, con’t CPU Cache Memory bus on-chip What if the block size is four words and if a fast page mode DRAM is used? cycle to send 1 st address cycles to read DRAM cycles to return last data word total clock cycles miss penalty Number of bytes transferred per clock cycle (bandwidth) for a single miss is bytes per clock 25 cycles 8 cycles *8 = (4 x 4)/51 = 0.314

22Chapter 7 Interleaved Memory Organization  For a block size of four words cycle to send 1 st address cycles to read DRAM cycles to return last data word total clock cycles miss penalty CPU Cache Memory bank 1 bus on-chip Memory bank 0 Memory bank 2 Memory bank 3  Number of bytes transferred per clock cycle (bandwidth) for a single miss is bytes per clock 25 cycles (4 x 4)/30 = =

23Chapter 7 DRAM Memory System Summary Its important to match the cache characteristics –caches access one block at a time (usually more than one word) with the DRAM characteristics –use DRAMs that support fast multiple word accesses, preferably ones that match the block size of the cache with the memory-bus characteristics –make sure the memory-bus can support the DRAM access rates and patterns –with the goal of increasing the Memory-Bus to Cache bandwidth

24Chapter 7 Review: The Memory Hierarchy Increasing distance from the processor in access time L1$ L2$ Main Memory Secondary Memory Processor (Relative) size of the memory at each level Inclusive– what is in L1$ is a subset of what is in L2$ is a subset of what is in MM that is a subset of is in SM 4-8 bytes (word) 1 to 4 blocks 1,024+ bytes (disk sector = page) 8-32 bytes (block)  Take advantage of the principle of locality to present the user with as much memory as is available in the cheapest technology at the speed offered by the fastest technology

25Chapter 7 The Memory Hierarchy: Why Does it Work? Temporal Locality (Locality in Time):  Keep most recently accessed data items closer to the processor Spatial Locality (Locality in Space):  Move blocks consisting of contiguous words to the upper levels Lower Level Memory Upper Level Memory To Processor From Processor Blk X Blk Y

26Chapter 7 The Memory Hierarchy: Terminology Hit: data is in some block in the upper level (Blk X) –Hit Rate: the fraction of memory accesses found in the upper level –Hit Time: Time to access the upper level which consists of RAM access time + Time to determine hit/miss Miss: data is not in the upper level so needs to be retrieved from a block in the lower level (Blk Y) –Miss Rate = 1 - (Hit Rate) –Miss Penalty: Time to replace a block in the upper level + Time to deliver the block to the processor –Hit Time << Miss Penalty Lower Level Memory Upper Level Memory To Processor From Processor Blk X Blk Y

27Chapter 7 How is the Hierarchy Managed? registers  memory –by compiler (programmer?) cache  main memory –by the cache controller hardware main memory  disks –by the operating system (virtual memory) –virtual to physical address mapping assisted by the hardware (TLB) –by the programmer (files)

28Chapter 7 Two questions to answer (in hardware): –Q1: How do we know if a data item is in the cache? –Q2: If it is, how do we find it? Direct mapped –For each item of data at the lower level, there is exactly one location in the cache where it might be - so lots of items at the lower level must share locations in the upper level –Address mapping: (block address) modulo (# of blocks in the cache) –First consider block sizes of one word Cache

29Chapter 7 Caching: A Simple First Example Cache 0000xx 0001xx 0010xx 0011xx 0100xx 0101xx 0110xx 0111xx 1000xx 1001xx 1010xx 1011xx 1100xx 1101xx 1110xx 1111xx Main Memory TagData Q1: Is it there? Compare the cache tag to the high order 2 memory address bits to tell if the memory block is in the cache Valid Two low order bits define the byte in the word (32-b words) Q2: How do we find it? Use next 2 low order memory address bits – the index – to determine which cache block (i.e., modulo the number of blocks in the cache) (block address) modulo (# of blocks in the cache) Index

30Chapter 7 Caching: A Simple First Example Cache Main Memory Q2: How do we find it? Use next 2 low order memory address bits – the index – to determine which cache block (i.e., modulo the number of blocks in the cache) TagData Q1: Is it there? Compare the cache tag to the high order 2 memory address bits to tell if the memory block is in the cache Valid 0000xx 0001xx 0010xx 0011xx 0100xx 0101xx 0110xx 0111xx 1000xx 1001xx 1010xx 1011xx 1100xx 1101xx 1110xx 1111xx Two low order bits define the byte in the word (32b words) (block address) modulo (# of blocks in the cache) Index

31Chapter 7 Direct Mapped Cache Consider the main memory word reference string Start with an empty cache - all blocks initially marked as not valid

32Chapter 7 Direct Mapped Cache Consider the main memory word reference string Mem(0) 00 Mem(1) 00 Mem(0) 00 Mem(1) 00 Mem(2) miss hit 00 Mem(0) 00 Mem(1) 00 Mem(2) 00 Mem(3) 01 Mem(4) 00 Mem(1) 00 Mem(2) 00 Mem(3) 01 Mem(4) 00 Mem(1) 00 Mem(2) 00 Mem(3) 01 Mem(4) 00 Mem(1) 00 Mem(2) 00 Mem(3) Mem(1) 00 Mem(2) 00 Mem(3) Start with an empty cache - all blocks initially marked as not valid –8 requests, 6 misses

33Chapter 7 One word/block, cache size = 1K words MIPS Direct Mapped Cache Example 20 Tag 10 Index Data IndexTagValid Byte offset What kind of locality are we taking advantage of? 20 Data 32 Hit Temporal!

34Chapter 7 Read hits (I$ and D$) –this is what we want! Write hits (D$ only) –allow cache and memory to be inconsistent »write the data only into the cache block (write-back the cache contents to the next level in the memory hierarchy when that cache block is “evicted”) »need a dirty bit for each data cache block to tell if it needs to be written back to memory when it is evicted –require the cache and memory to be consistent »always write the data into both the cache block and the next level in the memory hierarchy (write-through) so don’t need a dirty bit »writes run at the speed of the next level in the memory hierarchy – so slow! – or can use a write buffer, so only have to stall if the write buffer is full Handling Cache Hits

35Chapter 7 Write Buffer for Write-Through Caching Write buffer between the cache and main memory –Processor: writes data into the cache and the write buffer –Memory controller: writes contents of the write buffer to memory The write buffer is just a FIFO –Typical number of entries: 4 –Works fine if store frequency (w.r.t. time) << 1 / DRAM write cycle Memory system designer’s nightmare –When the store frequency (w.r.t. time) → 1 / DRAM write cycle leading to write buffer saturation »One solution is to use a write-back cache; another is to use an L2 cache (next lecture) Processor Cache write buffer DRAM

36Chapter 7 Review: Why Pipeline? For Throughput! I n s t r. O r d e r Time (clock cycles) Inst 0 Inst 1 Inst 2 Inst 4 Inst 3 ALU I$ Reg D$Reg ALU I$ Reg D$Reg ALU I$ Reg D$Reg ALU I$ Reg D$Reg ALU I$ Reg D$Reg To keep the pipeline running at its maximum rate both I$ and D$ need to satisfy a request from the datapath every cycle. What happens when they can’t do that? To avoid a structural hazard need two caches on-chip: one for instructions (I$) and one for data (D$)

37Chapter 7 Another Reference String Mapping Consider the main memory word reference string Start with an empty cache - all blocks initially marked as not valid

38Chapter 7 Another Reference String Mapping Consider the main memory word reference string miss 00 Mem(0) Mem(4) Mem(0) Mem(0) Mem(0) Mem(4) Mem(4) 0 00 Start with an empty cache - all blocks initially marked as not valid Ping pong effect due to conflict misses - two memory locations that map into the same cache block –8 requests, 8 misses

39Chapter 7 Sources of Cache Misses Compulsory (cold start or process migration, first reference): –First access to a block, “cold” fact of life, not a whole lot you can do about it –If you are going to run “millions” of instruction, compulsory misses are insignificant Conflict (collision): –Multiple memory locations mapped to the same cache location –Solution 1: increase cache size –Solution 2: increase associativity Capacity: –Cache cannot contain all blocks accessed by the program –Solution: increase cache size

40Chapter 7 Handling Cache Misses Read misses (I$ and D$) –stall the entire pipeline, fetch the block from the next level in the memory hierarchy, install it in the cache and send the requested word to the processor, then let the pipeline resume Write misses (D$ only) 1.stall the pipeline, fetch the block from next level in the memory hierarchy, install it in the cache (which may involve having to evict a dirty block if using a write-back cache), write the word from the processor to the cache, then let the pipeline resume 2.Write allocate – just write the word into the cache updating both the tag and data, no need to check for cache hit, no need to stall (normally used in write-back caches) 3.No-write allocate – skip the cache write and just write the word to the write buffer (and eventually to the next memory level), no need to stall if the write buffer isn’t full; must invalidate the cache block since it will be inconsistent (now holding stale data) No-write allocate is normally used in write-through caches with a write buffer)

41Chapter 7 Multiword Block Direct Mapped Cache 8 Index Data IndexTagValid Byte offset 20 Tag HitData 32 Block offset Four words/block, cache size = 1K words What kind of locality are we taking advantage of?

42Chapter 7 Taking Advantage of Spatial Locality 0 Let cache block hold more than one word Start with an empty cache - all blocks initially marked as not valid

43Chapter 7 Taking Advantage of Spatial Locality 0 Let cache block hold more than one word Mem(1) Mem(0) miss 00 Mem(1) Mem(0) hit 00 Mem(3) Mem(2) 00 Mem(1) Mem(0) miss hit 00 Mem(3) Mem(2) 00 Mem(1) Mem(0) miss 00 Mem(3) Mem(2) 00 Mem(1) Mem(0) hit 00 Mem(3) Mem(2) 01 Mem(5) Mem(4) hit 00 Mem(3) Mem(2) 01 Mem(5) Mem(4) 00 Mem(3) Mem(2) 01 Mem(5) Mem(4) miss Start with an empty cache - all blocks initially marked as not valid –8 requests, 4 misses

44Chapter 7 Miss Rate vs Block Size vs Cache Size Miss rate goes up if the block size becomes a significant fraction of the cache size because the number of blocks that can be held in the same size cache is smaller (increasing capacity misses)

45Chapter 7 Block Size Tradeoff –Larger block size means larger miss penalty »Latency to first word in block + transfer time for remaining words Miss Penalty Block Size Miss Rate Exploits Spatial Locality Fewer blocks compromises Temporal Locality Block Size Average Access Time Increased Miss Penalty & Miss Rate Block Size  In general, Average Memory Access Time = Hit Time + Miss Penalty x Miss Rate Larger block sizes take advantage of spatial locality but –If the block size is too big relative to the cache size, the miss rate will go up

46Chapter 7 Multiword Block Considerations Read misses (I$ and D$) –Processed the same as for single word blocks – a miss returns the entire block from memory –Miss penalty grows as block size grows »Early restart – datapath resumes execution as soon as the requested word of the block is returned »Requested word first – requested word is transferred from the memory to the cache (and datapath) first –Nonblocking cache – allows the datapath to continue to access the cache while the cache is handling an earlier miss Write misses (D$) –Can’t use write allocate or will end up with a “garbled” block in the cache (e.g., for 4 word blocks, a new tag, one word of data from the new block, and three words of data from the old block), so must fetch the block from memory first and pay the stall time

47Chapter 7 Cache Summary The Principle of Locality: –Program likely to access a relatively small portion of the address space at any instant of time »Temporal Locality: Locality in Time »Spatial Locality: Locality in Space Three major categories of cache misses: –Compulsory misses: sad facts of life. Example: cold start misses –Conflict misses: increase cache size and/or associativity Nightmare Scenario: ping pong effect! –Capacity misses: increase cache size Cache design space –total size, block size, associativity (replacement policy) –write-hit policy (write-through, write-back) –write-miss policy (write allocate, write buffers)