1. EENG449b/Savvides Lec 18.1 4/7/05 April 7, 2005 Prof. Andreas Savvides Spring 2005 http://www.eng.yale.edu/courses/2005s/een g449b EENG 449bG/CPSC 439bG Computer Systems Lecture 17 Memory Hierarchy Design Part I

2. EENG449b/Savvides Lec 18.2 4/7/05 CPU-DRAM Gap 1980: no cache in µproc; 1995 2-level cache on chip (1989 first Intel µproc with a cache on chip) Who Cares About the Memory Hierarchy?

3. EENG449b/Savvides Lec 18.3 4/7/05 Review of Caches Cache is the name given to the first level of the memory hierarchy encountered one the address leaves the CPU –Cache hit / cache miss when data is found / not found –Block – a fixed size collection of data containing the requested word –Spatial / temporal localities –Latency – time to retrieve the first word in the block –Bandwidth – time to retrieve the rest of the block –Address space is broken into fixed-size blocks called pages –Page fault – when CPU references something that is not on cache or main memory

4. EENG449b/Savvides Lec 18.4 4/7/05 Generations of Microprocessors Time of a full cache miss in instructions executed: 1st Alpha: 340 ns/5.0 ns = 68 clks x 2 or136 2nd Alpha:266 ns/3.3 ns = 80 clks x 4 or320 3rd Alpha:180 ns/1.7 ns =108 clks x 6 or648 1/2X latency x 3X clock rate x 3X Instr/clock  ­5X

5. EENG449b/Savvides Lec 18.5 4/7/05 Processor-Memory Performance Gap “Tax” Processor % Area %Transistors (­cost)(­power) Alpha 2116437%77% StrongArm SA11061%94% Pentium Pro64%88% –2 dies per package: Proc/I$/D$ + L2$ Caches have no “inherent value”, only try to close performance gap

6. EENG449b/Savvides Lec 18.6 4/7/05 What is a cache? Small, fast storage used to improve average access time to slow memory. Exploits spacial and temporal locality In computer architecture, almost everything is a cache! –Registers “a cache” on variables – software managed –First-level cache a cache on second-level cache –Second-level cache a cache on memory –Memory a cache on disk (virtual memory) –TLB a cache on page table –Branch-prediction a cache on prediction information? Proc/Regs L1-Cache L2-Cache Memory Disk, Tape, etc. BiggerFaster

7. EENG449b/Savvides Lec 18.7 4/7/05 Miss-oriented Approach to Memory Access: –CPI Execution includes ALU and Memory instructions Review: Cache performance Separating out Memory component entirely –AMAT = Average Memory Access Time –CPI ALUOps does not include memory instructions

8. EENG449b/Savvides Lec 18.8 4/7/05 Impact on Performance Suppose a processor executes at –Clock Rate = 200 MHz (5 ns per cycle), Ideal (no misses) CPI = 1.1 –50% arith/logic, 30% ld/st, 20% control Suppose that 10% of memory operations get 50 cycle miss penalty Suppose that 1% of instructions get same miss penalty CPI = ideal CPI + average stalls per instruction =1.1(cycles/ins) + [ 0.30 (DataMops/ins) x 0.10 (miss/DataMop) x 50 (cycle/miss)] + [ 1 (InstMop/ins) x 0.01 (miss/InstMop) x 50 (cycle/miss)] = (1.1 + 1.5 +.5) cycle/ins = 3.1 58% of the time the proc is stalled waiting for memory!

9. EENG449b/Savvides Lec 18.9 4/7/05 Traditional Four Questions for Memory Hierarchy Designers Q1: Where can a block be placed in the upper level? (Block placement) –Fully Associative, Set Associative, Direct Mapped Q2: How is a block found if it is in the upper level? (Block identification) –Tag/Block Q3: Which block should be replaced on a miss? (Block replacement) –Random, LRU Q4: What happens on a write? (Write strategy) –Write Back or Write Through (with Write Buffer)

10. EENG449b/Savvides Lec 18.10 4/7/05 Q1: Where can a Block Be Placed in a Cache?

11. EENG449b/Savvides Lec 18.11 4/7/05 Set Associatively Direct mapped = one-way set associative –(block address) MOD (Number of blocks in cache) Fully associative = set associative with 1 set –Block can be placed anywhere in the cache Set associative – a block can be placed in a restricted set of places –Block first mapped to a set and then placed anywhere in the set »(Block address) MOD (Number of sets in cache) –If there are n blocks in a set, then the cache is n-way set associative Most popular cache configurations in today’s processors –Direct mapped, 2-way set associative, 4-way set associative

12. EENG449b/Savvides Lec 18.12 4/7/05 Q2: How is a block found if it is in the cache? Selects the desired data from the block Selects the set Compared against for a hit If cache size remains the same increasing associativity increases The number of blocks per set => decrease index size and increase tag

13. EENG449b/Savvides Lec 18.13 4/7/05 Q3: Which Block Should be Replaced on a Cache Miss? Directly mapped cache –No choice – a single block is checked for a hit. If there is a miss, data is fetched into that block Fully associative and set associative –Random –Least Recently Used(LRU) – locality principles –First in, First Out(FIFO) - Approximates LRU

14. EENG449b/Savvides Lec 18.14 4/7/05 Q4: What Happens on a Write? Cache accesses dominated by reads E.g on MIPS 10% stores and 37% loads = 21% of cache traffic is writes Writes are much slower than reads –Block is read from cache at the same time a block is read and compared »If the write is a hit the block is passed to the CPU –Writing cannot begin unless the address is a hit Write through – information is written to both the cache and the lower-level memory Write back – information only written to cache. Written to memory only on block replacement –Dirty bit – used to indicate whether a block has been changed While in cache

15. EENG449b/Savvides Lec 18.15 4/7/05 Write Through vs. Write Back Write back – uses cache speed, all entries updated once during the writing of a block Write through – slower, BUT cache is always clean –Cache read misses never result in writes at the lower level –Next lower level of the cache has the most current copy of the data

16. EENG449b/Savvides Lec 18.16 4/7/05 Example: Alpha 21264 Data Cache 2-way set associative Write back Each block has 64 bytes of data –Offset points to the data we want –Total cache size 65,536 bytes Index 2 9 =512 points to the block Tag comparison determines if we have a hit Victim buffer to helps with write back

17. EENG449b/Savvides Lec 18.17 4/7/05 Address Breakdown Physical address is 44 bits wide, 38-bit block address and 6-bit offset (2^6=64) Calculating cache index size field Blocks are 64 bytes so offset needs 6 bits Tag size = 38 – 9 = 29 bits

18. EENG449b/Savvides Lec 18.18 4/7/05 Writing to cache If word to be written is in the cache, first 3 steps are the same as read The 21264 processor uses write back so the block cannot simply discarded on a miss If the “victim” was modified its data and address are sent to the victim buffer. Data cache cannot apply all processor needs –Separate Instruction and Data Caches may be needed

19. EENG449b/Savvides Lec 18.19 4/7/05 Unified vs Split Caches Unified vs Separate Instruction and Data caches Example: –16KB I&D: Inst miss rate=0.64%, Data miss rate=6.47% –32KB unified: Aggregate miss rate=1.99% –Using miss rate in the evaluation may be misleading! Which is better (ignore L2 cache)? –Assume 25% data ops  75% accesses from instructions (1.0/1.33) –hit time=1, miss time=50 –Note that data hit has 1 stall for unified cache (only one port) AMAT Harvard =75%x(1+0.64%x50)+25%x(1+6.47%x50) = 2.05 AMAT Unified =75%x(1+1.99%x50)+25%x(1+1+1.99%x50)= 2.24 Proc I-Cache-1 Proc Unified Cache-1 Unified Cache-2 D-Cache-1 Proc Unified Cache-2

20. EENG449b/Savvides Lec 18.20 4/7/05 Impact of Caches on Performance Consider a in-order execution computer –Cache miss penalty 100 clock cycles, CPI=1 –Average miss rate 2% and an average of 1.5 memory references per instruction –Average # of cache misses: 30 per 1000 instructions Performance with cache misses

21. EENG449b/Savvides Lec 18.21 4/7/05 Impact of Caches on Performance Calculate the performance using miss rate 4x increase in CPU time from “perfect cache” No cache – 1.0 + 100 x 1.5 = 151 – a factor of 40 compared to a system with cache Minimizing memory access does not always imply reduction in CPU time

22. EENG449b/Savvides Lec 18.22 4/7/05 How to Improve Cache Performance? Four main categories of optimizations 1.Reducing miss penalty - multilevel caches, critical word first, read miss before write miss, merging write buffers and victim caches 2.Reducing miss rate - larger block size, larger cache size, higher associativity, way prediction and pseudoassociativity and computer optimizations 2. Reduce the miss penalty or miss rate via parallelism - non-blocking caches, hardware prefetching and compiler prefetching 3. Reduce the time to hit in the cache - small and simple caches, avoiding address translation, pipelined cache access

23. EENG449b/Savvides Lec 18.23 4/7/05 Where to misses come from? Classifying Misses: 3 Cs –Compulsory —The first access to a block is not in the cache, so the block must be brought into the cache. Also called cold start misses or first reference misses. (Misses in even an Infinite Cache) –Capacity —If the cache cannot contain all the blocks needed during execution of a program, capacity misses will occur due to blocks being discarded and later retrieved. (Misses in Fully Associative Size X Cache) –Conflict —If block-placement strategy is set associative or direct mapped, conflict misses (in addition to compulsory & capacity misses) will occur because a block can be discarded and later retrieved if too many blocks map to its set. Also called collision misses or interference misses. (Misses in N-way Associative, Size X Cache)

24. EENG449b/Savvides Lec 18.24 4/7/05 3Cs Absolute Miss Rate (SPEC92) Conflict Miss rate

25. EENG449b/Savvides Lec 18.25 4/7/05 Cache Organization? Assume total cache size not changed: What happens if: 1)Change Block Size: 2)Change Associativity: 3) Change Compiler: Which of 3Cs is obviously affected?

26. EENG449b/Savvides Lec 18.26 4/7/05 Increasing Block Size Larger block sizes reduce compulsory misses –Takes advantage of spatial locality Larger blocks increase miss penalty –Our goal: reduce miss rate and miss penalty! Block size selection depends on latency and bandwidth of lower level memory –High latency & high BW => large block sizes –Low latency & low BW => smaller block sizes

27. EENG449b/Savvides Lec 18.27 4/7/05 Larger Block Size (fixed size&assoc) Reduced compulsory misses Increased Conflict Misses What else drives up block size?

28. EENG449b/Savvides Lec 18.28 4/7/05 Cache Size Old rule of thumb: 2x size => 25% cut in miss rate What does it reduce?

29. EENG449b/Savvides Lec 18.29 4/7/05 Increase Associativity Higher set-associativity improves miss rates Rule of thumb for caches –A direct mapped cache of size N has about the same miss rate as a two-way, set-associative cache of N/2 –Holds for < 128Kb Disadvantages –Reduces miss rate but increases miss penalty –Greater associativity can come at the cost of inreased hit time.

30. EENG449b/Savvides Lec 18.30 4/7/05 Associativity Conflict

31. EENG449b/Savvides Lec 18.31 4/7/05 3Cs Relative Miss Rate Conflict Flaws: for fixed block size Good: insight => invention

32. EENG449b/Savvides Lec 18.32 4/7/05 Associativity vs Cycle Time Beware: Execution time is only final measure! Why is cycle time tied to hit time? Will Clock Cycle time increase?

33. EENG449b/Savvides Lec 18.33 4/7/05 Next Time Cache Tradeoffs for Performance Reducing Hit Times