Cache - Optimization

Performance CPU time = (CPU execution cycles + Memory-stall cycles) x cycle time Memory-stall cycles = read-stall cycles + write-stall cycles Read-stall cycles = #reads x read miss rate x read miss penalty Write-stall cycles = (#writes x write miss rate x write miss penalty) + write buffer stalls (assume write-through cache) Assume the following, to make a simpler calculation In most write-through cache, read and write miss penalties are the same Write buffer stalls are negligible Memory-stall cycles = #memory accesses x miss rate x miss penalty = #misses x miss penalty

Cache Performance Example SPECInt2000 Miss rates = 2% (I-cache), 4% (D-cache ) CPI = 2 (without memory stall) Miss penalty = 100 cycles Memory accesses = 36% of instructions Speedup of a perfect cache over the above The above (assume I is the #instructions) CPU time (cycles) = I x 2 CPI + memory stall cycles memory stall cycles = I x 2% miss rate x 100 cycles + I x 36% x 4% miss rate x 100 cycles = 3.44 x I Thus, CPU time = 2 x I + 3.44 x I = 5.44 x I Perfect cache CPU time (cycles) = I x 2 CPI + 0 cycles = 2 x I Speedup = 5.44 / 2 = 2.72 Perfect cache would have made an almost three times faster system!

Performance on Increased Clock Rate Doubling the clock rate in the previous example Cycle time decreases to a half But miss penalty increases from 100 to 200 cycles Speedup of the fast clock system CPU time of the previous (slow) system Assume C is the cycle time of the slow clock system CPU time slow clock = 5.44 x I x C (seconds) CPU time of the faster clock system CPU time (cycles) = I x 2 CPI + memory stall cycles memory stall cycles = I x 2% miss rate x 200 cycles + I x 36% x 4% miss rate x 200 cycles = 6.88 x I CPU time fast clock = (2 x I + 6.88 x I) x (C x ½) = 4.44 x I x C (seconds) Speedup = 5.44 / 4.44 = 1.23 Not twice faster than the slow clock system!

Improving Cache Performance Increase the performance of cache Decrease the average access time Average access time = hit time + miss rate x miss penalty Reduce the components of average access time Hit time Miss rate Miss penalty

Associativity Flexible block placement data cache tag search: main Fully associative (any block) Direct mapped (12 mod 8) = 4 2-way set associative (12 mod 4) = 0 cache main memory set# 0 1 2 3 block# 0 1 2 3 4 5 6 7 memory address 12 search: 12 tag data

Set-Associative Cache Multiple blocks in a set Each memory block maps to a unique set first Within the set, can be placed in any cache block in the set N-way set associative cache: each set consists of N blocks Mapping of a block to cache: Select a set Set index = (Block number) modulo (#Sets in the cache) Parallel tag matching Parallel comparisons of tags in the selected set

Four-way Set Associative Cache More HW increases cost Parallel tag comparison Multiplexor Multiplexor increases Hit time Increased associativity Reduces conflicts  Reduces miss rates Requires more complex MUX  Increases hit time a set set index tag

Multiple-Word Blocks In a Set Two-way set associative cache with 32 byte long cache block memory block# 32 bytes valid tag cache block set 0: 1 … 31 … valid tag cache block valid tag cache block set 1: 32 33 … 63 valid tag cache block • • • valid tag cache block set 31: valid tag cache block 227-1 22 bits 5 bits 00001 31 tag set index block offset selected set Address Space

Accessing a Word Tag matching and word selection Must compare the tag in each valid block in the selected set (a), (b) Select a word by using a multiplexer (c) selected set (I): (a) The valid bit must be set for the matching block. 1 Y MUX 1 X w0 w1 w2 w3 w4 w5 w6 w7 (c) If (a) and (b), then cache hit, block offset selects a word = ? (b) The tag bits in one of the cache blocks must match the tag bits in the address = ? 22 bits 5 bits 5 bits X I 10000 31 tag set index block offset

Block Replacement Easy for direct mapped Single candidate to replace Set associative or fully associative Random is good enough with large caches LRU is better with small size and low associativity caches Associativity 2-way 4-way 8-way Size LRU Random 16 KB 5.18% 5.69% 4.67% 5.29% 4.39% 4.96% 64 KB 1.88% 2.01% 1.54% 1.66% 1.39% 1.53% 256 KB 1.15% 1.17% 1.13% 1.12%

Types of Cache Misses Compulsory (or cold start, first reference) misses First access to an address Misses even with an infinite sized cache Capacity misses Misses because cache is not large enough to fit working set e.g. Block replaced from cache and later accessed again Misses in fully associative cache of a certain size Conflict (or collision, interference) misses Misses due to not enough associativity e.g. Two addresses map to the same block in direct mapped cache Coherence misses (only in multi-processor cache) Misses caused by cache coherence

Example: Cache Associativity Four one-word blocks in caches The size of cache is 16 bytes ( = 4 x 4 bytes ) Compare three caches A direct-mapped cache A 2-way set associative cache A fully associative cache Find the number misses for the following accesses Reference sequence: 0, 8, 0, 6, 8

Example: Direct Mapped Cache Block placement: direct mapping 0 mod 4 = 0 6 mod 4 = 2 8 mod 4 = 0 Cache simulation 8 6 # of cache misses = 5 (3 compulsory and 2 conflict misses) block# 0 1 2 3 M[0] miss miss M[8] M[0] miss M[0] M[6] miss miss M[8] M[6]

Example: 2-way Set Associative Cache Two sets with two cache blocks 0 mod 2 = 0th set 6 mod 2 = 0th set 8 mod 2 = 0th set Replacement policy within a set: Least Recently Used (LRU) Cache simulation 8 6 # of cache misses = 4 (3 compulsory and 1 conflict misses) set# 0 1 M[0] miss miss M[0] M[8] M[0] M[8] hit M[0] M[6] miss miss M[8] M[6]

Example: Fully Associative Cache Four cache blocks in a single set Cache simulation 8 6 # of cache misses = 3 Only the compulsory misses in this case one set with all blocks M[0] miss M[0] M[8] miss M[0] M[8] hit M[0] M[8] M[6] miss M[0] M[8] M[6] hit

Cache Parameters Cache size, block size, associativity Larger cache Affect the cache performance (hit time and miss rate) Need to select appropriate ones Larger cache Obvious way to reduce capacity misses But, higher cost and longer hit time Larger block size Spatial locality reduces compulsory misses (prefetch effect) But, large block size will increase conflict misses Higher associativity Reduce conflict misses But, large tag matching logic will increase the hit time

Block Size vs. Miss Rate Need appropriate block size Fixed size & associativity Increased conflict misses Popular choices Reduced compulsory misses

Miss Rates from 3 Types of Misses Associativity matters High associativity reduces miss rates for the same size caches SPEC2000 Conflict Capacity Compulsory 1-way 2-way 4-way 8-way Miss rate Per type

Multilevel Caches Reduce miss penalty Second level cache reduces the miss penalty of the first cache Optimize each cache for different constraints Exploit cost/capacity trade-offs at different levels L1 cache (first level cache, primary cache) Provides fast access time (1~3 CPU cycles) Built on the same chip as processors 8~64 KB, direct-mapped ~ 4-way set-associative L2 cache (second level cache, secondary cache) Provides low miss penalty (20~30 CPU cycles) Can be on the same chip or off-chip in a separate set of SRAMs 256KB~4MB, 4~16-way set-associative Pentium IV L1 and L2 caches on the same chip

Summary Performance with cache Reduce hit time Reduce miss rate from Reduce average access time (= hit time + miss rate x miss penalty) Reduce hit time Small cache with low associativity (direct-mapped cache) Reduce miss rate from Capacity misses  use large cache ( could increase hit-time) Compulsory misses  use large sized block ( could increase conflict) Conflict misses  use high associativity ( could increase hit-time) Select appropriate cache size, block size, and associativity Reduce miss penalty Multilevel cache reduces miss penalty for the primary cache