1. claudio.talarico@mail.ewu.edu1 Computing Systems Memory Hierarchy

2. 2 Memory: characteristics  Location  on-chip, off-chip  Cost  Dollars per bit  Performance  access time, cycle time, transfer rate  Capacity  word size, number of words  Unit of transfer  word, block  Access  sequential, random, associative  Alterability  read/write, read only (ROM)  Storage type  static, dynamic, volatile, nonvolatile  Physical type  semiconductor, magnetic, optical

3. 3 Memory: array organization  Storage cells are organized on a rectangular array  The address is divided in row and column parts  There are M = 2 r rows of N bits each  M and N are usually powers of 2  Total size of a memory chip = M x N bits  The row address (r bits) select a full row of N bits  The column address (c bits) selects k bits out of N  The memory is organized as T=2 r+c addresses of k-bit words

4. 4 Memory: array organization Example: 4 MB memory

5. 5 Memories  Static Random Access Memory (SRAM):  value is stored on a pair of inverting gates  very fast but takes up more space than DRAM Dynamic Random Access Memory (DRAM):  value is stored as a charge on capacitor (must be refreshed)  very small but slower than SRAM (factor of 5 to 10)

6. 6 SRAM structure

7. 7 SRAM internal snapshot

8. 8 Exploiting memory hierarchy  Users want large and fast memories !  SRAM access times are 0.5-5ns at cost of $4000 to $10000 per GB.  DRAM access times are 50-70ns at cost of $100 to $200 per GB.  Disk access times are 5 to 20 ms at cost of $0.50 to $2 per GB.  Give them the illusion !  build a memory hierarchy

9. 9 Basic structure of a memory hierarchy

10. 10 Locality  A principle that makes having a memory hierarchy a good idea  Locality principle states that programs access a relatively small portion of their address space at any instant of time  If an item is referenced,  temporal locality: it will tend to be referenced again soon  spatial locality: nearby items will tend to be referenced soon. Why does code have locality?  A memory hierarchy can consists of multiple levels, but …  data is copied between only two adjacent levels at the time  Our focus: two levels (upper, lower)  Block (or line): minimum unit of data  hit: data requested is in the upper level  miss: data requested is not in the upper level

11. 11 Cache  Two issues:  How do we know if a data item is in the cache?  if it is, how do we find it?  The simplest solution:  block size is one word of data  "direct mapped" for each item of data at the upper level (Mp), there is exactly one location in the cache where it might be. i.e., lots of items at the lower level (cache) share locations in the upper level (Mp)

12. 12 Direct mapped cache  Mapping: (block address) modulo (number of blocks in the cache)

13. 13 Direct mapped cache for MIPS What kind of locality are we taking advantage of ? Temporal

14. 14 Direct mapped cache  Taking advantage of spatial locality (multiword cache block):

15. 15 Example: bits in a cache  Assuming a 32 bit address, how many total bits are required for a direct-mapped cache with 16 KB of data and blocks composed of 4 words each ?  We have 2 12 words of data in cache:  Each block (=line) of data cache is composed of 4 words.  Assuming memory is word-addressed (MIPS), we need only 30 bits of the 32-bit address issued by the CPU  Each block has: 4 x 32 = 128 bits of data, 1 valid bit, and 30–10–2 = 18 bits of tag  Thus:  We need 15% more bits than needed for the storage of data (18.375 KB / 16 KB = 1.15)...2 10 lines

16. 16 Example: mapping an address to a multiword cache block  Consider a cache with 64 blocks and a block size of 16 bytes. What block number does byte address 1200 map to ?  byte address 1200 is word address: 1200 / 4 = 300  word address 300 is block address: 300 / 4 = 75  cache address is: 75 modulo 64 = 11 64 blocks …

17. 17 Hits vs. misses  Read hits  this is what we want !  Read misses  stall the CPU, fetch block from memory, deliver to cache, restart  Write hits:  can replace data in cache and memory (write-through)  write the data only into the cache (write-back to memory later)  Write misses:  read the entire block into the cache, then write the word

18. 18 Performance Simplified model: execution time = (execution cycles + stall cycles)  cycle time stall cycles = # of memory accesses  miss rate  miss penalty Two ways of improving performance:  decreasing the miss rate  decreasing the miss penalty What happens if we increase block size?

19. 19 Performance  Increasing the block size “usually” decrease the miss rate  However, if the block size becomes a significant portion of the cache  the number of blocks that can be held in cache will become small   there will be a great competition for those few blocks   the benefit in the miss rate become smaller

20. 20 Performance  Increasing the block size, increases the miss penalty  Miss penalty is determined by:  time to fetch the block from the next lower level of the hierarchy and load it into the cache  It has two parts:  latency to the first word (it will not change with block size), and  transfer time for the rest of the block  To decrease miss penalty:  common solutions:  the bandwidth of main memory must be increased to transfer cache blocks more efficiently  making memory wider  Interleaving  advanced solutions:  Early restart  Critical word first

21. 21 Example: Cache Performance  cache miss rate for a program (= instruction cache miss rate) 3%  data cache miss rate 4%  processor CPI of 2.0  miss penalty 100 cycles for all kind of misses  load/store instruction frequency 36%  How much faster a processor would run with a perfect cache that never miss ?  instruction miss cycles = IC x 3% x 100 = IC x 3 cycles  data miss cycles = IC x 36% x 4% x 100 = IC x 1.44 cycles  memory-stall-cycles = IC x 3 + IC x 1.44 = IC x 4.44 cycles  execution cycles with stalls = CPU cycles + memory-stall-cycles = = IC x 2.0 + IC x 4.44 = 6.44 cycles  The amount of execution time spent on memory stalls is 69% (= 4.44/6.44)

22. 22 Example: Cache Performance  What is the impact if we reduce the CPI of the previous computer from 2 to 1 ?  CPI perfect = 1 cycle / instruction  CPI with stalls = 1 + 4.44 = 5.44 cycle / instruction  The amount of execution time spent on memory stalls is 82% (= 4.44/5.44)  What if we keep the same CPI but we double the CPU clock rate ?  measured in the faster clock cycles, the new miss penalty will double (200 cycles)  stall-cycles per instruction = 3% x 200 + 36% x 4% x 200 = 8.88 cycles / instruction  execution cycles per instruction with stalls = 2.0 + 8.88 = 10.88 cycles / instruction Because of the cache, doubling clock rate makes the computer only 1.18 times faster instead of 2 times faster (Amdahl’s Law) we stall 82% (8.88/10.88) of the time

23. 23 Designing the memory system to support caches Increase the physical width of the memory system Increase the logical width of the memory system additional silicon area potential increase in cache access time Instead of making the entire path between memory and cache wider we organize the memory chips in banks

24. 24 Example: higher memory bandwidth  Cache block of 4 words  1 memory bus clock cycle to send the address  15 memory bus clock cycles for each DRAM access initiated  1 memory clock cycle to send a word  One-word memory ? miss penalty = 1+ 4 x 15 + 4 x 1 = 65 memory cycles bandwidth = 4x4 / 65 = 0.25 bytes / memory cycle  Two-word memory ? miss penalty = 1 + 2 x 15 + 2 x 1 = 33 memory cycles bandwidth = 4x4 / 33 = 0.48 bytes / memory cycle  Four-word memory ? miss penalty = 1 + 15 + 1 = 17 memory cycles bandwidth = 4x4 / 17 = 0.96 bytes / memory cycle  Interleaving with four banks ? miss penalty = 1 + 1 x 15 + 4 x 1 = 20 memory cycles bandwidth = 4x4 / 20 = 0.80 bytes / memory cycle

25. 25 Decreasing miss ratio with associativety  Basic idea:  use a more flexible scheme for placing blocks  Direct mapping  a memory block can be placed in exactly one location in cache  Fully associative  a block can be placed in any location in the cache  to find a given block in cache, all the entries must be searched  set associative (n-way)  there is a fixed number of locations n (with n at least 2) where each block can be placed  all entries in a set must be searched  A direct mapping cache can be thought as a one-way associative cache  A fully associative cache with m-entries can be thought as an m-way set associative cache  block replacement policy: the most commonly used is LRU

26. 26 Decreasing miss ratio with associativety

27. 27 An implementation Costs: extra comparators and mux extra tag and valid bits delay imposed by having to do the compare and the muxing set 4-way set associative cache

28. 28 Choosing cache types

29. 29 Example: misses and associativety  Assume 3 small caches, each consisting of four one-word blocks. One cache is fully associative, a second two-way set associative, and the third is direct mapped.  Find the number of misses for each cache given the following sequence of block addresses: 0,8,0,6,8. Assume LRU replacement policy  Direct mapped  0 modulo 4 = 0 8 modulo 4 = 0 6 modulo 4 = 2 addressHit or miss Content of cache blocks 0123 0missMem[0] 8missMem[8] 0missMem[0] 6missMem[0]Mem[6] 8missMem[8]Mem[6] 5 miss for 5 accesses

30. 30 Example: misses and associativety  Two-way set associative  0 modulo 2 = 0 8 modulo 2 = 0 6 modulo 2 = 0  Full associative addressHit or miss Content of cache blocks Set 0 Set 1 0missMem[0] 8missMem[0]Mem[8] 0hitMem[0]Mem[8] 6missMem[0]Mem[6] 8missMem[8]Mem[6] 4 miss 1 hit addressHit or miss Content of cache blocks 0123 0missMem[0] 8missMem[0]Mem[8] 0hitMem[0]Mem[8] 6missMem[0]Mem[8]Mem[6] 8hitMem[0]Mem[8]Mem[6] 3 miss, 2 hit

31. 31 Reducing miss penalty with multi-level caches  Add a second level cache:  usually primary cache is on the same chip as the processor  use SRAMs to add another cache above main memory (DRAM)  miss penalty goes down if data is in 2nd level cache rather then main memory  Using multilevel caches:  Focus on minimizing the hit time on the 1st level cache  Focus on minimizing the miss rate on the 2nd level cache  Primary cache often uses a smaller block size (access time is more critical)  Secondary cache, is often 10 or more times larger than the primary

32. 32 Example: performance of multilevel caches System A  CPU with base CPI = 1.0  Clock rate 5 GHz ( Tclock = 0.2 ns)  Main-Memory access time 100 ns  Miss rate per instruction in Cache-1 is 2% System B  Let’s add Cache-2  reduce the miss rate to Main-Memory to 0.5%  Cache-2 access time 5ns  How faster is System B ?

33. 33 Cache complexities  Not always easy to understand implications of caches: Radix sort Quicksort Size (K items to sort) 0 481632 200 400 600 800 1000 1200 64128256512102420484096 Radix sort Quicksort Size (K items to sort) 0 481632 400 800 1200 1600 2000 64128256512102420484096 Theoretical behavior of Radix sort vs. Quicksort Observed behavior of Radix sort vs. Quicksort instructions / item clock cycles / item

34. 34  Here is why:  Memory system performance is often critical factor  multilevel caches, pipelined processors, make it harder to predict outcomes  Compiler optimizations to increase locality sometimes hurt ILP  Difficult to predict best algorithm: need experimental data Cache complexities