1. Lecture 7 Memory Hierarchy and Cache Design
Pradondet Nilagupta
( Based on notes by Robert F. Hodson -- CNU)
( Based on notes by Prof. Mike Schulte)

2. Why is memory important?
Processor performance has increased at a much faster rate than memory performance, making main memory the bottleneck.
CPU-DRAM Gap
1980: no cache in mproc; level cache, 60% trans. on Alpha mproc

3. General Principles of Memory
Locality
Temporal Locality: referenced memory is likely to be referenced again soon (e.g. code within a loop)
Spatial Locality: memory close to referenced memory is likely to be referenced soon (e.g., data in a sequentially access array)
Locality + smaller HW is faster = memory hierarchy
Levels: each smaller, faster, more expensive/byte than level below
Inclusive: data found in top also found in the bottom
Definitions
Upper: closer to processor
Block: minimum unit that present or not in upper level
Block address: location of block in memory
Hit time: time to access upper level, including hit determination

4. Memory Hierarchy Secondary Storage Disks DRAM Memory Hierarchy
L2 Cache
L1 Cache
Processor
Registers

5. Differences in Memory Levels

6. Cache: A definition Webster’s Dictionary : In CS (originally)
cache (kash), a safe place for hiding or storing things
In CS (originally)
the first level of memory hierarchy encounter after the CPU
In CS today (typically)
a term applied to any level of buffering employed to reuse commonly accessed items

7. Cache Measures Hit rate: fraction found in that level
So high that usually talk about Miss rate
Miss rate fallacy: as MIPS to CPU performance, miss rate to average memory access time in memory
Average memory-access time = Hit rate x Hit access time Miss rate x Miss penalty
Miss penalty: time to replace a block from lower level, including time to replace in CPU
access time: time to lower level = ƒ(lower level latency)
transfer time: time to transfer block = ƒ(BW upper & lower, block size)

8. Block Size vs. Cache Measures
Increasing Block Size generally increases Miss Penalty and decreases Miss Rate
Miss
Penalty
Miss
Rate
Avg.
Memory
Access
Time X =
Block Size
Block Size
Block Size

9. Implications For CPU Fast hit check since every memory access
Hit is the common case
Unpredictable memory access time
10s of clock cycles: wait
1000s of clock cycles:
Interrupt & switch & do something else
New style: multithreaded execution
How to handle miss (10s of cycles=> HW, s of cycles => SW)?

10. Four Questions for Memory Hierarchy Designers
Q1: Where can a block be placed in the upper level? (Block placement)
Q2: How is a block found if it is in the upper level? (Block identification)
Q3: Which block should be replaced on a miss? (Block replacement)
Q4: What happens on a write? (Write strategy)

11. Q1: Where can a block be placed in the upper level?
Direct Mapped: Each block has only one place that it can appear in the cache.
Fully associative: Each block can be placed anywhere in the cache.
Set associative: Each block can be placed in a restricted set of places in the cache.
If there are n blocks in a set, the cache placement is called n-way set associative
What is the associativity of a direct mapped cache?

12. Associativity Examples (Figure 5.2, pg. 376)
Fully associative:
Block 12 can go anywhere
Direct mapped:
Block no. = (Block address) mod
(No. of blocks in cache)
Block 12 can go only into block 4
(12 mod 8)
Set associative:
Set no. = (Block address) mod
(No. of sets in cache)
Block 12 can go anywhere in set 0
(12 mod 4)

13. Q1: Where can a block be placed in the upper level?
Direct
Mapped
Full
Associative
Set
Associative 12 31
Block 12 placed in 8 block cache

14. Q2: How Is a Block Found If It Is in the Upper Level?
The address can be divided into two main parts
Block offset: selects the data from the block
offset size = log2(block size)
Block address: tag + index
index: selects set in cache
index size = log2(#blocks/associativity)
tag: compared to tag in cache to determine hit
tag size = addreess size - index size - offset size
Tag
Index

15. Q2: How Is a Block Found If It Is in the Upper Level?
Tag on each block
No need to check index or block offset
Increasing associativity shrinks index, expands tag
Block Address
Block Address
Block offset
Tag
Index
FA: No index
DM: No tag

16. Address organization for Set Associative Cache
n-bits of
CPU address
k-bits of Block
Offset
i-bits of
Tag
j-bits of
Index
n= i + j + k

17. Cache organization N-way set associative means there are N-blocks
TAG
BLOCK 1
N-way set
associative means
there are N-blocks
per Set
SET 0
TAG
BLOCK N
Index used to
select Set j
SET 2 -1

18. Cache organization There must also be a mechanism to in-
TAG
BLOCK 1
SET 0
There must also be
a mechanism to in-
dicate invalid block
data. This is commonly
done by attaching a
valid bit to the Tag
field (not shown).
TAG
BLOCK N
TheTag is associatively
compared to all tags in
a set. If there is a macth
we have a cache hit. j
SET 2 -1

19. Cache organization Each block is broken into
TAG
BLOCK 1
SET 0
TAG
BLOCK N
Each block is broken into
2 elements. The Block Offset
is used to select the which
element. k j
SET 2 -1
The Tag and Index identify the block & the
Block Offset identifies the element.

20. Q3: Which Block Should be Replaced on a Miss?
Easy for Direct Mapped
Set Associative or Fully Associative:
Random - easier to implement
Least Recently used - harder to implement - may approximate
Miss rates for caches with different size, associativity and replacemnt algorithm.
Associativity: 2-way 4-way 8-way
Size LRU Random LRU Random LRU Random
16 KB 5.18% 5.69% 4.67% 5.29% 4.39% 4.96%
64 KB 1.88% 2.01% 1.54% 1.66% 1.39% 1.53%
256 KB 1.15% 1.17% 1.13% 1.13% 1.12% 1.12%
For caches with low miss rates, random is almost as good as LRU.

21. Q4: What Happens on a Write?
Write through: The information is written to both the block in the cache and to the block in the lower-level memory.
Write back: The information is written only to the block in the cache. The modified cache block is written to main memory only when it is replaced.
is block clean or dirty? (add a dirty bit to each block)
Pros and Cons of each:
Write through
read misses cannot result in writes to memory,
easier to implement
Always combine with write buffers to avoid memory latency
Write back
Less memory traffic
Perform writes at the speed of the cache

22. Q4: What Happens on a Write?
Since data does not have to be brought into the cache on a write miss, there are two options:
Write allocate
The block is brought into the cache on a write miss
Used with write-back caches
Hope subsequent writes to the block hit in cache
No-write allocate
The block is modified in memory, but not brought into the cach
Used with write-through caches
Writes have to go to memory anyway, so why bring the block into the cache

23. Example: Alpha 21064 Data Cache
A cache read has 4 steps
(1) The address from the cache
is divided into the tag, index,
and block offset
(2) The index selects block
(3) The address tag is compared
with the tag in the cache, the
valid bit is checked, and data
to be loaded is selected
(4) If the valid bit is set, the data
is loaded into the processor
If there is a write, the data is also
sent to the write buffer

24. Writes in Alpha 21064
No write merging vs. write merging in write buffer V V V V
100 1
Buffer fills up on
4 stores to seq
addresses
104 1
108 1
112 1
100 1 1 1 1
Seq stores merged
to a single block

25. Split vs. Unified Cache
Unified cache (mixed cache): Data and instructions are stored together (von Neuman architecture)
Split cache: Data and instructions are stored separately (Harvard architecture)
Why do instructions caches have a lower miss ratio?
Size Instruction Cache Data Cache Unified Cache
1 KB 3.06% 24.61% 13.34%
2 KB 2.26% 20.57% 9.78%
4 KB 1.78% 15.94% 7.24%
8 KB 1.10% 10.19% 4.57%
16 KB 0.64% 6.47% 2.87%
32 KB 0.39% 4.82% 1.99%
64 KB 0.15% 3.77% 1.35%
128 KB 0.02% 2.88% 0.95%

26. Example: Split vs. Unified Cache
Which has the lower average memeory access time?
Split cache : 16 KB instructions + 16 KB data
Unified cache: 32 KB instructions + data
Assumptions
Use miss rates from previous chart
Miss penalty is 50 cycles
Hit time is 1 cycle
Load or store hit takes an extra cycle, since there is only one port for instructions and data

27. Example: Split vs. Unified Cache
Average memory-access time = Hit time + Miss rate x Miss penalty
AMAT = %instr x (instr hit time + instr miss rate x instr miss penalty) +
%data x (data hit time + data miss rate x data miss penalty)
For the split cache:
AMAT = 75% x ( %x 50) + 25% ( % x 50) = 2.05
For the unified cache
AMAT = 75% x ( %x 50) + 25% x ( % x 50) = 2.24
The unified cache has a longer AMAT, even though its miss rate is lower, due to conflicts for instruction and data hazards.
What are advantages of a split cache? Of unified cache?

28. CPU Performance
CPU time = (CPU execution clock cycles + Memory stall clock cycles) x clock cycle time
CPU execution clock cycle = Number of cycles for instructions assuming all hits (ignoring memory stalls)
Memory stall clock cycles = (Reads x Read miss rate x Read miss penalty + Writes x Write miss rate x Write miss penalty) Memory stall
Combining Reads & Writes (approx. of the above) clock cycles = Memory accesses x Miss rate x Miss penalty

29. CPU Performance
CPUtime = IC x (CPIexecution + Mem accesses per instruction x Miss rate x Miss penalty) x Clock cycle time
(Cache) Misses per instruction = Memory accesses per instruction x Miss rate
CPUtime = IC x (CPIexecution + Misses per instruction x Miss penalty) x Clock cycle time

30. Performance Example
Compare CPU time of a processor with and w/o a cache.
Hit Rate 98% 50 cycle miss penalty
2 cycles/instruction 1.33 memory ref/instr
With cache
CPU Time = IC x [ x (1-.98) x 50] x clock cycle time
no cache
CPU Time = IC x [ x 50 ] x clock cycle time
Speedup = 68.5/3.33 = 20.5

31. 2-Way Set Associative Cache
Features of an 8 KB 2-way set
associative cache
5 bit block offset
7 bit index
22 bit tag
The set associative cache has
extra hardware for
2 tag comparisons
mux to select data
Compared to the direct mapped
cache, the set associative cache
will tend to have a smaller miss rate
but a larger hit time.

32. Example: Two different cache organization
See example pp. 387

33. Improving Cache Performance
AMAT = Hit time + Miss rate x Miss penalty
Improve performance by:
1. Reduce the miss rate (or increase hit rate),
2. Reduce the miss penalty, or
3. Reduce the time to hit in the cache.

34. Summary
CPU-Memory gap is major performance obstacle for performance, HW and SW
Take advantage of program behavior: locality
Time of program still only reliable performance measure
4Qs of memory hierarchy
Where can a block be placed?
How is a block found?
Which block should be replaces on a miss?
What happens on a write?