1. 10/18: Lecture Topics Using spatial locality
memory blocks
Write-back vs. write-through
Types of cache misses
Cache performance
Cache tradeoffs
Cache summary

2. Locality
Temporal locality: the principle that data being accessed now will probably be accessed again soon
Useful data tends to continue to be useful
Spatial locality: the principle that data near the data being accessed now will probably be needed soon
If data item n is useful now, then it’s likely that data item n+1 will be useful soon

3. Memory Access Patterns
Memory accesses don’t look like this
random accesses
Memory accesses do look like this
hot variables
step through arrays

4. Locality
Last time, we improved memory performance by taking advantage of temporal locality
When a word in memory was accessed we loaded it into memory
This does nothing for spatial locality

5. Possible Spatial Locality Solution
Store one word per cache line
When memory word N is accessed, load word N, word N+1 and N+2 … into the cache
This is called prefetching
What’s a drawback?
Example: What if we access the word at address ?
Index
000
001
010
011
100
101
110
111
Tag
Valid
Value

6. Memory Blocks Divide memory into blocks
If any word in a block is accessed, then load an entire block into the cache
Block 0 0x –0x F
Block 1 0x –0x F
Block 2 0x –0x000000BF
Cache line for 16 word block size
tag
valid w0 w1 w2 w3 w4 w5 w6 w7 w8 w9
w10
w11
w12
w13
w14
w15

7. Address Tags Revisited
A cache block size > 1 word requires the address to be divided differently
Instead of a byte offset into a word, we need a byte offset into the block
Assuming we had 10-bit addresses, and 4 words in a block…
10 bit Address
Tag (3)
Index (3)
Block Offset (4)
010
110
0111

8. Cache Diagram Cache with a block size of 4 words
Index
000
001
010
011
100
101
110
111
Tag
Valid
Values
What does the cache look like after accesses to these 10 bit addresses? ,

9. Cache miss; access memory
Cache Lookup
32 bit addr., 64K DM cache, 4 words/block
ref. address
Index into cache
Cache hit; select word
yes
Is valid bit on?
yes
Do tags match?
return data no no
Cache miss; access memory

10. Cache Example
Suppose the L1 cache is 32KB, 2-way set associative and has 8 words per block, how do we partition the 32-bit address?
How many bits for the block offset?
How many bits for the index?
How many bits for the tag?

11. The Effects of Block Size
Big blocks are good
Fewer first time misses
Exploits spatial locality
Small blocks are good
Don’t evict so much other data when bringing in a new entry
More likely that all items in the block will turn out to be useful
How do you choose a block size?

12. Reads vs. Writes Caching is essentially making a copy of the data
When you read, the copies still match when you’re done
When you write, the results must eventually propagate to both copies
Especially at the lowest level, which is in some sense the permanent copy

13. Write-Through Caches
Write the update to the cache and the memory immediately
Advantages:
The cache and the memory are always consistent
Evicting a cache line is cheap because no data needs to be written back
Easier to implement
Disadvantages?

14. Write-Back Caches
Write the update to the cache only. Write to the memory only when the cache block is evicted.
Advantages:
Writes go at cache speed rather than memory speed.
Some writes never need to be written to the memory.
When a whole block is written back, can use high bandwidth transfer.
Disadvantages?

15. Dirty bit When evicting a block from a write-back cache, we could
always write the block back to memory
write it back only if we changed it
Caches use a “dirty bit” to mark if a line was changed
the dirty bit is 0 when the block is loaded
it is set to 1 if the block is modified
when the line is evicted, it is written back only if the dirty bit is 1

16. Dirty Bit Example
Use the dirty bit to determine when evicted cache lines need to be written back to memory
Index 00 01 10 11
Tag
Valid
Dirty
Values
Assume 8 bit addresses
Assume all memory words are initialized to 7
$r2 = Mem[ ]
Mem[ ] = 10
$r3 = Mem[ ]
$r4 = Mem[ ]
Mem[ ] = 10

17. i-Cache and d-Cache
There usually are two separate caches for instructions and data. Why?
Avoids structural hazards in pipelining
The combined cache is twice as big but still has an access time of a small cache
Allows both caches to operate in parallel, for twice the bandwidth

18. Handling i-Cache Misses
Stall the pipeline and send the address of the missed instruction to the memory
Instruct memory to perform a read; wait for the access to complete
3. Update the cache
4. Restart the instruction, this time fetching it successfully from the cache
d-Cache misses are even easier, but still require a pipeline stall

19. Cache Replacement How do you decide which cache block to replace?
If the cache is direct-mapped, it’s easy
Otherwise, common strategies:
Random
Least Recently Used (LRU)
Other strategies are used at lower levels of the hierarchy. More on those later.

20. LRU Replacement
Replace the block that hasn’t been used for the longest time.
Reference stream:
A B C D B D E B A C B C E D C B

21. LRU Implementations
LRU is very difficult to implement for high degrees of associativity
4-way approximation:
1 bit to indicate least recently used pair
1 bit per pair to indicate least recently used item in this pair
Much more complex approximations at lower levels of the hierarchy

22. The Three C’s of Caches Three reasons for cache misses:
Compulsory miss: item has never been in the cache
Capacity miss: item has been in the cache, but space was tight and it was forced out (occurs even with fully associative caches)
Conflict miss: item was in the cache, but the cache was not associative enough, so it was forced out (never occurs with fully associative caches)

23. Eliminating Cache Misses
What cache parameters (cache size, block size, associativity) can you change to eliminate the following kinds of misses
compulsory
capacity
conflict

24. Multi-Level Caches
Use each level of the memory hierarchy as a cache over the next lowest level
Inserting level 2 between levels 1 and 3 allows:
level 1 to have a higher miss rate (so can be smaller and cheaper)
level 3 to have a larger access time (so can be slower and cheaper)
The new effective access time equation:

25. Which cache system is better?
32 KB unified data and instruction cache
hit rate of 97%
16 KB data cache
hit rate of 92%
And 16 KB instruction cache
hit rate of 98%
Assume
20% of instructions are loads or stores

26. Cache Parameters and Tradeoffs
If you are designing a cache, what choices do you have and what are their tradeoffs?

27. Cache Comparisons Alpha 21164 MIPS R10000 Pentium Pro UltraSparc 1 8KB
direct-mapped
32B block
32KB
2-way (LRU)
64B block
4-way
16KB
pseudo 2-way L1
i-Cache
Alpha 21164
MIPS R10000
Pentium Pro
UltraSparc 1
8KB
direct-mapped
32B block
32KB
2-way (LRU)
2-way
16KB L1
d-Cache
Alpha 21164
Pentium Pro
96KB
3-way
64B block
on chip
256KB
4-way
32B block
same package L2
unified
Cache

28. Summary: Classifying Caches
Where can a block be placed?
Direct mapped, Set/Fully associative
How is a block found?
Direct mapped: by index
Set associative: by index and search
Fully associative: by search
What happens on a write access?
Write-back or Write-through
Which block should be replaced?
Random
LRU (Least Recently Used)