1. 10/16: Lecture Topics Memory problem Memory Solution: Caches Locality
Types of caches
Fully associative
Direct mapped
n-way associative

2. Memory Problem
If all memory accesses (lw/sw) accessed main memory, programs would run 20 times slower
And it’s getting worse
processors speed up by 50% annually
memory accesses speed up by 9% annually
it’s becoming harder and harder to keep these processors fed

3. Solution: Memory Hierarchy
Keep all data in the big, slow, cheap storage
Keep copies of the “important” data in the small, fast, expensive storage
fast, small, expensive
storage
slow, large, cheap storage

4. Memory Hierarchy Memory Level Fabrication Tech Access Time (ns)
Size (bytes)
$/MB
Registers
<0.5
256
1000
L1 Cache
SRAM 2 8K
100
L2 Cache 10 1M
Memory
DRAM 50
128M
0.75
Disk
Magnetic Disk
10M
32G
0.0035

6. What is a Cache?
A cache allows for fast accesses to a subset of a larger data store
Your web browser’s cache gives you fast access to pages you visited recently
faster because it’s stored locally
subset because the web won’t fit on your disk
The memory cache gives the processor fast access to memory that it used recently
faster because it’s located on the CPU

7. Locality
Temporal locality: the principle that data being accessed now will probably be accessed again soon
Useful data tends to continue to be useful
Spatial locality: the principle that data near the data being accessed now will probably be needed soon
If data item n is useful now, then it’s likely that data item n+1 will be useful soon

8. Memory Access Patterns
Memory accesses don’t look like this
random accesses
Memory accesses do look like this
hot variables
step through arrays

9. Cache Terminology Hit—the data item is in the cache
Miss—the data item is not in the cache
Hit rate—the percentage of time the data item is in the cache (hit ratio)
Miss rate—the percentage of time the data item is not in the cache (hit ratio)
Hit time—the time required to access data in the cache
Miss time—the time required to access data not in the cache (miss penalty)
Access time—the time required to access a level of the memory hierarchy

10. teffective = htcache + (1-h)tmemory
Effective Access Time
teffective = htcache + (1-h)tmemory
cache access time
memory access time
effective access time
cache hit rate
cache miss rate
aka, Average Memory Access Time (AMAT)

11. Access Time Example Suppose tmemory for main memory is 50ns
Suppose tcache for the processor cache is 2ns
We want an effective access time of 3ns
What hit rate h do we need?

12. Cache Contents When do we put something in the cache?
when it is used for the first time
When do we take something out of the cache?
when it hasn’t been used for a long time
subset because all of memory won’t fit on the CPU

13. Concepts in Caching Assume a two level hierarchy:
Level 1: a cache that can hold 8 words
Level 2: a memory that can hold 32 words
cache
memory

14. Fully Associative Cache
In a fully associative cache,
any memory word can be placed in any cache line
each cache line stores which address it contains
accesses are slow (but not as slow as you would think)
Address
Valid
Value Y 0x N
0x09D91D11 0x
0x00012D10 0x
0xDEADBEEF
0x000123A8 0x

15. Direct Mapped Caches Fully associative caches are too slow
With direct mapped caches the address of the item determines where in the cache to store it
In this case, the lower five bits of the address dictate the cache entry

16. Direct Mapped Cache
Index
00000
00100
01000
01100
10000
10100
11000
11100
Address
Valid
Value Y 0x N
0x09D91D11 0x
0x00012D10 0x
0xDEADBEEF
0x000123A8 0x
The lower 5 bits of the address determine where it is located in the table

17. Address Tags
A tag is a label for a cache entry indicating where it came from
The upper bits of the data item’s address
7 bit Address
Tag (2)
Index (3)
Byte Offset (2) 10
111 01

18. Cache with Address Tag Index 000 001 010 011 100 101 110 111 Tag Valid
Value 11 Y 0x 10 N
0x09D91D11 01 0x 00
0x00012D10 0x
0xDEADBEEF
0x000123A8 0x

19. Reference Stream Example
index vb
tag
data
000
001
010
011
100
101
110
111
11010, 10111, 00001, 11010, 11011, 11111, 01101, 11010

20. Sample L1 Cache Suppose a L1 cache has the following characteristics
one word stored per entry
1024 entries
direct mapped
How many total bytes are stored in the cache?
How many bits are used for the index and tag fields of the address?
How many total bits are stored in the cache?

21. N-way Set Associative Caches
Direct mapped caches cannot store two addresses with the same index
If two addresses collide, then you have to kick one of them out
2-way associative caches can store two different addresses with the same index
Reduces misses due to conflicts
Also, 3-way, 4-way and 8-way set associative
Larger sets imply slower accesses

22. 2-way Associative Cache
Index
000
001
010
011
100
101
110
111
Tag
Valid
Value 11 Y 0x 10 N
0x09D91D11 01 0x 00
0x00012D10 0x
0xDEADBEEF
0x000123A8 0x
Tag
Valid
Value 00 Y 0x 10 N
0x B 11
0x000000CF
0x000000A2 0x 0x 01
0x0000C002 0x

23. Associativity Spectrum
Direct Mapped
Fast to access
Conflict Misses
N-way Associative
Slower to access
Fewer Conflict Misses
Fully Associative
Slow to access
No Conflict Misses