1. Improving Memory Access 2/3 The Cache and Virtual Memory
EECS 322 Computer Architecture
Improving Memory Access 2/3 The Cache and Virtual Memory

2. The Art of Memory System Design
Optimize the memory system organization
to minimize the average memory access time
for typical workloads
Workload or
Benchmark
programs
Processor
reference stream
<op,addr>, <op,addr>,<op,addr>,<op,addr>, . . .
op: i-fetch, read, write
Memory $
MEM

3. Principle of Locality
• Principle of Locality states that programs access a relatively small portion of their address space at any instance of time
• Two types of locality
• Temporal locality (locality in time) If an item is referenced, then the same item will tend to be referenced soon “the tendency to reuse recently accessed data items”
• Spatial locality (locality in space) If an item is referenced, then nearby items will be referenced soon “the tendency to reference nearby data items”

4. Memory Hierarchy of a Modern Computer System
By taking advantage of the principle of locality:
Present the user with as much memory as is available in the cheapest technology.
Provide access at the speed offered by the fastest technology.
Processor
Control
Tertiary
Storage
(Disk)
Secondary
Storage
(Disk)
Second
Level
Cache
(SRAM)
Main
Memory
(DRAM)
The design goal is to present the user with as much memory as is available in the cheapest technology (points to the disk).
While by taking advantage of the principle of locality, we like to provide the user an average access speed that is very close to the speed that is offered by the fastest technology.
(We will go over this slide in details in the next lecture on caches).
+1 = 16 min. (X:56)
Datapath
On-Chip
Cache
Registers
Speed (ns): 1s
10s
100s
10,000,000s
(10s ms)
10,000,000,000s
(10s sec)
Size (bytes):
100s Ks Ms Gs Ts

5. Memory Hierarchy of a Modern Computer System
By taking advantage of the principle of locality:
Present the user with as much memory as is available in the cheapest technology.
Provide access at the speed offered by the fastest technology.
DRAM is slow but cheap and dense:
Good choice for presenting the user with a BIG memory system
SRAM is fast but expensive and not very dense:
Good choice for providing the user FAST access time.
Let’s summarize today’s lecture. The first thing we covered is the principle of locality.
There are two types of locality: temporal, or locality of time and spatial, locality of space.
We talked about memory system design.
The key idea of memory system design is to present the user with as much memory as possible in the cheapest technology while by taking advantage of the principle of locality, create an illusion that the average access time is close to that of the fastest technology.
As far as Random Access technology is concerned, we concentrate on 2: DRAM and SRAM.
DRAM is slow but cheap and dense so is a good choice for presenting the use with a BIG memory system.
SRAM, on the other hand, is fast but it is also expensive both in terms of cost and power, so it is a good choice for providing the user with a fast access time.
I have already showed you how DRAMs are used to construct the main memory for the SPARCstation 20.
On Friday, we will talk about caches.
+2 = 78 min. (Y:58)

6. Spatial Locality
• Temporal only cache cache block contains only one word (No spatial locality).
• Spatial locality Cache block contains multiple words.
• When a miss occurs, then fetch multiple words.
• Advantage Hit ratio increases because there is a high probability that the adjacent words will be needed shortly.
• Disadvantage Miss penalty increases with block size

7. Direct Mapped Cache: Mips Architecture
Figure 7.7
Tag
Index
Data
Compare Tags
Hit

8. Cache schemes write-through cache Always write the data into both the
cache and memory and then wait for memory.
Chip Area
Speed
write buffer
write data into cache and write buffer.
If write buffer full processor must stall.
No amount of buffering can help if writes are being generated faster
than the memory system can accept them.
write-back cache
Write data into the cache block and only write to memory when block is modified but complex to implement in hardware.

9. Spatial Locality: 64 KB cache, 4 words
Figure 7.10
• 64KB cache using four-word (16-byte word)
• 16 bit tag, 12 bit index, 2 bit block offset, 2 bit byte offset.

10. Designing the Memory System
Figure 7.13
Make reading multiple words easier by using banks of memory
It can get a lot more complicated...

11. Memory organizations One word wide memory organization Advantage
Figure 7.13
One word wide memory organization
Advantage
Easy to implement, low hardware overhead
Disadvantage
Slow: 0.25 bytes/clock transfer rate
Chip Area
Speed
Interleave memory organization
Advantage
Better: 0.80 bytes/clock transfer rate
Banks are valuable on writes: independently
Disadvantage
more complex bus hardware
Wide memory organization
Advantage
Fastest: 0.94 bytes/clock transfer rate
Disadvantage
Wider bus and increase in cache access time

12. Increased Miss Penalty
Block Size Tradeoff
In general, larger block size take advantage of spatial locality BUT:
Larger block size means larger miss penalty:
Takes longer time to fill up the block
If block size is too big relative to cache size, miss rate will go up
Too few cache blocks
In gerneral, Average Access Time:
= Hit Time x (1 - Miss Rate) + Miss Penalty x Miss Rate
Average
Access
Time
Miss
Penalty
As I said earlier, block size is a tradeoff. In general, larger block size will reduce the miss rate because it take advantage of spatial locality.
But remember, miss rate NOT the only cache performance metrics. You also have to worry about miss penalty.
As you increase the block size, your miss penalty will go up because as the block gets larger, it will take you longer to fill up the block.
Even if you look at miss rate by itself, which you should NOT, bigger block size does not always win. As you increase the block size, assuming keeping cache size constant, your miss rate will drop off rapidly at the beginning due to spatial locality.
However, once you pass certain point, your miss rate actually goes up.
As a result of these two curves, the Average Access Time (point to equation), which is really the more important performance metric than the miss rate, will go down initially because the miss rate is dropping much faster than the increase in miss penalty.
But eventually, as you keep on increasing the block size, the average access time can go up rapidly because not only is the miss penalty is increasing, the miss rate is increasing as well.
Let me show you why your miss rate may go up as you increase the block size by another extreme example.
+3 = 33 min. (Y:13)
Miss
Rate
Exploits Spatial Locality
Increased Miss Penalty
& Miss Rate
Fewer blocks:
compromises
temporal locality
Block Size
Block Size
Block Size

13. Fully associative cache 2-way set associative cache
Cache associativity
Figure 7.15
Fully associative cache
Direct-mapped cache
2-way set associative cache

14. Cache associativity
Figure 7.16
Compared to direct mapped, give a series of references that:
results in a lower miss ratio using a 2-way set associative cache
results in a higher miss ratio using a 2-way set associative cache
assuming we use the “least recently used” replacement strategy

15. A Two-way Set Associative Cache
N-way set associative: N entries for each Cache Index
N direct mapped caches operates in parallel
Example: Two-way set associative cache
Cache Index selects a “set” from the cache
The two tags in the set are compared in parallel
Data is selected based on the tag result
Cache Index
Valid
Cache Tag
Cache Data
Cache Data
Cache Block 0
Cache Tag
Valid :
Cache Block 0 : : :
This is called a 2-way set associative cache because there are two cache entries for each cache index. Essentially, you have two direct mapped cache works in parallel.
This is how it works: the cache index selects a set from the cache. The two tags in the set are compared in parallel with the upper bits of the memory address.
If neither tag matches the incoming address tag, we have a cache miss.
Otherwise, we have a cache hit and we will select the data on the side where the tag matches occur.
This is simple enough. What is its disadvantages?
+1 = 36 min. (Y:16)
Compare
Adr Tag
Compare 1
Sel1
Mux
Sel0 OR
Cache Block
Hit

16. A 4-way set associative implementation
Figure 7.19

17. Disadvantage of Set Associative Cache
N-way Set Associative Cache versus Direct Mapped Cache:
N comparators vs. 1
Extra MUX delay for the data
Data comes AFTER Hit/Miss decision and set selection
Cache Data
Cache Block 0
Cache Tag
Valid :
Cache Index
Mux 1
Sel1
Sel0
Cache Block
Compare
Adr Tag OR
Hit
First of all, a N-way set associative cache will need N comparators instead of just one comparator (use the right side of the diagram for direct mapped cache).
A N-way set associative cache will also be slower than a direct mapped cache because of this extra multiplexer delay.
Finally, for a N-way set associative cache, the data will be available AFTER the hit/miss signal becomes valid because the hit/mis is needed to control the data MUX.
For a direct mapped cache, that is everything before the MUX on the right or left side, the cache block will be available BEFORE the hit/miss signal (AND gate output) because the data does not have to go through the comparator.
This can be an important consideration because the processor can now go ahead and use the data without knowing if it is a Hit or Miss. Just assume it is a hit.
Since cache hit rate is in the upper 90% range, you will be ahead of the game 90% of the time and for those 10% of the time that you are wrong, just make sure you can recover.
You cannot play this speculation game with a N-way set-associatvie cache because as I said earlier, the data will not be available to you until the hit/miss signal is valid.
+2 = 38 min. (Y:18)

18. Fully Associative Fully Associative Cache Forget about the Cache Index
Compare the Cache Tags of all cache entries in parallel
Example: Block Size = 2 B blocks, we need N 27-bit comparators
By definition: Conflict Miss = 0 for a fully associative cache 31 4
Cache Tag (27 bits long)
Byte Select
Ex: 0x01
While the direct mapped cache is on the simple end of the cache design spectrum, the fully associative cache is on the most complex end.
It is the N-way set associative cache carried to the extreme where N in this case is set to the number of cache entries in the cache.
In other words, we don’t even bother to use any address bits as the cache index.
We just store all the upper bits of the address (except Byte select) that is associated with the cache block as the cache tag and have one comparator for every entry.
The address is sent to all entries at once and compared in parallel and only the one that matches are sent to the output. This is called an associative lookup.
Needless to say, it is very hardware intensive. Usually, fully associative cache is limited to 64 or less entries.
Since we are not doing any mapping with the cache index, we will never push any other item out of the cache because multiple memory locations map to the same cache location.
Therefore, by definition, conflict miss is zero for a fully associative cache. This, however, does not mean the overall miss rate will be zero.
Assume we have 64 entries here. The first 64 items we accessed can fit in.
But when we try to bring in the 65th item, we will need to throw one of them out to make room for the new item. This bring us to the third type of cache misses: Capacity Miss.
+3 = 41 min. (Y:21)
Cache Tag
Valid Bit
Cache Data : X
Byte 31
Byte 1
Byte 0 : X
Byte 63
Byte 33
Byte 32 X X : : : X

19. Performance
Figure 7.29

20. Decreasing miss penalty with multilevel caches
Add a second level cache:
often primary cache is on the same chip as the processor
use SRAMs to add another cache above primary memory (DRAM)
miss penalty goes down if data is in 2nd level cache
Example:
CPI of 1.0 on a 500Mhz machine with a 5% miss rate, 200ns DRAM access
Adding 2nd level cache with 20ns access time decreases miss rate to 2%
Using multilevel caches:
try and optimize the hit time on the 1st level cache
try and optimize the miss rate on the 2nd level cache

21. Decreasing miss penalty with multilevel caches
Add a second level cache:
often primary cache is on the same chip as the processor
use SRAMs to add another cache above primary memory (DRAM)
miss penalty goes down if data is in 2nd level cache

22. Decreasing miss penalty with multilevel caches
Example:
CPI of 1.0 on a 500Mhz machine with a 5% miss rate, 200ns DRAM access
Adding 2nd level cache with 20ns access time decreases miss rate to 2%
Using multilevel caches:
try and optimize the hit time on the 1st level cache
try and optimize the miss rate on the 2nd level cache

23. A Summary on Sources of Cache Misses
Compulsory (cold start or process migration, first reference): first access to a block
“Cold” fact of life: not a whole lot you can do about it
Note: If you are going to run “billions” of instruction, Compulsory Misses are insignificant
Conflict (collision):
Multiple memory locations mapped to the same cache location
Solution 1: increase cache size
Solution 2: increase associativity
Capacity:
Cache cannot contain all blocks access by the program
Solution: increase cache size
Invalidation: other process (e.g., I/O) updates memory
(Capacity miss) That is the cache misses are due to the fact that the cache is simply not large enough to contain all the blocks that are accessed by the program.
The solution to reduce the Capacity miss rate is simple: increase the cache size.
Here is a summary of other types of cache miss we talked about.
First is the Compulsory misses. These are the misses that we cannot avoid. They are caused when we first start the program.
Then we talked about the conflict misses. They are the misses that caused by multiple memory locations being mapped to the same cache location.
There are two solutions to reduce conflict misses. The first one is, once again, increase the cache size. The second one is to increase the associativity.
For example, say using a 2-way set associative cache instead of directed mapped cache.
But keep in mind that cache miss rate is only one part of the equation. You also have to worry about cache access time and miss penalty. Do NOT optimize miss rate alone.
Finally, there is another source of cache miss we will not cover today. Those are referred to as invalidation misses caused by another process, such as IO , update the main memory so you have to flush the cache to avoid inconsistency between memory and cache.
+2 = 43 min. (Y:23)

24. Virtual Memory
Main memory can act as a cache for the secondary storage (disk) Advantages:
illusion of having more physical memory
program relocation
protection

25. Pages: virtual memory blocks
Page faults: the data is not in memory, retrieve it from disk
huge miss penalty, thus pages should be fairly large (e.g., 4KB)
reducing page faults is important (LRU is worth the price)
can handle the faults in software instead of hardware
using write-through is too expensive so we use writeback

26. Pages: virtual memory blocks

27. Page Tables

28. Page Tables

29. Basic Issues in Virtual Memory System Design
size of information blocks that are transferred from
secondary to main storage (M)
block of information brought into M, and M is full, then some region
of M must be released to make room for the new block -->
replacement policy
which region of M is to hold the new block --> placement policy
missing item fetched from secondary memory only on the occurrence
of a fault --> demand load policy
disk
mem
cache
reg
pages
frame
Paging Organization
virtual and physical address space partitioned into blocks of equal size
page frames
pages

30. TLBs: Translation Look-Aside Buffers
A way to speed up translation is to use a special cache of recently used page table entries -- this has many names, but the most frequently used is Translation Lookaside Buffer or TLB
Virtual Address Physical Address Dirty Ref Valid Access
TLB access time comparable to cache access time
(much less than main memory access time)

31. Making Address Translation Fast
A cache for address translations: translation lookaside buffer

32. Translation Look-Aside Buffers
Just like any other cache, the TLB can be organized as fully associative, set associative, or direct mapped
TLBs are usually small, typically not more than entries even on high end machines. This permits fully associative lookup on these machines. Most mid-range machines use small n-way set associative organizations.
hit VA PA
miss
TLB
Lookup
Cache
Main
Memory
CPU
Translation
with a TLB
miss
hit
Trans-
lation
data
1/2 t t
20 t

33. TLBs and caches

34. Modern Systems
Figure 7.32
Very complicated memory systems:

35. Summary: The Cache Design Space
Several interacting dimensions
cache size
block size
associativity
replacement policy
write-through vs write-back
write allocation
The optimal choice is a compromise
depends on access characteristics
workload
use (I-cache, D-cache, TLB)
depends on technology / cost
Simplicity often wins
Cache Size
Associativity
Block Size
No fancy replacement policy is needed for the direct mapped cache.
As a matter of fact, that is what cause direct mapped trouble to begin with: only one place to go in the cache--causes conflict misses.
Besides working at Sun, I also teach people how to fly whenever I have time.
Statistic have shown that if a pilot crashed after an engine failure, he or she is more likely to get killed in a multi-engine light airplane than a single engine airplane.
The joke among us flight instructors is that: sure, when the engine quit in a single engine stops, you have one option: sooner or later, you land. Probably sooner.
But in a multi-engine airplane with one engine stops, you have a lot of options. It is the need to make a decision that kills those people.
Bad
Good
Factor A
Factor B
Less
More

36. Summary: TLB, Virtual Memory
Caches, TLBs, Virtual Memory all understood by examining how they deal with 4 questions: 1) Where can block be placed? 2) How is block found? 3) What block is repalced on miss? 4) How are writes handled?
Page tables map virtual address to physical address
TLBs are important for fast translation
TLB misses are significant in processor performance: (funny times, as most systems can’t access all of 2nd level cache without TLB misses!)
Let’s do a short review of what you learned last time.
Virtual memory was originally invented as another level of memory hierarchy such that programers, faced with main memory much smaller than their programs, do not have to manage the loading and unloading portions of their program in and out of memory.
It was a controversial proposal at that time because very few programers believed software can manage the limited amount of memory resource as well as human.
This all changed as DRAM size grows exponentially in the last few decades.
Nowadays, the main function of virtual memory is to allow multiple processes to share the same main memory so we don’t have to swap all the non-active processes to disk.
Consequently, the most important function of virtual memory these days is to provide memory protection.
The most common technique, but we like to emphasis not the only technique, to translate virtual memory address to physical memory address is to use a page table.
TLB, or translation lookaside buffer, is one of the most popular hardware techniques to reduce address translation time.
Since TLB is so effective in reducing the address translation time, what this means is that TLB misses will have a significant negative impact on processor performance.
+3 = 3 min. (X:43)

37. Summary: Memory Hierachy
VIrtual memory was controversial at the time: can SW automatically manage 64KB across many programs?
1000X DRAM growth removed the controversy
Today VM allows many processes to share single memory without having to swap all processes to disk; VM protection is more important than memory hierarchy
Today CPU time is a function of (ops, cache misses) vs. just f(ops): What does this mean to Compilers, Data structures, Algorithms?
Let’s do a short review of what you learned last time.
Virtual memory was originally invented as another level of memory hierarchy such that programers, faced with main memory much smaller than their programs, do not have to manage the loading and unloading portions of their program in and out of memory.
It was a controversial proposal at that time because very few programers believed software can manage the limited amount of memory resource as well as human.
This all changed as DRAM size grows exponentially in the last few decades.
Nowadays, the main function of virtual memory is to allow multiple processes to share the same main memory so we don’t have to swap all the non-active processes to disk.
Consequently, the most important function of virtual memory these days is to provide memory protection.
The most common technique, but we like to emphasis not the only technique, to translate virtual memory address to physical memory address is to use a page table.
TLB, or translation lookaside buffer, is one of the most popular hardware techniques to reduce address translation time.
Since TLB is so effective in reducing the address translation time, what this means is that TLB misses will have a significant negative impact on processor performance.
+3 = 3 min. (X:43)