Cache Memory By Ed Martinez

 The fastest and most expensive memory on a computer system that is used to store collections of data.  Uses very short time to access recently and frequently used data. This helps improve computing performance by decreasing data access time.  With the speeds of modern CPUs ever increasing, accessing main memory has become a major performance bottleneck. Effective use of cache can help to minimize this problem. What is Cache Memory?

Cache Levels  Cache are referred to in levels, such as L1 or L2.  The level describes the connection or physical proximity of the cache to the CPU.  Traditionally, L1 cache, usually the smaller of the two, is located on the processor and runs at the same speed as the processor. L2 cache is outside, but near the processor and runs at the speed of the motherboard or FSB.  Recent designs have begun to integrate L2 cache onto the processor card or into the CPU chip itself, like L1 cache. This speeds access to the larger L2 cache, which improves the computer’s performance.

CPU Roadmap This photo shows a road map of the inside of the CPU, notice the different areas, and their functions. Can you find the L1 cache? This photo shows a road map of the inside of the CPU, notice the different areas, and their functions. Can you find the L1 cache?

Cache Memory This photo shows level 2 cache memory on the Processor board, beside the CPU This photo shows level 2 cache memory on the Processor board, beside the CPU

Cache Mapping  Associative  Direct  Set Associative

Associative Mapping  Sometimes known as fully associative mapping, any block of memory can be mapped to any line in the cache.  The block number is appended to the line as a tag.  There is a bit to indicate whether the line is valid or not.  Difficult and expensive to implement.

Direct Mapping  Each block in main memory can be loaded to only one line in cache.  Each cache line contains a tag to identify the block of main memory in the cache.  This is easy to implement, but inflexible.

Set-Associative Mapping  Blocks can be mapped to a subset of the lines in cache.  A block can be mapped to either 2 or 4 lines in cache, usually referred to as “Two-way” or “Four-way”.  LRU is the most popular replacement policy used with this mapping process.  More flexible than direct mapping, but easier to implement than associative mapping.

Why use a cache replacement policy?  Cache is small compared to available system memory  Cache maintains copies of recently referenced data items to speed access time  Once full it must use a replacement policy to decide how to handle additional data items  The replacement policy can either ignore the new data or decide which of the old data items to remove to make room

Cache Replacement Policy (cont)  Two main issues in determining which policy to use:  Increase the hit ratio by keeping the items that are referenced most frequently  Policy should be inexpensive to implement

Cache Replacement Policies  FIFO  LRU  Random

FIFO Replacement  FIFO (First in, first out) – the oldest data item in the cache frame is replaced.  Easy to implement and has fast operation.  Since it maintains no reference statistics, depending on the sequence of the data you may be constantly removing and replacing the same block of memory.

LRU Replacement  Least Recently Used – The item that was accessed least recently is replaced with the new data item.  Easy to implement, follows the idea that items which have been recently used are likely to be used again.  Keeps a list of data items that are currently in the cache. Referenced data items are moved to the front of the list  Data items at the back of the list are removed as space is needed  Primary drawback is that it requires time to analyze the reference statistics to determine which item to remove. Also requires additional cache space to maintain the statistics

Random Replacement  Random – a randomly generated address determines which data item in the cache frame to replace.  Has no need to store statistics, so it doesn’t waste time analyzing anything.  Performance is close to that of the other policies, and the implementation is fairly simple.

Cache Replacement Policies  Lets assume a computer system has an associative, a direct- mapped, or a two-way set-associative cache of 8 bytes.  The CPU accesses the following locations in order (t ):  The CPU accesses the following locations in order (the subscript is the low-order 3 bits of its address in physical memory ): A 0 B 0 C 2 A 0 D 1 B 0 E 4 F 5 A 0 C 2 D 1 B 0 G 3 C 2 H 7 I 6 A 0 B 0

Associative Cache (FIFO Replacement Policy) DataABCADBEFACDBGCHIAB CACHECACHE AAAAAAAAAAAAAAAIII BBBBBBBBBBBBBBBAA CCCCCCCCCCCCCCCB DDDDDDDDDDDDDD EEEEEEEEEEEE FFFFFFFFFFF GGGGGG HHHH Hit? * * **** * Hit ratio = 7/18 A 0 B 0 C 2 A 0 D 1 B 0 E 4 F 5 A 0 C 2 D 1 B 0 G 3 C 2 H 7 I 6 A 0 B 0

Direct mapped Cache Hit ratio = 3/18 DataABCADBEFACDBGCHIAB CACHECACHE 0ABBAABBBAAABBBBBAB 1 DDDDDDDDDDDDDD 2 CCCCCCCCCCCCCCCC 3 GGGGGG 4 EEEEEEEEEEEE 5 FFFFFFFFFFF 6 III 7 HHHH Hit? ** * A 0 B 0 C 2 A 0 D 1 B 0 E 4 F 5 A 0 C 2 D 1 B 0 G 3 C 2 H 7 I 6 A 0 B 0

Two-way set associative cache (LRU Replacement Policy) Hit ratio = 7/18 A 0 B 0 C 2 A 0 D 1 B 0 E 4 F 5 A 0 C 2 D 1 B 0 G 3 C 2 H 7 I 6 A 0 B 0 DataABCADBEFACDBGCHIAB CACHECACHE 0A-0A-1 A-0 A-1E-0 E-1 B-0 B-1B-0 0 B-1 B-0B-1 A-0 A-1 A-0A-1 1 D-0 D-1 D-0 1 F-0 F-1 2 C-0 C-1 2 I-0 3 G-0 G-1 3 H-0 Hit? * * ** * **

Replacement with 2 byte line size Now lets consider data lines of two bytes. The data pairs that make up the lines are:  A and J; B and D; C and G; E and F; and I and H.

Associative Cache with 2 byte line size (FIFO Replacement Policy) Hit ratio = 11/18 A 0 B 0 C 2 A 0 D 1 B 0 E 4 F 5 A 0 C 2 D 1 B 0 G 3 C 2 H 7 I 6 A 0 B 0 A and J; B and D; C and G; E and F; and I and H DataABCADBEFACDBGCHIAB CACHECACHE AAAAAAAAAAAAAAIIII JJJJJJJJJJJJJJHHHH BBBBBBBBBBBBBBBAA DDDDDDDDDDDDDDDJJ CCCCCCCCCCCCCCCB GGGGGGGGGGGGGGGD EEEEEEEEEEEE FFFFFFFFFFFF Hit? *** ******* *

Direct-mapped Cache with line size of 2 bytes Hit ratio 7/18 DataABCADBEFACDBGCHIAB CACHECACHE 0ABBABBBBAABBBBBBAB 1JDDJDDDDJJDDDDDDJD 2 CCCCCCCCCCCCCCCC 3 GGGGGGGGGGGGGGGG 4 EEEEEEEEEEEE 5 FFFFFFFFFFFF 6 IIII 7 HHHH Hit? * * * *** * A 0 B 0 C 2 A 0 D 1 B 0 E 4 F 5 A 0 C 2 D 1 B 0 G 3 C 2 H 7 I 6 A 0 B 0 A and J; B and D; C and G; E and F; and I and H

Two-way set Associative Cache with line size of 2 bytes Hit ratio = 12/18 Data ABCADBEFACDBGCHIAB CACHECACHE 0A-0A-1 A-0A-1 E-0 E-1B-0 B-1B-0 1J-0J-1 J-0J-1 F-0 F-1D-0 D-1D-0 0 B-0 B-1B-0 B-1 A-0 A-1 A-0A-1 1 D-0 D-1D-0 D-1 J-0 J-1 J-0J-1 2 C-0 C-1 3 G-0 G-1 2 I-0 3 H-0 Hit? *** * * **** *** A 0 B 0 C 2 A 0 D 1 B 0 E 4 F 5 A 0 C 2 D 1 B 0 G 3 C 2 H 7 I 6 A 0 B 0 A and J; B and D; C and G; E and F; and I and H

Cache Performance  Primary reason for including cache memory in a computer is to improve system performance by reducing the time needed to access data in memory.  The cache hit ratio – hits/(hits + misses).  A well-designed cache will have a hit ratio close to 1.  Cache hits outnumber the misses by far. When more data is retrieved from the faster cache memory rather than from main memory system performance is improved.

Cache Performance (cont) hTMTM ns ns ns ns ns ns ns ns ns ns ns  Hit ratios and average memory access times  As the number of hits increase the average memory access time decreases h = hit ratio T M = weighted average cache access time.

References  Carpielli, John D, Computer Systems Organization & Architecture  Comer, Douglas E, Essentials of Computer Architecture  