2. Memory Hierarchy: Terminology Hit: data appears in some block in the upper level (example: Block X)  Hit Rate : the fraction of memory access found in the upper level  Hit Time: Time to access the upper level which consists of RAM access time + Time to determine hit/miss Miss: data needs to be retrieve from a block in the lower level (Block Y)  Miss Rate = 1 - (Hit Rate)  Miss Penalty: Time to replace a block in the upper level + Time to deliver the block the processor Hit Time << Miss Penalty Lower Level Memory Upper Level Memory To Processor From Processor Blk X Blk Y

3. General Principles of Memory Locality  Temporal Locality: referenced memory is likely to be referenced again soon (e.g. code within a loop)  Spatial Locality: memory close to referenced memory is likely to be referenced soon (e.g., data in a sequentially access array) Definitions  Upper: memory closer to processor  Block: minimum unit that is present or not present  Block address: location of block in memory  Hit: Data is found in the desired location  Hit time: time to access upper level  Miss rate: percentage of time item not found in upper level Locality + smaller HW is faster = memory hierarchy  Levels: each smaller, faster, more expensive/byte than level below  Inclusive: data found in upper level also found in the lower level

4. What is Cache? “A computer memory with very short access time used for storage of frequently used instructions or data” – webster.com

5. Types of Cache L1/L2/L3 Cache RAM Cache Disk Cache Software Level Cache

6. Memory Hierarchy Comparison

7. L1/L2/L3 Cache (Cache Memory) Cache closer to the CPU that stores recently accessed data from RAM Holds instructions to be executed next and variables for the CPU

8. L2 Cache COAST: Cache on a Stick

9. Cache Memory Continued Level 1 Cache is stored on the CPU L2/L3 Cache stored near, but not on the CPU L1/L2/L3 Cache is more expensive than RAM

10. Different CPU Caches:

11. Cache Memory Replacement Policies CPU requests data from cached memory Cache memory has data, sends to CPU (Hit) RAM has data, sends to CPU (Miss) Cache performance measured by hits/misses where the ideal algorithm has a ratio close to 1.

12. Replacement Policies On a miss the data from RAM is loaded in to Cache Due to the nature of most programs (Variables are accessed multiple times in the same program) this cached data will have a high chance of being accessed often. One possible replacement algorithm is a FIFO based one where the oldest cached data is replaced first.

13. CPU Cache - Write Policies CPU writes data in cache (hit) Write-through – Cache and RAM updated Write-back – Cache updated, RAM updated when new data is going to replace that data in cache Dirty bit keeps track of if a part of cache has changed

14. CPU Cache - Write Policies CPU writes but data not in cache (Miss) Write-no-allocate – RAM updated Write-allocate – RAM updated, data loaded into cache

15. CPU Cache – SMP Problems Multiprocessor systems have trouble with write- through and write-back IE. One processor updates data cached by both processors Possible Solutions: Cache only unshared memory Update all cache simultaneously

16. RAM Cache Same as Cache Memory or CPU Cache, except instead of the L1/L2 cache caching RAM, RAM caches data from the hard disk. For example, Windows loads multiple required libraries for running in memory, but code isn’t executed from them constantly.

17. Disk Cache Like cache memory, disk cache holds data recently read or written to from a hard disk. This is to speed up subsequent reads from recently accessed data. Most newer hard drives have around 8mb of cache memory.

19. Software Cache Software also needs to cache data into RAM that it uses often – however, more often than not, this data isn’t directly from the hard disk. The data software caches is often processed data from a file translated by the program into another data structure. The process of translating may take a long time, so the cache in RAM is used to speed up the rest of the programs run time.

20. Software Cache - Example Games such as Half-Life and Unreal Tournament 2004 have a loading time before each game where the map, models, sounds, and other important data are all loaded into RAM. Example: Website loading lots of data from an SQL server each page load may take a few seconds, but if the first load caches all the data, the load may only take a few milliseconds.

21. Software Cache - Example Internet Explorer History – Keeps a cache of recently viewed websites for easy access Cache – Keeps images and other files on disk so websites load faster the second time you visit them. These files are updated when there is a newer version of them on the server, or the cache is cleared.

23. Interesting Facts The first Celeron CPU had no caching, however it performed almost as fast as the PII with cache. Because some chipsets couldn’t cache more than the first 64mb of RAM a rumor was spread that Windows 98 could only use up to 64mb of RAM, however in reality it supports up to 2GB of RAM. In these chipsets, adding more than 64MB of RAM decreased performance.