CS/COE 1541 (term 2174) Jarrett Billingsley Memory – Caching CS/COE 1541 (term 2174) Jarrett Billingsley

Class Announcements AaaaaaAAAAAAAAAAAAAAAAAAAAAAAAAA one day, you'll have your work back. (I'm just pipelining my work) Homework 2 Little smaller! Due in one week Project 1! I had an idea but then I talked to someone and got another idea and aaaaaaaaaaaa But ultimately I think I'll stick with the first idea – writing a cache simulator in C You'll have about a month to do it, so no pressure but some of you will start it a few days before it's due so like whatever 2/8/2017 CS/COE 1541 term 2174

Caching 2/8/2017 CS/COE 1541 term 2174

Terms, terms... A cache holds temporary copies of data. A cache is made of multiple lines/blocks, which can each hold multiple data items. We'll be talking about the memory cache – the layer between the CPU registers and the system memory. Caches take advantage of locality: temporal (in time) and spatial (in space/addresses). When we want to access data, a hit means it's in the cache. A miss means it's not in the cache. Hit rate/miss rate are the percentage of accesses that hit or miss. Both hits and misses take time: hit time and miss penalty. 2/8/2017 CS/COE 1541 term 2174

For those of you who are unfamiliar with caches... I want to make a read-only cache! (we'll deal with writes later.) Let's just start with lw instructions. Here are some ideas to get started: Every memory access has an address. We need some way of seeing if things are in the cache. We need some way of inserting things into the cache. And what if you run out of cache space? Give me some ideas on how you'd approach designing a cache. I'm not looking for "right" answers! 2/8/2017 CS/COE 1541 term 2174

Cache design goals Unlike the BTB, false positives/negatives are unacceptable. Caching is not a predictive heuristic. It's the real data. Getting data from the cache when it hits should be faster than accessing the next lower level in the memory hierarchy. Kinda obvious, but this means we can't waste too many cycles on complex caching schemes, or the effort won't be worth it! We want our miss rate to be as low as possible... But miss rate isn't everything, as we'll see. 2/8/2017 CS/COE 1541 term 2174

Direct-mapped Caches 2/8/2017 CS/COE 1541 term 2174

Easy peasy In a direct-mapped cache, each real memory location maps to one location in the cache. Implementing this is easy! Memory 000000 000001 000010 000011 000100 000101 000110 000111 Cache For this 8-entry cache, to find the cache block for a memory address, take the lowest 3 address bits. 000 001 010 011 100 101 110 111 001000 001001 001010 001011 001100 001101 001110 001111 But if our program accesses 001000, then 000000, how do we tell them apart? (How did the BTB handle this?) 2/8/2017 CS/COE 1541 term 2174

Tags Each cache block has a tag which indicates the memory address that it corresponds to. For address 110011, 011 is the block, and 110 is the tag. This seems redundant... Tag Data 000 001000 DEADBEEF 001 010 011 110011 CAFEFACE 100 101100 B0DECA7 101 110 111 Tag Data 000 001 DEADBEEF 010 011 110 CAFEFACE 100 101 B0DECA7 111 How do we tell what entries are empty/full? Just add another bit (a valid bit)! 2/8/2017 CS/COE 1541 term 2174

Watching it in action When the program first starts, we set all the valid bits to 0. The cache is empty. Now let's try a sequence of reads... do these hit or miss? How do the cache contents change? V Tag Data 000 001 010 011 100 101 110 111 1 010 something 1 000 something 000000 100101 100100 010000 miss hit Why did these misses happen? 1 100 something 1 100 something Why did this miss happen? 2/8/2017 CS/COE 1541 term 2174

The 3 C's There are three ways we can have cache misses: Compulsory (or cold-start) misses: when you've never had that block in the cache before. Capacity misses: when the cache isn't big enough to hold all the data that you need. Collision (or conflict) misses: when two different addresses map to the same cache block. How could we improve/avoid each of these? Many of the techniques we'll be discussing will be focusing on reducing each category of miss. 2/8/2017 CS/COE 1541 term 2174

The steps and hardware Split the address into two (three?) parts: block and tag (and byte). a bus splitter Use the block index to find the right cache entry. a mux, or more realistically a decoder + tristate bus If the valid bit is 1, and the entry's tag matches, an equality comparator and an AND gate It's a hit! Read the cached data. Optionally, select the right byte/halfword, with a mux. Otherwise... It's a miss. Huh, what do we do now?? Stall! Or find something else to do, if you're out-of-order ;o 2/8/2017 CS/COE 1541 term 2174

How many bits will each have? One word per block? The example shows one word in each block. But is that the best? What about spatial locality? In reality, most caches put multiple words in each block. Common sizes are 4-8 words. (i7 Skylake uses 64 bytes = 8 8-byte words) Memory buses are usually designed to the same/half width as a cache block/line to improve bandwidth. Memory has high latency, but can have great throughput! Now the address is split into four parts... 0110 1010 0011 1000 tag block word byte How many bits will each have? 2/8/2017 CS/COE 1541 term 2174

Multiple words per block means… Multiple valid bits! A block can be either partly or completely valid. Each memory access has to check the per-word valid bit. This complicates the memory controller because it has to account for memory transfers smaller than a whole block… But it reduces memory traffic, and gives better granularity for determining validity. But when you have a conflict… You have to invalidate all the other words when you change tag. V Tag Data0 Data1 000 00 001 01 11 23 010 011 100 3A BB 101 110 111 2/8/2017 CS/COE 1541 term 2174

How big... Picking the right block size is a balancing act: Larger blocks reduce compulsory misses. But they increase competition for cache space. This is because for the same amount of cached data, you now have fewer rows! E.g. for a cache with 32 words, one word per block means 32 rows, but 2 words per block means only 16 rows. 2/8/2017 CS/COE 1541 term 2174

Handling writes 2/8/2017 CS/COE 1541 term 2174

oh boy Once you introduce writes, you greatly increase the complexity of cache behavior. The cache is supposed to be an accurate copy of the data in the next lower level. So what if you change the data in the cache? The cache is now inconsistent or invalid. How do we ensure consistency? What about exceptions???? What about multiprocessor/core systems?????? What about multiple caches that all cache the same memory???? All this, and more, on the next episode... 2/8/2017 CS/COE 1541 term 2174