b10100 Numerical Accuracy ENGR xD52 Eric VanWyk Fall 2012

Acknowledgements Mark L. Chang lecture notes for Computer Architecture (Olin ENGR3410)

Today Fast = Expensive Slow = Cheap Computer Architecture = Best of Both Worlds Because we are awesome

Recap So far, we have three types of memory: – Registers (between pipeline stages) – Register File – Giant Magic Black Box for lw and sw Inside the box is CompArch’s favorite thing: – More Black Boxes

Recap So far, we have three types of memory: – Registers (between pipeline stages) – Register File – Giant Magic Black Box for lw and sw Inside the box is CompArch’s favorite thing: – More Black Boxes – WITH KNOBS ON THEM WE CAN ADJUST

Cost vs Speed Faster Memories are more expensive per bit – Slow and Expensive technologies are phased out – Expensive means chip area, which means $$$ TechnologyAccess Time 2004 $/GB in 2004$/GB in 2013 SRAM0.5-5 ns$4000-$10,000$1400 DRAM50-70 ns$100-$200>$3 (Nov ‘12) >$6 (Nov ‘13) Disk (HDD)(5-20)x10 6 ns$0.50-$2>$.05 Disk (SSD)>100ns$0.60

Static Random Access Memory Like a register file, but: – Only one Port (unified read/write) – Many more words (Deeper) – Several of you accidentally invented it Different Cell Construction – Smaller than a D-Flip Flop Different Word Select Construction – Much fancier to allow for poorer cells – Hooray for VLSI

SRAM Cell M1-M3 store bit weakly Word Line accesses cell – Open M5, M6 Bit Lines – Read bit (weak signal) – Write bit (strong signal) Squint to see SR latch!

DRAM Cell Capacitor stores bit – For a while – Must be “refreshed” FET controls access Smaller than SRAM Slower than SRAM

Technology Trends Processor-DRAM Memory Gap (latency) µProc 60%/yr. (2X/1.5yr) DRAM 9%/yr. (2X/10 yrs) DRAM CPU 1982 Processor-Memory Performance Gap: (grows 50% / year) Performance Time “Moore’s Law”

The Problem The Von Neumann Bottleneck – Logic gets faster – Memory capacity gets larger – Memory speed is not keeping up with logic How do we get the full Daft Punk experience? – Faster, Bigger, Cheaper, Stronger?

Fast, Big, Cheap: Pick 3 Design Philosophy – Use a hybrid approach that uses aspects of both – Lots of Slow’n’Cheap – Small amount of Fast’n’Costly “Cache” – Make the common case fast – Keep frequently used things in a small amount of fast/expensive memory

The Solution By taking advantage of the principle of locality: – Provide as much memory as is available in the cheapest technology. – Provide access at the speed offered by the fastest technology. Control Datapath Secondary Storage (Disk) Processor Registers Main Memory (DRAM) Second Level Cache (SRAM) On-Chip Cache NameRegisterCacheMain MemoryDisk Memory Speed<1ns<10ns60ns10 ms Size100 BsKBsMBsGBs

Cache Terminology Hit: Data appears in that level – Hit rate – percent of accesses hitting in that level – Hit time – Time to access this level Hit time = Access time + Time to determine hit/miss Miss: Data does not appear in that level and must be fetched from lower level – Miss rate – percent of misses at that level = (1 – hit rate) – Miss penalty – Overhead in getting data from a lower level Miss penalty = Lower level access time + Replacement time + Time to deliver to processor Miss penalty is usually MUCH larger than the hit time

Cache Access Time Average access time – Access time = (hit time) + (miss penalty)x(miss rate) – Want high hit rate & low hit time, since miss penalty is large Average Memory Access Time (AMAT) – Apply average access time to entire hierarchy.

Handling A Cache Miss Data Miss 1.Stall the pipeline (freeze following instructions) 2.Instruct memory to perform a read and wait 3.Return the result from memory and allow the pipeline to continue Instruction Miss 1.Send the original PC to the memory 2.Instruct memory to perform a read and wait (no write enables) 3.Write the result to the appropriate cache line 4.Restart the instruction

Cache Access Time Example Note: Numbers are local hit rates – the ratio of access that go to that cache that hit (remember, higher levels filter accesses to lower levels) LevelHit TimeHit Rate Access Time L11 cycle95% L210 cycles90% Main Memory50 cycles99% Disk50,000 cycles100%

Cache Access Time Example Note: Numbers are local hit rates – the ratio of access that go to that cache that hit (remember, higher levels filter accesses to lower levels) LevelHit TimeHit Rate Access Time L11 cycle95% L210 cycles90% Main Memory50 cycles99% Disk50,000 cycles100%50,000

Cache Access Time Example Note: Numbers are local hit rates – the ratio of access that go to that cache that hit (remember, higher levels filter accesses to lower levels) LevelHit TimeHit Rate Access Time L11 cycle95% L210 cycles90% Main Memory50 cycles99% * = 550 Disk50,000 cycles100%50,000

Cache Access Time Example Note: Numbers are local hit rates – the ratio of access that go to that cache that hit (remember, higher levels filter accesses to lower levels) LevelHit TimeHit Rate Access Time L11 cycle95% L210 cycles90% * 550 = 65 Main Memory50 cycles99% * = 550 Disk50,000 cycles100%50,000

Cache Access Time Example Note: Numbers are local hit rates – the ratio of access that go to that cache that hit (remember, higher levels filter accesses to lower levels) LevelHit TimeHit Rate Access Time L11 cycle95% * 65 = 4.25 L210 cycles90% * 550 = 65 Main Memory50 cycles99% * = 550 Disk50,000 cycles100%50,000

How do we get those Hit Rates? There ain’t no such thing as a free lunch – If access was totally random, we’d be hosed – But we have knowledge a priori about programs Locality – Temporal Locality – If an item has been accessed recently, it will tend to be accessed again soon – Spatial Locality – If an item has been accessed recently, nearby items will tend to be accessed soon

Example What does this code do? What type(s) of locality does it have? char *index = string; while (*index != 0) { /* C strings end in 0 */ if (*index >= ‘a’ && *index <= ‘z’) *index = *index +(‘A’ - ‘a’); index++; }

Exploiting Locality Temporal locality – Keep more recently accessed items closer to the processor – When we must evict items to make room for new ones, attempt to keep more recently accessed items Spatial locality – Move blocks consisting of multiple contiguous words to upper level

Basic Cache Design 2^n Blocks of Data 2^M bytes wide Tag indicates what block is being stored n -1 ……… … Valid BitTag (Address)Data ………

Whose Line is it, Anyway? How do we map System Memory into Cache? Idea One: Direct Mapping – Each address maps to a single cache line – Lower M bits are address within a block – Next N bits are the cache line address – Remainder are the address Tag N bitsM bitsAddress Tag

Direct Mapped Cache A B C E D F Cache Address

Direct Mapped Cache A B C E D F Cache Address

Direct Mapped Cache A B C E D F Cache Address M[C] Tag C / 4 = 3

Cache Access Example Assume 4 byte cache (N=2, M=0) Access pattern: Valid Bit Tag Data

Cache Access Example Assume 4 byte cache (N=2, M=0) Access pattern: Valid Bit Tag Data M[00001]

Cache Access Example Assume 4 byte cache (N=2, M=0) Access pattern: Valid Bit Tag Data M[00001] M[00110]001

Cache Access Example Assume 4 byte cache (N=2, M=0) Access pattern: Valid Bit Tag Data M[00001] M[00110]001

Cache Access Example Assume 4 byte cache (N=2, M=0) Access pattern: Valid Bit Tag Data M[00001] M[00110]001 != 110

Direct Mapping Simple Control Logic Great Spatial Locality Awful Temporal Locality When does this fail in a direct mapped cache? char *image1, *image2; int stride; for(int i=0;i<size;i+=stride) { diff +=abs(image1[i] - image2[i]); }

Block Size Tradeoff With fixed cache size, increasing Block size(M): – Worse Miss penalty: longer to fill block – Better Spatial Locality – Worse Temporal Locality Miss Penalty Block Size Miss Rate Exploits Spatial Locality Fewer blocks: compromises temporal locality Average Access Time Increased Miss Penalty & Miss Rate Block Size

N-Way Associative Mapping Like N parallel direct map caches = =? … Valid … TagBlock …… Valid … TagBlock … = =? Addr Cache Tag Hit Cache Block

Associative Mapping Can handle up to N conflicts without eviction Better Temporal Locality – Assuming good Eviction policy More Complicated Control Logic – Slower to confirm hit or miss – Slower to find associated cache line

Fully Associative Cache n -1 … …… … Valid BitTagData …… … 012 m -1 = =? Full Associative = 2^N Way Associative – Everything goes Anywhere!

What gets evicted? Approach #1: Random – Just arbitrarily pick from possible locations Approach #2: Least Recently Used (LRU) – Use temporal locality – Must track somehow – extra bits to recent usage In practice, Random ~12% worse than LRU

Cache Arrangement Direct Mapped – Memory addresses map to particular location in that cache – 1-Way Associative Set Fully Associative – Data can be placed anywhere in the cache – 2^n-Way Associative Set N-way Set Associative – Data can be placed in a limited number of places in the cache depending upon the memory address

My L1 Data Cache M = ? N = ? Tag Size = ?

My L1 Data Cache M = 6 bits – 64-byte line size – 64 byte blocks – 2^6 = 64 N = 6 bits – 32kByte -> 15 bits – 8 way -> 3 fewer bits – M=6 -> 6 fewer bits Tag Size = 52 bits – 64 bit address space – Whatever is left over

3 C’s of Cache Misses Compulsory/Coldstart – First access to a block – basically unavoidable – For long-running programs this is a small fraction of misses Capacity – Had been loaded, evicted by too many other accesses – Can only be mitigated by cache size Conflict – Had been loaded, evicted by mapping conflict – Mitigated by associativity

Cache Miss Example 8-word cache, 8-byte blocks. Determine types of misses (CAP, COLD, CONF). Byte AddrBlock AddrDirect Mapped2-Way AssocFully Assoc Total Miss:

Cache Miss Example 8-word cache, 8-byte blocks. Determine types of misses (CAP, COLD, CONF). Byte AddrBlock AddrDirect Mapped2-Way AssocFully Assoc 00 Cold 40 Hit (in 0’s block) 81 Cold 243 Cold 567 Cold 81 Conf (w/56) 243 Hit 162 Cold 00 Hit Total Miss: 6

Split Caches Often separate Instruction and Data Caches – Harvard Architecture: Split I & D – von Neumann Architecture: Unified Higher bandwidth Optimize to usage Slightly higher miss rate: each cache is smaller

With Remaining Time Finish Cache Example (Answers Below) Find out what your computer’s caches are Write your own “Pop Quiz” on Caching We will exchange & take quizes – Not graded – Then discuss

Quiz Ideas Describe a specific Cache: – How big is the tag? Where does address X map? Describe an access pattern: – Hit Rate? Average Access Time?

Cache Summary Software Developers must respect the cache! – Usually a good place to start optimizing Especially on laptops/desktops/servers – Cache Line size? Page size? Cache design implies a usage pattern – Very good for instruction & stack – Not as good for Heap – Modern languages are very heapy

Matrix Multiplication for (k = 0; k < n; k++){ for (i = 0; i < n; i++){ for (j = 0; j < n; j++){ c[k][i] = c[k][i] + a[k][j]*b[j][i]; }}} Vs for (k = 0; k < n; k++){ for (i = 0; i < n; i++){ for (j = 0; j < n; j++){ c[i][j] = c[i][j] + a[i][k]*b[k][j]; }}} Which has better locality?

Cache Miss Comparison Fill in the blanks: Zero, Low, Medium, High, Same for all Direct MappedN-Way Set AssociativeFully Associative Cache Size: Small, Medium, Big? Compulsory Miss: Capacity Miss Conflict Miss Invalidation Miss

Cache Miss Comparison Fill in the blanks: Zero, Low, Medium, High, Same for all Direct MappedN-Way Set AssociativeFully Associative Cache Size: Small, Medium, Big? Big (Few comparators) MediumSmall (lots of comparators) Compulsory Miss:Same Capacity MissLowMediumHigh Conflict MissHighMediumZero Invalidation MissSame

Cache Miss Example 8-word cache, 8-byte blocks. Determine types of misses (CAP, COLD, CONF). Byte AddrBlock AddrDirect Mapped2-Way AssocFully Assoc 00 Cold 40 Hit (in 0’s block) 81 Cold 243 Cold 567 Cold 81 Conf (w/56) Hit 243 HitConf (w/8)Hit 162 Cold 00 Hit Cap Total Miss: 676

56 Replacement Methods If we need to load a new cache line, where does it go? Direct-mapped Only one possible location Set Associative N locations possible, optimize for temporal locality? Fully Associative All locations possible, optimize for temporal locality?

Cache Access Example Assume 4 byte cache Access pattern: Valid Bit Tag Data

58 Cache Access Example Assume 4 byte cache Access pattern: M[00001] M[00110] Valid Bit Tag Data Compulsory/Cold Start miss

59 Cache Access Example (cont.) Assume 4 byte cache Access pattern: M[00001] M[11010] Valid Bit Tag Data Compulsory/Cold Start miss

60 Cache Access Example (cont. 2) Assume 4 byte cache Access pattern: M[00001] M[00110] Valid Bit Tag Data Conflict Miss