Prof. John Nestor ECE Department Lafayette College Easton, Pennsylvania ECE Computer Organization Lecture 20 - Memory Hierarchy 1 Fall 2004 Reading: Portions of these slides are derived from: Textbook figures © 1998 Morgan Kaufmann Publishers all rights reserved Tod Amon's COD2e Slides © 1998 Morgan Kaufmann Publishers all rights reserved Dave Patterson’s CS 152 Slides - Fall 1997 © UCB Rob Rutenbar’s Slides - Fall 1999 CMU other sources as noted

ECE 313 Fall 2004Lecture 20 - Memory2 Roadmap for the term: major topics  Overview / Abstractions and Technology  Instruction sets  Logic & arithmetic  Performance  Processor Implementation  Single-cycle implemenatation  Multicycle implementation  Pipelined Implementation  Memory systems   Input/Output

ECE 313 Fall 2004Lecture 20 - Memory3 Outline - Memory Systems  Overview   Motivation  General Structure and Terminology  Memory Technology  Static RAM  Dynamic RAM  Disks  Cache Memory  Virtual Memory

ECE 313 Fall 2004Lecture 20 - Memory4 Memory Systems - the Big Picture  Memory provides processor with  Instructions  Data  Problem: memory is too slow and too small Control Datapath Memory Processor Input Output Instructions Data “Five Classics Components” Picture

ECE 313 Fall 2004Lecture 20 - Memory5 Memory Hierarchy - the Big Picture  Problem: memory is too slow and too small  Solution: memory hierarchy Fastest Slowest Smallest Biggest Highest Lowest Speed: Size: Cost: Control Datapath Secondary Storage (Disk) Processor Registers L2 Off-Chip Cache Main Memory (DRAM) L1 On-Chip Cache

ECE 313 Fall 2004Lecture 20 - Memory6 Why Hierarchy Works  The principle of locality  Programs access a relatively small portion of the address space at any instant of time.  Temporal locality: recently accessed data is likely to be used again  Spatial locality: data near recently accessed data is likely to be used soon  Result: the illusion of large, fast memory Address Space 02 n - 1 Probability of reference

ECE 313 Fall 2004Lecture 20 - Memory7 Memory Hierarchy - Speed vs. Size Control Datapath Secondary Storage (Disk) Processor Registers L2 Off-Chip Cache Main Memory (DRAM) L1 On-Chip Cache ,000,000 (5ms)Speed (ns): <1K Size (bytes):>100G <16G<16M

ECE 313 Fall 2004Lecture 20 - Memory8 Memory Hierarchy - Terminology Processor Blocks of Data Hit: Data in Upper Level Miss: Data not in Upper Level

ECE 313 Fall 2004Lecture 20 - Memory9 Memory Hierarchy Terminology (cont’d)  Hit: data appears in some block in the upper level (green block)  Hit Rate: the fraction of memory accesses that “hit”  Hit Time: time to access the upper level (time to determine hit/miss + access time)  Miss: data must be retrieved from block in lower level (orange block)  Miss Rate = 1 - (Hit Rate)  Miss Penalty: Time to replace block in upper level + Time to deliver data to the processor  Hit Time > Miss Rate

ECE 313 Fall 2004Lecture 20 - Memory10 Typical Memory Hierarchy - Details  Registers - Small, fastest on-chip storage  Managed by compiler and run-time system  Cache - Small, fast on-chip storage  Associative lookup - managed by hardware  Memory - Slower, Larger off-chip storage  Limited size <16Gb - managed by hardware, OS  Disk - Slowest, Largest off-chip storage  Virtual memory - simulate a large memory using disk, hardware, and operating system  File storage - store data files using operating system

ECE 313 Fall 2004Lecture 20 - Memory11 Outline - Memory Systems  Overview  Motivation  General Structure and Terminology  Memory Technology   Static RAM  Dynamic RAM  Cache Memory  Virtual Memory

ECE 313 Fall 2004Lecture 20 - Memory12 Memory Types  Static RAM  Storage using latch circuits  Values saved while power on  Dynamic RAM  Storage using capacitors  Values must be refreshed bit word / row select bit C

ECE 313 Fall 2004Lecture 20 - Memory13 Tradeoffs - Static vs. Dynamic RAM  Static RAM (SRAM) - used for L1, L2 cache  Fast ns access time (less for on-chip)  Larger, More Expensive  Higher power consumption  Dynamic RAM (DRAM) - used for PC main memory  Slower ns access time*  Smaller, Cheaper  Lower power consumption

ECE 313 Fall 2004Lecture 20 - Memory14 DRAM Organization Row Decoder Column Selector / Latch / IO Row Address Column Address /RAS /CAS DATA Row Select Line Bit (data) Line

ECE 313 Fall 2004Lecture 20 - Memory DRAM Read Operation Row Decoder Column Selector / Latch / IO Row Address Column Address /RAS /CAS DATA

ECE 313 Fall 2004Lecture 20 - Memory16 DRAM Trends  RAM size: 4X every 3 years  RAM speed: 2X every 10 years DRAM YearSizeCycle Time Kb250 ns Kb220 ns Mb190 ns Mb165 ns Mb145 ns Mb120 ns 1997?128 Mb ?? ns 1999?256 Mb?? ns Size change: 1000:1! Speed change: 2:1!

ECE 313 Fall 2004Lecture 20 - Memory17 The Processor/Memory Speed Gap DRAM 9%/yr. (2X/10 yrs) DRAM CPU 1982 Processor-Memory Performance Gap: (grows 50% / year) Performance Time “Moore’s Law”

ECE 313 Fall 2004Lecture 20 - Memory18 Addressing the Speed Gap  Latency depends on physical limitations  Bandwidth can be increased using:  Parallelism - transfer more bits / word  Burst transfers - transfer successive words on each cycle  So... use bandwidth to support memory hierarchy!  Use cache to support locality of reference  Design hierarchy to transfer large blocks of memory

ECE 313 Fall 2004Lecture 20 - Memory19 Current DRAM Parts  Synchronous DRAM (SDRAM) - clocked transfer of bursts of data starting at a specific address  Double-Data Rate SDRAM - transfer two bits/clock cycle  Quad-Data Rate SDRAM - transfer four bits / clock cycle  Rambus RDRAM - High-speed interface for fast transfers  Current PCs use some form of SDRAM/RDRAM  SDRAM w/ PC100 or PC133 memory bus  RDRAM w/ PC800 memory bus

ECE 313 Fall 2004Lecture 20 - Memory20 Memory Configuration in Current PCs Processor System Controller L1 Cache Main Memory (DRAM) L2/L3 Cache (SRAM) (I/O Bus)

ECE 313 Fall 2004Lecture 20 - Memory21 Outline - Memory Systems  Overview  Motivation  General Structure and Terminology  Memory Technology  Static RAM  Dynamic RAM  Cache Memory   Virtual Memory

ECE 313 Fall 2004Lecture 20 - Memory22 CPU Hit: Data in Cache (no penalty) Miss: Data not in Cache (miss penalty) Cache Memory DRAM Memory Processor addrdata addrdata Cache Operation  Insert between CPU, Main Mem.  Implement with fast Static RAM  Holds some of a program’s  data  instructions  Operation:

ECE 313 Fall 2004Lecture 20 - Memory23 Four Key Cache Questions: 1.Where can block be placed in cache? (block placement) 2.How can block be found in cache? (block identification) 3.Which block should be replaced on a miss? (block replacement) 4.What happens on a write? (write strategy)

ECE 313 Fall 2004Lecture 20 - Memory24 Basic Cache Design  Organized into blocks or lines  Block Contents  tag - extra bits to identify block (part of block address)  data - data or instruction words- contiguous memory locations  Our example:  One-word (4 byte) block size  30-bit tag  Two blocks in cache CPU tag 0data 0 CPU tag 1data 1 0x x x x C 0x b0 b1 Cache Main Memory

ECE 313 Fall 2004Lecture 20 - Memory25 Cache Example (2)  Assume:  r1==0, r2==1, r4==2  1 cycle for cache access  5 cycles for main. mem. access  1 cycle for instr. execution  At cycle 1 - PC=0x00  Fetch instruction from memory look in cache MISS - fetch from main mem (5 cycle penalty) CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L MISSMISS

ECE 313 Fall 2004Lecture 20 - Memory26 Cache Example (3)  At cycle 6  Execute instr. add r1,r1,r2 CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…000 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x…0

ECE 313 Fall 2004Lecture 20 - Memory27 Cache Example (4)  At cycle 6 - PC=0x04  Fetch instruction from memory look in cache MISS - fetch from main mem (5 cycle penalty) CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x…0 MISSMISS 6-10 FETCH 0x…4

ECE 313 Fall 2004Lecture 20 - Memory28 Cache Example (5)  At cycle 11  Execute instr. bne r4,r1,L CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…000 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…004 bne r4,r1,L 0x…1 110x…4 bne r4,r1,L 1

ECE 313 Fall 2004Lecture 20 - Memory29 Cache Example (6)  At cycle 11 - PC=0x00  Fetch instruction from memory  HIT - instruction in cache CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…4 bne r4,r1,L 0x…1 HITHIT 110x…4 bne r4,r1,L 1 11 FETCH 0x…0 1

ECE 313 Fall 2004Lecture 20 - Memory30 Cache Example (7)  At cycle 12  Execute add r1, r1, 2 CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…4 bne r1,r2,L 0x…1 110x…4 bne r4,r1,L 1 12 FETCH 0x… add r1,r1,2 2

ECE 313 Fall 2004Lecture 20 - Memory31 Cache Example (8)  At cycle 12 - PC=0x04  Fetch instruction from memory  HIT - instruction in cache CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…4 bne r4,r1,L 0x…1 110x…4 bne r4,r1,L 1 12 FETCH 0x… add r1,r1, FETCH 0x04 HITHIT

ECE 313 Fall 2004Lecture 20 - Memory32 Cache Example (9)  At cycle 13  Execute instr. bne r4, r1, L  Branch not taken CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…4 bne r4,r1,L 0x…1 110x…4 bne r4,r1,L 1 12 FETCH 0x… add r1,r1, FETCH 0x04 13 bne r4, r1, L

ECE 313 Fall 2004Lecture 20 - Memory33 Cache Example (10)  At cycle 13 - PC=0x08  Fetch Instruction from Memory  MISS - not in cache CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…4 bne r4,r1,L 0x…1 110x…4 bne r4,r1,L 1 12 FETCH 0x… add r1,r1, FETCH 0x04 13 bne r4, r1, L 13 FETCH 0x08 MISSMISS

ECE 313 Fall 2004Lecture 20 - Memory34 Cache Example (11)  At cycle 17 - PC=0x08  Put instruction into cache  Replace existing instruction CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…4 bne r4,r1,L 0x…1 110x…4 bne r4,r1,L 1 12 FETCH 0x… add r1,r1, FETCH 0x04 13 bne r4, r1, L FETCH 0x08 sub r1,r1,r1 0x…2

ECE 313 Fall 2004Lecture 20 - Memory35 Cache Example (12)  At cycle 18  Execute sub r1, r1, r1 CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r FETCH 0x…4 bne r4,r1,L 0x…1 110x…4 bne r4,r1,L 1 12 FETCH 0x… add r1,r1, FETCH 0x bne r4, r1, L FETCH 0x sub r1, r1, r1 0 sub r1,r1,r1 0x…2

ECE 313 Fall 2004Lecture 20 - Memory36 Cache Example (13)  At cycle 18  Fetch instruction from memory  MISS - not in cache CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…4 bne r4,r1,L 0x…1 110x…4 bne r4,r1,L 1 12 FETCH 0x… add r1,r1, FETCH 0x bne r4, r1, L FETCH 0x08 2 sub r1,r1,r1 18 sub r1, r1, r FETCH 0x0C MISSMISS

ECE 313 Fall 2004Lecture 20 - Memory37 Cache Example (14)  At cycle 22  Put instruction into cache  Replace existing instruction CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r2 1 L: add r1,r1,r2 0x… FETCH 0x…4 bne r1,r2,L 0x…1 110x…4 bne r4,r1,L 1 12 FETCH 0x… add r1,r1, FETCH 0x bne r4, r1, L FETCH 0x sub r1, r1, r FETCH 0x0C j L 0x…3 sub r1,r1,r1 0x…2

ECE 313 Fall 2004Lecture 20 - Memory38 Cache Example (15) CycleAddressOp/Instr. r1 1-5 FETCH 0x…0 60x…0 add r1,r1,r FETCH 0x…4 110x…4 bne r3,r1,L 11 FETCH 0x…0 120x…8 add r1,r1, FETCH 0x…4 130x…4 bne r4,r1,L FETCH 0x…8 180x…8 sub r1,r1,r FETCH 0x..C 230x…8 j L CPU (empty) CPU (empty) L: add r1,r1,r2 0x x x x C 0x b0 b1 Cache Main Memory bne r4,r1,L sub r1,r1,r1 L: j L  At cycle 23  Execute j L j L 0x…3 sub r1,r1,r1 0x…2

ECE 313 Fall 2004Lecture 20 - Memory39 Compare No-cache vs. Cache CycleAddressOp/Instr. 1-5 FETCH 0x…0 60x…0 add r1,r1,r 6-10 FETCH 0x…4 110x…4 bne r3,r1,L FETCH 0x…0 160x…0 add r1,r1, FETCH 0x…4 210x…4 bne r3,r1,L FETCH 0x…8 260x…8 sub r1,r1,r FETCH 0x..C 310x…C j L CycleAddressOp/Instr. 1-5 FETCH 0x…0 60x…0 add r1,r1,r 6-10 FETCH 0x…4 110x…4 bne r3,r1,L 11 FETCH 0x…0 120x…0 add r1,r1,2 12 FETCH 0x…4 130x…4 bne r3,r1,L FETCH 0x…8 180x…8 sub r1,r1,r FETCH 0x..C 230x…C j L NO CACHE CACHE M M H H M M

ECE 313 Fall 2004Lecture 20 - Memory40 Cache Miss and the MIPS Pipeline Compare in Cycle 1 Fetch Completes (Pipeline Restarts) Miss Detected in Cycle 2  Instruction Fetch Clock Cycle 1 Clock Cycle 2+N Clock Cycle 3+N Clock Cycle 4+N Clock Cycle 5+N Clock Cycle 6+N

ECE 313 Fall 2004Lecture 20 - Memory41 Cache Miss and the MIPS Pipeline Compare in Cycle 4 Miss Detected in Cycle 5 Load Completes (Pipeline Restarts)  Load Instruction Clock Cycle 1 Clock Cycle 2 Clock Cycle 3 Clock Cycle 4 Clock Cycle 5 Clock Cycle 5+N Clock Cycle 6+N

ECE 313 Fall 2004Lecture 20 - Memory42 Coming Up: Four Key Cache Questions: 1.Where can block be placed in cache? (block placement) 2.How can block be found in cache? …using a tag (block identification) 3.Which block should be replaced on a miss? (block replacement) 4.What happens on a write? (write strategy)