1. ECE 445 – Computer Organization
The Memory Hierarchy
(Lectures #22)
The slides included herein were taken from the materials accompanying
Computer Organization and Design, 4th Edition, by Patterson and Hennessey,
and were used with permission from Morgan Kaufmann Publishers.

2. ECE 445 - Computer Organization
Material to be covered ...
Chapter 5: Sections 1 – 5, 11 – 12
Fall 2010
ECE Computer Organization

3. ECE 445 - Computer Organization
Address Subdivision
Fall 2010
ECE Computer Organization

4. Example: Larger Block Size
64 blocks, 16 bytes/block (i.e. 4 words/block)
To what block number does address 1200 map?
What about address 1212?
(Memory) Block address = 1200/16 = 75
(Cache) Block number = 75 modulo 64 = 11
Tag
Index
Offset 3 4 9 10 31
4 bits
6 bits
22 bits
Fall 2010
ECE Computer Organization

5. Block Size Considerations
Larger blocks should reduce miss rate
Due to spatial locality
But in a fixed-sized cache
Larger blocks  fewer of them
More competition  increased miss rate
Larger blocks  pollution
Larger miss penalty
Can override benefit of reduced miss rate
Early restart and critical-word-first can help
Fall 2010
ECE Computer Organization

6. ECE 445 - Computer Organization
Cache Misses
On cache hit, CPU proceeds normally
On cache miss
Stall the CPU pipeline
Fetch block from next level of memory hierarchy
Instruction cache miss
Restart instruction fetch
Data cache miss
Complete data access
Fall 2010
ECE Computer Organization

7. ECE 445 - Computer Organization
Write-Through
On data-write hit, could just update the block in cache
But then cache and memory would be inconsistent
Write-through: also update main memory
But makes writes take longer
e.g., if base CPI = 1, 10% of instructions are stores, write to memory takes 100 cycles
Effective CPI = ×100 = 11
Solution: write buffer
Holds data waiting to be written to memory
CPU continues immediately
Only stalls on write if write buffer is already full
Fall 2010
ECE Computer Organization

8. ECE 445 - Computer Organization
Write-Back
Alternative to Write-Through
On data-write hit, just update the block in cache
Keep track of whether each block is dirty
When a dirty block is replaced
Write it back to memory
Can use a write buffer to allow replacing block to be read first
Fall 2010
ECE Computer Organization

9. ECE 445 - Computer Organization
Write Allocation
What should happen on a write miss?
Alternatives for write-through
Allocate on miss: fetch the block
Write around: don’t fetch the block
For write-back
Usually fetch the block
Fall 2010
ECE Computer Organization

10. Example: Intrinsity FastMATH
Embedded MIPS processor
12-stage pipeline
Instruction and data access on each cycle
Split cache: separate I-cache and D-cache
Each 16KB: 256 blocks × 16 words/block
D-cache: write-through or write-back (configurable)
SPEC2000 miss rates
I-cache: 0.4%
D-cache: 11.4%
Weighted average: 3.2%
Fall 2010
ECE Computer Organization

11. Example: Intrinsity FastMATH
Fall 2010
ECE Computer Organization

12. Main Memory Supporting Caches
Use DRAMs for main memory
Fixed width (e.g., 1 word)
Connected by fixed-width clocked bus
Bus clock is typically slower than CPU clock
Example: cache block read from main memory
1 bus cycle for address transfer (to main memory)
15 bus cycles per DRAM access
1 bus cycle per data transfer (from main memory)
For 4-word block, 1-word-wide DRAM
Miss penalty = 1 + 4×15 + 4×1 = 65 bus cycles
Bandwidth = 16 bytes / 65 cycles = 0.25 B/cycle
Fall 2010
ECE Computer Organization

13. Increasing Memory Bandwidth
4-word wide memory
Miss penalty = = 17 bus cycles
Bandwidth = 16 bytes / 17 cycles = 0.94 B/cycle
4-bank interleaved memory
Miss penalty = ×1 = 20 bus cycles
Bandwidth = 16 bytes / 20 cycles = 0.8 B/cycle
Fall 2010
ECE Computer Organization

14. ECE 445 - Computer Organization
Cache Performance
Fall 2010
ECE Computer Organization

15. Measuring Cache Performance
Components of CPU time
Program execution cycles
Includes cache hit time
Memory stall cycles
Mainly from cache misses
With simplifying assumptions:
Fall 2010
ECE Computer Organization

16. Cache Performance Example
Given
I-cache miss rate = 2%
D-cache miss rate = 4%
Miss penalty = 100 cycles
Base CPI (ideal cache) = 2
Load & stores are 36% of instructions
Miss cycles per instruction
I-cache: 0.02 × 100 = 2
D-cache: 0.36 × 0.04 × 100 = 1.44
Actual CPI = = 5.44
Ideal CPU is 5.44/2 =2.72 times faster
Fall 2010
ECE Computer Organization

17. ECE 445 - Computer Organization
Average Access Time
Hit time is also important for performance
Average memory access time (AMAT)
AMAT = Hit time + Miss rate × Miss penalty
Example
CPU with 1ns clock, hit time = 1 cycle, miss penalty = 20 cycles, I-cache miss rate = 5%
AMAT = × 20 = 2ns
2 cycles per instruction
Fall 2010
ECE Computer Organization

18. ECE 445 - Computer Organization
Performance Summary
When CPU performance increased
Miss penalty becomes more significant
Decreasing base CPI
Greater proportion of time spent on memory stalls
Increasing clock rate
Memory stalls account for more CPU cycles
Can’t neglect cache behavior when evaluating system performance
Fall 2010
ECE Computer Organization

19. ECE 445 - Computer Organization
Associative Caches
Fall 2010
ECE Computer Organization

20. ECE 445 - Computer Organization
Associative Caches
Fully associative
Allow a given block to go in any cache entry
Requires all entries to be searched at once
Comparator per entry (expensive)
n-way set associative
Each set contains n entries
Block number determines which set
(Block number) modulo (#Sets in cache)
Search all entries in a given set at once
n comparators (less expensive)
Fall 2010
ECE Computer Organization

21. Associative Cache Example
Memory address being accessed is in Block #12 in Main memory.
Fall 2010
ECE Computer Organization

22. Spectrum of Associativity
For a cache with 8 entries
Fall 2010
ECE Computer Organization

23. Associativity Example
Compare 4-block caches
Direct mapped (aka. 1-way set associative)
2-way set associative
Fully associative
Block access sequence: 0, 8, 0, 6, 8
Fall 2010
ECE Computer Organization

24. Associativity Example
Direct mapped
# of Cache Blocks = 4
Block address
Cache index
Hit/miss
Cache content after access 1 2 3
miss
Mem[0] 8
Mem[8] 6
Mem[6]
Cache index = (Block Address) modulo (# of Cache Blocks)
Fall 2010
ECE Computer Organization

25. Associativity Example
2-way set associative
# of Cache Sets = 2
# of Entries per Set = 2
Block address
Cache index
Hit/miss
Cache content after access
Set 0
Set 1
miss
Mem[0] 8
Mem[8]
hit 6
Mem[6]
Cache index = (Block Address) modulo (# of Sets in Cache)
Fall 2010
ECE Computer Organization

26. Associativity Example
Fully associative
# of Cache Sets = 1
# of Entries per Set = 4
Block address
Hit/miss
Cache content after access
miss
Mem[0] 8
Mem[8]
hit 6
Mem[6]
Any memory address can be located in any entry of the cache.
Fall 2010
ECE Computer Organization

27. How Much Associativity
Increased associativity decreases miss rate
But with diminishing returns
Simulation of a system with 64KB D-cache,
16-word blocks; using SPEC2000 benchmark
1-way: 10.3%
2-way: 8.6%
4-way: 8.3%
8-way: 8.1%
Fall 2010
ECE Computer Organization

28. Set Associative Cache Organization
4-way Set Associative Cache
Fall 2010
ECE Computer Organization

29. ECE 445 - Computer Organization
Questions?
Fall 2010
ECE Computer Organization