1. ECE 445 – Computer Organization
The Memory Hierarchy
(Lectures #23)
The slides included herein were taken from the materials accompanying
Computer Organization and Design, 4th Edition, by Patterson and Hennessey,
and were used with permission from Morgan Kaufmann Publishers.

2. ECE 445 - Computer Organization
Material to be covered ...
Chapter 5: Sections 1 – 5, 11 – 12
Fall 2010
ECE Computer Organization

3. ECE 445 - Computer Organization
Associative Caches
Fall 2010
ECE Computer Organization

4. ECE 445 - Computer Organization
Associative Caches
Fully associative
Allow a given block to go in any cache entry
Requires all entries to be searched at once
Comparator per entry (expensive)
n-way set associative
Each set contains n entries
Block number determines which set
(Block number) modulo (#Sets in cache)
Search all entries in a given set at once
n comparators (less expensive)
Fall 2010
ECE Computer Organization

5. Associative Cache Example
Memory address being accessed is in Block #12 in Main memory.
Fall 2010
ECE Computer Organization

6. Spectrum of Associativity
For a cache with 8 entries
Fall 2010
ECE Computer Organization

7. Associativity Example
Compare 4-block caches
Direct mapped (aka. 1-way set associative)
2-way set associative
Fully associative
Block access sequence: 0, 8, 0, 6, 8
Fall 2010
ECE Computer Organization

8. Associativity Example
Direct mapped
# of Cache Blocks = 4
Block address
Cache index
Hit/miss
Cache content after access 1 2 3
miss
Mem[0] 8
Mem[8] 6
Mem[6]
Cache index = (Block Address) modulo (# of Cache Blocks)
Fall 2010
ECE Computer Organization

9. Associativity Example
2-way set associative
# of Cache Sets = 2
# of Entries per Set = 2
Block address
Cache index
Hit/miss
Cache content after access
Set 0
Set 1
miss
Mem[0] 8
Mem[8]
hit 6
Mem[6]
Cache index = (Block Address) modulo (# of Sets in Cache)
Fall 2010
ECE Computer Organization

10. Associativity Example
Fully associative
# of Cache Sets = 1
# of Entries per Set = 4
Block address
Hit/miss
Cache content after access
miss
Mem[0] 8
Mem[8]
hit 6
Mem[6]
Any memory address can be located in any entry of the cache.
Fall 2010
ECE Computer Organization

11. How Much Associativity
Increased associativity decreases miss rate
But with diminishing returns
Simulation of a system with 64KB D-cache,
16-word blocks; using SPEC2000 benchmark
1-way: 10.3%
2-way: 8.6%
4-way: 8.3%
8-way: 8.1%
Fall 2010
ECE Computer Organization

12. Set Associative Cache Organization
4-way Set Associative Cache
Fall 2010
ECE Computer Organization

13. ECE 445 - Computer Organization
Replacement Policy
Fall 2010
ECE Computer Organization

14. ECE 445 - Computer Organization
Replacement Policy
Direct mapped: no choice
Each block in main memory is mapped to exactly one location in the cache.
Set associative
Prefer non-valid entry, if there is one
Otherwise, choose among entries in the set
Least-recently used (LRU)
Choose the one unused for the longest time
Simple for 2-way, manageable for 4-way, too hard beyond that.
Random
Gives approximately the same performance as LRU for high associativity.
Fall 2010
ECE Computer Organization

15. ECE 445 - Computer Organization
Multi-level Caches
Fall 2010
ECE Computer Organization

16. ECE 445 - Computer Organization
Multilevel Caches
Primary (L1) cache attached to CPU
Small, but fast
Level-2 (L2) cache services misses from primary cache
Larger, slower, but still faster than main memory
Main memory services L2 cache misses
Some high-end systems include L3 cache
Main
Memory
CPU L2 L1
Fall 2010
ECE Computer Organization

17. Multilevel Cache Example
Given
CPU base CPI = 1, clock rate = 4GHz
Miss rate/instruction = 2%
Main memory access time = 100ns
With just primary cache
Miss penalty = 100ns/0.25ns = 400 cycles
Effective CPI = × 400 = 9
Fall 2010
ECE Computer Organization

18. ECE 445 - Computer Organization
Example (cont.)
Now add L2 cache
Access time = 5ns
Global miss rate to main memory = 0.5%
Primary miss with L2 hit
Penalty = 5ns/0.25ns = 20 cycles
Primary miss with L2 miss
Extra penalty = (0.5%) * (100ns/0.25ns)
CPI = × × 400 = 3.4
Performance ratio = 9/3.4 = 2.6
Fall 2010
ECE Computer Organization

19. Multilevel Cache Considerations
Primary cache
Focus on minimal hit time
L2 cache
Focus on low miss rate to avoid main memory access
Hit time has less overall impact
Results
L1 cache usually smaller than a single cache
L1 block size smaller than L-2 block size
Fall 2010
ECE Computer Organization

20. ECE 445 - Computer Organization
Virtual Memory
Fall 2010
ECE Computer Organization

21. The term virtual memory as defined by Merriam-Webster:
“a section of a hard drive that can be used as if it were an extension of a computer's random-access memory”
Fall 2010
ECE Computer Organization

22. The term virtual memory as defined by Wikipedia:
“Virtual memory is a computer system technique which gives an application program the impression that it has contiguous working memory (an address space), while in fact it may be physically fragmented and may even overflow on to disk storage.”
Fall 2010
ECE Computer Organization

23. ECE 445 - Computer Organization
Courtesy of Ehamberg
(Wikipedia: Virtual Memory)
Fall 2010
ECE Computer Organization

24. ECE 445 - Computer Organization
Virtual Memory
Use main memory as a “cache” for secondary (disk) storage
Managed jointly by CPU hardware and the operating system (OS)
Programs share main memory
Each gets a private virtual address space holding its frequently used code and data
Protected from other programs
CPU and OS translate virtual addresses to physical addresses
VM “block” is called a page
VM translation “miss” is called a page fault
Fall 2010
ECE Computer Organization

25. ECE 445 - Computer Organization
Address Translation
Fixed-size pages (e.g., 4K)
Fall 2010
ECE Computer Organization

26. ECE 445 - Computer Organization
Page Fault Penalty
On page fault, the page must be fetched from disk
Takes millions of clock cycles
Handled by OS code
Try to minimize page fault rate
Fully associative placement
Smart replacement algorithms
Fall 2010
ECE Computer Organization

27. Page Tables According to Wikipedia:
“A page table is the data structure used by a virtual memory system in a computer operating system to store the mapping between virtual addresses and physical addresses. Virtual addresses are those unique to the accessing process. Physical addresses are those unique to the CPU, i.e., RAM.”
Fall 2010
ECE Computer Organization

28. ECE 445 - Computer Organization
Page Tables
Stores placement information
Array of page table entries, indexed by virtual page number
Page table register in CPU points to page table in physical memory
If page is present in memory
PTE stores the physical page number
Plus other status bits (referenced, dirty, …)
If page is not present
PTE can refer to location in swap space on disk
Fall 2010
ECE Computer Organization

29. Translation Using a Page Table
Fall 2010
ECE Computer Organization

30. Mapping Pages to Storage
Fall 2010
ECE Computer Organization

31. ECE 445 - Computer Organization
Questions?
Fall 2010
ECE Computer Organization