CS 152 Computer Architecture and Engineering Lecture 8 - Memory Hierarchy-III Krste Asanovic Electrical Engineering and Computer Sciences University of.

Slides:



Advertisements
Similar presentations
Lecture 12 Reduce Miss Penalty and Hit Time
Advertisements

CSE 490/590, Spring 2011 CSE 490/590 Computer Architecture Cache III Steve Ko Computer Sciences and Engineering University at Buffalo.
CMSC 611: Advanced Computer Architecture Cache Some material adapted from Mohamed Younis, UMBC CMSC 611 Spr 2003 course slides Some material adapted from.
1 Adapted from UCB CS252 S01, Revised by Zhao Zhang in IASTATE CPRE 585, 2004 Lecture 14: Hardware Approaches for Cache Optimizations Cache performance.
CSE 490/590, Spring 2011 CSE 490/590 Computer Architecture Cache II Steve Ko Computer Sciences and Engineering University at Buffalo.
CSE 490/590, Spring 2011 CSE 490/590 Computer Architecture Virtual Memory I Steve Ko Computer Sciences and Engineering University at Buffalo.
CSE 490/590, Spring 2011 CSE 490/590 Computer Architecture Cache IV Steve Ko Computer Sciences and Engineering University at Buffalo.
February 9, 2010CS152, Spring 2010 CS 152 Computer Architecture and Engineering Lecture 7 - Memory Hierarchy-II Krste Asanovic Electrical Engineering and.
1 COMP 206: Computer Architecture and Implementation Montek Singh Wed., Nov. 10, 2003 Topic: Caches (contd.)
1 Lecture 14: Cache Innovations and DRAM Today: cache access basics and innovations, DRAM (Sections )
CS 152 Computer Architecture and Engineering Lecture 15 - Advanced Superscalars Krste Asanovic Electrical Engineering and Computer Sciences University.
1 Chapter Seven Large and Fast: Exploiting Memory Hierarchy.
CS252/Culler Lec 4.1 1/31/02 CS203A Graduate Computer Architecture Lecture 14 Cache Design Taken from Prof. David Culler’s notes.
331 Lec20.1Fall :332:331 Computer Architecture and Assembly Language Fall 2003 Week 13 Basics of Cache [Adapted from Dave Patterson’s UCB CS152.
February 9, 2011CS152, Spring 2011 CS 152 Computer Architecture and Engineering Lecture 7 - Memory Hierarchy-II Krste Asanovic Electrical Engineering and.
EENG449b/Savvides Lec /13/04 April 13, 2004 Prof. Andreas Savvides Spring EENG 449bG/CPSC 439bG Computer.
CS 152 Computer Architecture and Engineering Lecture 7 - Memory Hierarchy-II Krste Asanovic Electrical Engineering and Computer Sciences University of.
CS 152 Computer Architecture and Engineering Lecture 7 - Memory Hierarchy-II Krste Asanovic Electrical Engineering and Computer Sciences University of.
1 COMP 206: Computer Architecture and Implementation Montek Singh Wed, Nov 9, 2005 Topic: Caches (contd.)
331 Lec20.1Spring :332:331 Computer Architecture and Assembly Language Spring 2005 Week 13 Basics of Cache [Adapted from Dave Patterson’s UCB CS152.
1 COMP 206: Computer Architecture and Implementation Montek Singh Wed., Oct. 30, 2002 Topic: Caches (contd.)
February 9, 2010CS152, Spring 2010 CS 152 Computer Architecture and Engineering Lecture 7 - Memory Hierarchy-II Krste Asanovic Electrical Engineering and.
1  2004 Morgan Kaufmann Publishers Chapter Seven.
1 SRAM: –value is stored on a pair of inverting gates –very fast but takes up more space than DRAM (4 to 6 transistors) DRAM: –value is stored as a charge.
Reducing Cache Misses 5.1 Introduction 5.2 The ABCs of Caches 5.3 Reducing Cache Misses 5.4 Reducing Cache Miss Penalty 5.5 Reducing Hit Time 5.6 Main.
CS 152 Computer Architecture and Engineering Lecture 7 - Memory Hierarchy-II Krste Asanovic Electrical Engineering and Computer Sciences University of.
February 11, 2010CS152, Spring 2010 CS 152 Computer Architecture and Engineering Lecture 8 - Memory Hierarchy-III Krste Asanovic Electrical Engineering.
Systems I Locality and Caching
Caches Hakim Weatherspoon CS 3410, Spring 2012 Computer Science Cornell University See P&H 5.1, 5.2 (except writes)
© Krste Asanovic, 2014CS252, Spring 2014, Lecture 10 CS252 Graduate Computer Architecture Spring 2014 Lecture 10: Memory Krste Asanovic
ECE 252 / CPS 220 Advanced Computer Architecture I Lecture 13 Memory – Part 2 Benjamin Lee Electrical and Computer Engineering Duke University
Lecture 15: Virtual Memory EEN 312: Processors: Hardware, Software, and Interfacing Department of Electrical and Computer Engineering Spring 2014, Dr.
February 9, 2012CS152, Spring 2012 CS 152 Computer Architecture and Engineering Lecture 7 - Memory Hierarchy-II Krste Asanovic Electrical Engineering and.
Caches Where is a block placed in a cache? –Three possible answers  three different types AnywhereFully associativeOnly into one block Direct mappedInto.
Spring 2003CSE P5481 Advanced Caching Techniques Approaches to improving memory system performance eliminate memory operations decrease the number of misses.
Computer Organization CS224 Fall 2012 Lessons 45 & 46.
1 Chapter Seven. 2 Users want large and fast memories! SRAM access times are ns at cost of $100 to $250 per Mbyte. DRAM access times are ns.
Chapter 5 Memory III CSE 820. Michigan State University Computer Science and Engineering Miss Rate Reduction (cont’d)
M E M O R Y. Computer Performance It depends in large measure on the interface between processor and memory. CPI (or IPC) is affected CPI = Cycles per.
Nov. 15, 2000Systems Architecture II1 Machine Organization (CS 570) Lecture 8: Memory Hierarchy Design * Jeremy R. Johnson Wed. Nov. 15, 2000 *This lecture.
CS.305 Computer Architecture Memory: Caches Adapted from Computer Organization and Design, Patterson & Hennessy, © 2005, and from slides kindly made available.
1 Chapter Seven CACHE MEMORY AND VIRTUAL MEMORY. 2 SRAM: –value is stored on a pair of inverting gates –very fast but takes up more space than DRAM (4.
1 Chapter Seven. 2 Users want large and fast memories! SRAM access times are ns at cost of $100 to $250 per Mbyte. DRAM access times are ns.
1  1998 Morgan Kaufmann Publishers Chapter Seven.
1  2004 Morgan Kaufmann Publishers Locality A principle that makes having a memory hierarchy a good idea If an item is referenced, temporal locality:
Cache (Memory) Performance Optimization. Average memory access time = Hit time + Miss rate x Miss penalty To improve performance: reduce the miss rate.
1 Chapter Seven. 2 SRAM: –value is stored on a pair of inverting gates –very fast but takes up more space than DRAM (4 to 6 transistors) DRAM: –value.
High Performance Computing1 High Performance Computing (CS 680) Lecture 2a: Overview of High Performance Processors * Jeremy R. Johnson *This lecture was.
Microprocessor Microarchitecture Memory Hierarchy Optimization Lynn Choi Dept. Of Computer and Electronics Engineering.
Memory Hierarchy— Five Ways to Reduce Miss Penalty.
CS203 – Advanced Computer Architecture Cache. Memory Hierarchy Design Memory hierarchy design becomes more crucial with recent multi-core processors:
Computer Organization & Design 计算机组成与设计 Weidong Wang ( 王维东 ) College of Information Science & Electronic Engineering 信息与通信工程研究所 Zhejiang.
Chapter 5 Memory Hierarchy Design. 2 Many Levels in Memory Hierarchy Pipeline registers Register file 1st-level cache (on-chip) 2nd-level cache (on same.
CENG709 Computer Architecture and Operating Systems Lecture 7 - Memory Hierarchy-II Murat Manguoglu Department of Computer Engineering Middle East Technical.
CSC 4250 Computer Architectures
The University of Adelaide, School of Computer Science
5.2 Eleven Advanced Optimizations of Cache Performance
Lecture: Cache Hierarchies
Krste Asanovic Electrical Engineering and Computer Sciences
Krste Asanovic Electrical Engineering and Computer Sciences
Electrical and Computer Engineering
Lecture: Cache Innovations, Virtual Memory
Siddhartha Chatterjee
Lecture 15: Memory Design
Summary 3 Cs: Compulsory, Capacity, Conflict Misses Reducing Miss Rate
CS 3410, Spring 2014 Computer Science Cornell University
CS 152 Computer Architecture and Engineering CS252 Graduate Computer Architecture Lecture 6 – Memory II Krste Asanovic Electrical Engineering and Computer.
CS 152 Computer Architecture and Engineering CS252 Graduate Computer Architecture Lecture 7 – Memory III Krste Asanovic Electrical Engineering and Computer.
Cache Performance Improvements
Lecture 14: Cache Performance
Presentation transcript:

CS 152 Computer Architecture and Engineering Lecture 8 - Memory Hierarchy-III Krste Asanovic Electrical Engineering and Computer Sciences University of California at Berkeley

2/19/2009CS152-Spring’09 2 Last time in Lecture 7 3 C’s of cache misses: – compulsory, capacity, conflict Average memory access time = hit time + miss rate * miss penalty To improve performance, reduce: –hit time –miss rate –and/or miss penalty Primary cache parameters: –Total cache capacity –Cache line size –Associativity

2/19/2009CS152-Spring’09 3 Multilevel Caches A memory cannot be large and fast Increasing sizes of cache at each level CPU L1$ L2$ DRAM Local miss rate = misses in cache / accesses to cache Global miss rate = misses in cache / CPU memory accesses Misses per instruction = misses in cache / number of instructions

2/19/2009CS152-Spring’09 4 A Typical Memory Hierarchy c.2008 L1 Data Cache L1 Instruction Cache Unified L2 Cache RF Memory Multiported register file (part of CPU) Split instruction & data primary caches (on-chip SRAM) Multiple interleaved memory banks (off-chip DRAM) Large unified secondary cache (on-chip SRAM) CPU

2/19/2009CS152-Spring’09 5 Presence of L2 influences L1 design Use smaller L1 if there is also L2 –Trade increased L1 miss rate for reduced L1 hit time and reduced L1 miss penalty –Reduces average access energy Use simpler write-through L1 with on-chip L2 –Write-back L2 cache absorbs write traffic, doesn’t go off-chip –At most one L1 miss request per L1 access (no dirty victim write back) simplifies pipeline control –Simplifies coherence issues –Simplifies error recovery in L1 (can use just parity bits in L1 and reload from L2 when parity error detected on L1 read)

2/19/2009CS152-Spring’09 6 Inclusion Policy Inclusive multilevel cache: –Inner cache holds copies of data in outer cache –External access need only check outer cache –Most common case Exclusive multilevel caches: –Inner cache may hold data not in outer cache –Swap lines between inner/outer caches on miss –Used in AMD Athlon with 64KB primary and 256KB secondary cache Why choose one type or the other?

2/19/2009CS152-Spring’09 7 Reducing penalty of associativity Associativity reduces conflict misses, but requires expensive (area, energy, delay) multi-way tag search Two optimizations to reduce cost of associativity –Victim caches –Way prediction

2/19/2009CS152-Spring’09 8 Victim Caches (Jouppi 1990) Unified L2 Cache RF CPU Evicted data from L1 Evicted data From VC where ? Hit data from VC (miss in L1) Victim cache is a small associative back up cache, added to a direct mapped cache, which holds recently evicted lines First look up in direct mapped cache If miss, look in victim cache If hit in victim cache, swap hit line with line now evicted from L1 If miss in victim cache, L1 victim -> VC, VC victim->? Fast hit time of direct mapped but with reduced conflict misses (HP 7200) Victim Cache Fully Assoc. 4 blocks L1 Data Cache Direct Map.

2/19/2009CS152-Spring’09 9 Way-Predicting Instruction Cache (Alpha like) PC addr inst Primary Instruction Cache 0x4 Add Sequential Way Branch Target Way way Jump target Jump control

2/19/2009CS152-Spring’09 10 Way Predicting Caches ( MIPS R10000 off-chip L2 cache ) Use processor address to index into way prediction table Look in predicted way at given index, then: HITMISS Return copy of data from cache Look in other way Read block of data from next level of cache MISS SLOW HIT (change entry in prediction table)

2/19/2009CS152-Spring’09 11 Reduce Miss Penalty of Long Blocks: Early Restart and Critical Word First Don’t wait for full block before restarting CPU Early restart—As soon as the requested word of the block arrives, send it to the CPU and let the CPU continue execution –Spatial locality  tend to want next sequential word, so not clear size of benefit of just early restart Critical Word First—Request the missed word first from memory and send it to the CPU as soon as it arrives; let the CPU continue execution while filling the rest of the words in the block –Long blocks more popular today  Critical Word 1 st Widely used block

2/19/2009CS152-Spring’09 12 Increasing Cache Bandwidth with Non-Blocking Caches Non-blocking cache or lockup-free cache allow data cache to continue to supply cache hits during a miss –requires Full/Empty bits on registers or out-of-order execution “hit under miss” reduces the effective miss penalty by working during miss vs. ignoring CPU requests “hit under multiple miss” or “miss under miss” may further lower the effective miss penalty by overlapping multiple misses –Significantly increases the complexity of the cache controller as there can be multiple outstanding memory accesses, and can get miss to line with outstanding miss (secondary miss) –Requires pipelined or banked memory system (otherwise cannot support multiple misses) –Pentium Pro allows 4 outstanding memory misses –(Cray X1E vector supercomputer allows 2,048 outstanding memory misses)

2/19/2009CS152-Spring’09 13 Value of Hit Under Miss for SPEC (old data) FP programs on average: AMAT= > > > 0.26 Int programs on average: AMAT= > > > KB Data Cache, Direct Mapped, 32B block, 16 cycle miss, SPEC 92 Integer Floating Point “Hit under n Misses” 0->1 1->2 2->64 Base

2/19/2009CS152-Spring’09 14 CS152 Administrivia Last three lectures (L6-L8) on memory hierarchy form material for Quiz 2 (Tuesday March 3)

2/19/2009CS152-Spring’09 15 Prefetching Speculate on future instruction and data accesses and fetch them into cache(s) –Instruction accesses easier to predict than data accesses Varieties of prefetching –Hardware prefetching –Software prefetching –Mixed schemes What types of misses does prefetching affect?

2/19/2009CS152-Spring’09 16 Issues in Prefetching Usefulness – should produce hits Timeliness – not late and not too early Cache and bandwidth pollution L1 Data L1 Instruction Unified L2 Cache RF CPU Prefetched data

2/19/2009CS152-Spring’09 17 Hardware Instruction Prefetching Instruction prefetch in Alpha AXP –Fetch two blocks on a miss; the requested block (i) and the next consecutive block (i+1) –Requested block placed in cache, and next block in instruction stream buffer –If miss in cache but hit in stream buffer, move stream buffer block into cache and prefetch next block (i+2) L1 Instruction Unified L2 Cache RF CPU Stream Buffer Prefetched instruction block Req block Req block

2/19/2009CS152-Spring’09 18 Hardware Data Prefetching Prefetch-on-miss: –Prefetch b + 1 upon miss on b One Block Lookahead (OBL) scheme –Initiate prefetch for block b + 1 when block b is accessed –Why is this different from doubling block size? –Can extend to N-block lookahead Strided prefetch –If observe sequence of accesses to block b, b+N, b+2N, then prefetch b+3N etc. Example: IBM Power 5 [2003] supports eight independent streams of strided prefetch per processor, prefetching 12 lines ahead of current access

2/19/2009CS152-Spring’09 19 Software Prefetching for(i=0; i < N; i++) { prefetch( &a[i + 1] ); prefetch( &b[i + 1] ); SUM = SUM + a[i] * b[i]; } What property do we require of the cache for prefetching to work ?

2/19/2009CS152-Spring’09 20 Software Prefetching Issues Timing is the biggest issue, not predictability –If you prefetch very close to when the data is required, you might be too late –Prefetch too early, cause pollution –Estimate how long it will take for the data to come into L1, so we can set P appropriately – Why is this hard to do? for(i=0; i < N; i++) { prefetch( &a[i + P ] ); prefetch( &b[i + P ] ); SUM = SUM + a[i] * b[i]; } Must consider cost of prefetch instructions

2/19/2009CS152-Spring’09 21 Compiler Optimizations Restructuring code affects the data block access sequence –Group data accesses together to improve spatial locality –Re-order data accesses to improve temporal locality Prevent data from entering the cache –Useful for variables that will only be accessed once before being replaced –Needs mechanism for software to tell hardware not to cache data ( “ no-allocate ” instruction hints or page table bits) Kill data that will never be used again –Streaming data exploits spatial locality but not temporal locality –Replace into dead cache locations

2/19/2009CS152-Spring’09 22 Loop Interchange for(j=0; j < N; j++) { for(i=0; i < M; i++) { x[i][j] = 2 * x[i][j]; } } for(i=0; i < M; i++) { for(j=0; j < N; j++) { x[i][j] = 2 * x[i][j]; } } What type of locality does this improve?

2/19/2009CS152-Spring’09 23 Loop Fusion for(i=0; i < N; i++) a[i] = b[i] * c[i]; for(i=0; i < N; i++) d[i] = a[i] * c[i]; for(i=0; i < N; i++) { a[i] = b[i] * c[i]; d[i] = a[i] * c[i]; } What type of locality does this improve?

2/19/2009CS152-Spring’09 24 for(i=0; i < N; i++) for(j=0; j < N; j++) { r = 0; for(k=0; k < N; k++) r = r + y[i][k] * z[k][j]; x[i][j] = r; } Blocking xyzjkj iik Not touchedOld accessNew access

2/19/2009CS152-Spring’09 25 for(jj=0; jj < N; jj=jj+B) for(kk=0; kk < N; kk=kk+B) for(i=0; i < N; i++) for(j=jj; j < min(jj+B,N); j++) { r = 0; for(k=kk; k < min(kk+B,N); k++) r = r + y[i][k] * z[k][j]; x[i][j] = x[i][j] + r; } Blocking xyzjkj iik What type of locality does this improve?

2/19/2009CS152-Spring’09 26 Workstation Memory System (Apple PowerMac G5, 2003) Dual 2GHz processors, each has: 64KB I-cache, direct mapped 32KB D-cache, 2-way 512KB L2 unified cache, 8-way All 128B lines Up to 8GB DDR SDRAM, 400MHz, 128-bit bus, 6.4GB/s 1GHz, 2x32-bit bus, 16GB/s North Bridge Chip AGP Graphics Card, 533MHz, 32-bit bus, 2.1GB/s PCI-X Expansion, 133MHz, 64-bit bus, 1 GB/s

2/19/2009CS152-Spring’09 27 Acknowledgements These slides contain material developed and copyright by: –Arvind (MIT) –Krste Asanovic (MIT/UCB) –Joel Emer (Intel/MIT) –James Hoe (CMU) –John Kubiatowicz (UCB) –David Patterson (UCB) MIT material derived from course UCB material derived from course CS252