University of Utah 1 The Effect of Interconnect Design on the Performance of Large L2 Caches Naveen Muralimanohar Rajeev Balasubramonian

University of Utah2 Motivation: Large Caches  Future processors will have large on-chip caches  Intel Montecito has 24MB on-chip cache  Wire delay dominates in large caches  Conventional design can lead to very high hit time (CACTI access time for 24 MB cache is 90 5GHz, 65nm Tech)  Careful network choices  Improve access time  Open room for several other optimizations  Reduces power significantly

University of Utah3 Effect of L2 Hit Time 8-issue, out-of-order processor (L2-hit time cycles) Avg = 17%

University of Utah4 Cache Design Input address Decoder Wordline Bitlines Tag array Data array Column muxes Sense Amps Comparators Output driver Valid output? Mux drivers Data output Output driver

University of Utah5 Existing Model - CACTI Decoder delay Wordline & bitline delay Cache model with 4 sub-arraysCache model with 16 sub-arrays

University of Utah6 Shortcomings  CACTI Suboptimal for large cache size Access delay is equal to the delay of slowest sub-array  Very high hit time for large caches Employs a separate bus for each cache bank for multi-banked caches

University of Utah7 Non-Uniform Cache Access (NUCA)  Large cache is broken into a number of small banks  Employs on-chip network for communication  Access delay  (distance between bank and cache controller) CPU & L1 Cache banks

University of Utah8 Shortcomings  NUCA Banks are sized such that the link latency is one cycle (Kim et al. ASPLOS 02) Increased routing complexity Dissipates more power

University of Utah9 Extension to CACTI  On-chip network Wire model is done using ITRS 2005 parameters Grid network No. of rows = No. of columns (or ½ the no. of columns)  Network latency vs Bank access latency tradeoff Modified the exhaustive search to include the network overhead

University of Utah10 Effect of Network Delay (32MB cache) Delay optimal point

University of Utah11 Outline Overview Cache Design Effect of Network Delay  Wire Design Space  Exploiting Heterogeneous Wires  Results

University of Utah12 Wire Characteristics  Wire Resistance and capacitance per unit length ResistanceCapacitanceBandwidth Width Spacing

University of Utah13 Design Space Exploration  Tuning wire width and spacing Base case B wires Fast but Low bandwidth L wires (Width & Spacing)   Delay  Bandwidth 

University of Utah14 Design Space Exploration  Tuning Repeater size and spacing Traditional Wires Large repeaters Optimum spacing Power Optimal Wires Smaller repeaters Increased spacing Delay Power

University of Utah15 Design Space Exploration Base case B wires 8x plane Base case W wires 4x plane Power optimized PW wires 4x plane Fast, low bandwidth L wires 8x plane Latency 1x Power 1x Area 1x Latency 1.6x Power 0.9x Area 0.5x Latency 3.2x Power 0.3x Area 0.5x Latency 0.5x Power 0.5x Area 5x

University of Utah16 Access time for different link types Bank Count Bank Access Time Avg Access time 8x-wires4x-wiresL-wires

University of Utah17 Outline Overview Cache Design Effect of Network Delay Wire Design Space  Exploiting Heterogeneous Wires  Results

University of Utah18 Cache Look-Up Total cache access time Network delay (req 6-8 bits to identify the cache Bank) Decoder, Wordline, Bitline delay (req bits of address) Comparator, output driver delay (req remaining address for tag match)  The entire access happens in a sequential manner Bank access

University of Utah19 Early Look-Up  Send partial address in L-wires  Initiate the bank lookup  Wait for the complete address  Complete the access L Early lookup (req bits of address) Tag match  We can hide 60-70% of the bank access delay

University of Utah20 Aggressive Look-Up  Send partial address bits on L-wires  Do early look-up and do partial tag match  Send all the matched blocks aggressively L Agg. lookup (req additional 8-bits of address fpr partial tag match) Tag match at cache controller Network delay reduced

University of Utah21 Aggressive Look-Up  Significant reduction in network delay (for address transfer)  Increase in traffic due to false match < 1%  Marginal increase in link overhead Additional 8-bits of L-wires compared to early lookup -Adds complexity to cache controller -Needs logic to do tag match

University of Utah22 Outline Overview Cache Design Effect of Network Delay Wire Design Space Exploiting Heterogeneous Wires  Results

University of Utah23 Experimental Setup  Simplescalar with contention modeled in detail  Single core, 8-issue out-of-order processor  32 MB, 8-way set-associative, on-chip L2 cache (SNUCA organization)  32KB I-cache and 32KB D-cache with hit latency of 3 cycles  Main memory latency 300 cycles

University of Utah24 Cache Models ModelBank Access (cycles) Bank CountNetwork LinkDescription 13512B-wiresBased on prior work 2664B-wiresCACTI-L2 3664B & L–wiresEarly Lookup 4664B & L–wiresAgg. Lookup 5664B & L–wiresUpper bound

University of Utah25 Performance Results (Global Wires) Model 2 (CACTI-L2) : Average performance improvement – 11% Performance improvement for L2 latency sensitive benchmarks – 16.3% Model 3 (Early Lookup): Average performance improvement – 14.4% Performance improvement for L2 latency sensitive benchmarks – 21.6% Model 4 (Aggressive Lookup): Average performance improvement – 17.6% Performance improvement for L2 latency sensitive benchmarks – 26.6% Model 6 (L-Network): Average performance improvement – 11.4% Performance improvement for L2 latency sensitive benchmarks – 16.2%

University of Utah26 Performance Results (4X – Wires) Wire delay constrained model Performance improvements are better Early lookup performs 5% better Aggressive model performs 28% better

University of Utah27 Future Work  Heterogeneous network in a CMP environment  Hybrid-network Employs a combination of point-to-point and bus for L- messages  Effective use of L-wires  Latency/bandwidth trade-off  Use of heterogeneous wires in DNUCA environment  Cache design focusing on power Pre-fetching (Power optimized wires) Writeback (Power optimized wires)

University of Utah28 Conclusion  Traditional design approaches for large caches is sub-optimal  Network parameters play a significant role in the performance of large caches  Modified CACTI model, that includes network overhead performs 16.3% better compared to previous models  Heterogeneous network has potential to further improve the performance Early lookup – 21.6% Aggressive lookup – 26.6%