Non-Uniform Power Access in Large Caches with Low-Swing Wires Aniruddha N. Udipi with Naveen Muralimanohar*, Rajeev Balasubramonian University of Utah and *HP Labs

University of Utah 2 Motivation Future CMPs likely to be power-limited Growing gap between processor and main memory performance – the Bandwidth Wall –Large caches required to alleviate this problem –Nehalem already has 8MB of last-level cache These large caches contribute significantly to energy consumption –They are often the cache coherence interface in CMPs –Cache energy contribution likely to rise as core energy reduces with simpler and more efficient cores

University of Utah 3 Executive Summary H-tree identified as energy bottleneck within large cache banks Study various techniques to introduce low-swing wiring to address this bottleneck Non-Uniform Power Access to allow access to different regions of cache at different energies Architectural mechanisms to increase fraction of accesses hitting in the low-power region Significant cache energy reductions at very modest performance penalties

University of Utah 4 Outline Cache design background Technique I – Single low-swing bus Technique II – Multiple low-swing buses Technique III – Fully-pipelined low-swing bus Technique IV – Non-Uniform Power Access Technique V – Architectural mechanisms Evaluation Conclusion

NUCA design Increasing disparity in access delays to different parts of the cache Non-Uniform Cache Access –Divide large cache into multiple “banks” –On-chip network connects these banks and transfers address and data –Bank count and size of each bank determined by relative contribution of banks and network to total energy/delay –Per CACTI 6.0, even a 64MB NUCA cache likely to have large 2 or 4MB banks University of Utah 5 Interconnect Cache Core Cache Core Cache Core Cache Core

Bank design basics University of Utah 6 Input address Decoder Wordline Bitlines Tag array Data array Column muxes Sense Amps Comparators Mux drivers Data output Output driver

Bank design considerations Naïve implementation would take the form of a single array of memory cells with centralized control logic, but such a design would not scale –Wordlines (area considerations) and bitlines (differential signaling) cannot be repeated – delay increase with cache size –Cache bandwidth is a function of cycle time – single array would have small bandwidth Performance limited by wordline/bitline length –Divide into multiple segments called “subarrays” –Subarrays connected by an internal network University of Utah 7

Bank organization Bank organization determined by NDWL,NDBL Fewer subarrays gives increased area efficiency, but larger delay due to longer wordlines/bitlines University of Utah 8 NDWL = 4 NDBL = 4 H-TREE SUBARRAY Interconnect Cache Core Cache Core Cache Core Cache Core

Bank Energy Consumption H-tree is clearly the dominant component of energy consumption University of Utah 9

Low-swing wires High power dissipation in global wires due to full swing requirement imposed by repeaters Use low-voltage swing differential signaling –Two wires per signal –Voltage swing as low as 100mV –Approx. 10X energy savings compared to full swing wires –Increased delay, cannot be used over long distances –Non-trivial pipelining costs What is the best way to use low-swing wires to build the H-tree? University of Utah 10

University of Utah 11 Outline Cache design background Technique I – Single low-swing bus Technique II – Multiple low-swing buses Technique III – Fully-pipelined low-swing bus Technique IV – Non-Uniform Power Access Technique V – Architectural mechanisms Evaluation Conclusion

Single low-swing bus Simplest solution, simply build entire H-tree with low- swing wires Best energy savings Significant performance drops –Cycle time becomes equal to access time –Increased contention Not worth considering unless energy is considerably more important than performance University of Utah 12

University of Utah 13 Outline Cache design background Technique I – Single low-swing bus Technique II – Multiple low-swing buses Technique III – Fully-pipelined low-swing bus Technique IV – Non-Uniform Power Access Technique V – Architectural mechanisms Evaluation Conclusion

Multiple low-swing buses Spread contention around Fast vertical bus, tristate buffers at intersections Energy overhead modeled accurately University of Utah 14 LOW-SWING BUS TRI-STATE BUFFERS

University of Utah 15 Outline Cache design background Technique I – Single low-swing bus Technique II – Multiple low-swing buses Technique III – Fully-pipelined low-swing bus Technique IV – Non-Uniform Power Access Technique V – Architectural mechanisms Evaluation Conclusion

Fully-pipelined low-swing bus Pipelining low-swing wires is non-trivial Differential transmitter and receiver required at every pipeline stage Amortized over 1mm, every transceiver is a 58% energy overhead Performance improves compared to non-pipelined low- swing University of Utah 16

University of Utah 17 Outline Cache design background Technique I – Single low-swing bus Technique II – Multiple low-swing buses Technique III – Fully-pipelined low-swing bus Technique IV – Non-Uniform Power Access Technique V – Architectural mechanisms Evaluation Conclusion

Non-Uniform Power Access University of Utah 18 LOW-SWING H-TREE TRUNK DEFAULT FULL- SWING H-TREE LOW-POWER REGION HIGH-POWER REGION

Non-Uniform Power Access Introduction of the low-swing trunk does not affect basic H-tree design significantly Limited low-swing length –Access time same as that for the default H-tree –New bus transparent to processor Energy savings proportional to fraction of rows accessible via the low-swing bus –Only two central rows - 1/16 th in our case (NDBL = 32) –Architectural mechanisms required to increase this fraction University of Utah 19

University of Utah 20 Outline Cache design background Technique I – Single low-swing bus Technique II – Multiple low-swing buses Technique III – Fully-pipelined low-swing bus Technique IV – Non-Uniform Power Access Technique V – Architectural mechanisms Evaluation Conclusion

Exploiting Non-Uniform Power Access Increase fraction of accesses served by the “low- power region” Assign a fraction of the ways of the set to the “low- power region (LP)” and the rest of the ways to the “high-power region (HP)” On every access, check all tags in parallel, if it hits in the LP region, it is a low-power access If not, bring the line into the low-power region at this point – the next use will then likely be a low-power access University of Utah 21

Swap scheme Bring block into low-power region on first-touch The block currently in LRU position in that set is swapped out into the high-power region –Most recently used (MRU) ways of every set are in the LP region Every low-power fetch incurs a swap which costs two low-power and two high-power accesses For Swap to consume less energy than baseline with N accesses –N * H > 2 * H + (N+1) * L –N > 2.5 University of Utah 22

Duplicate scheme Bring block into low-power and high-power region on first touch Block currently in LRU position in low-power region is –Simply dropped if clean – better than Swap –Written back to high-power region if dirty – same as Swap Every L2 miss results in one additional HP access initially Forming equations similar to Swap –N clean > 1.16 –N dirty > 2.6 University of Utah 23

Dynamic Reconfiguration Good energy savings if a modestly high hit-rate in the low-power region Below a certain threshold, extra energy required to move blocks between LP and HP region overshadows savings Track average reuse count and turn-off architectural mechanisms in bad phases, operate like default cache –Single five bit saturating counter for entire cache –Increment counter on hit in LP region, decrement on miss University of Utah 24

Comparison to L2/L3 or Filter Cache Data placement and mapping schemes do bear resemblance to L2/L3 hierarchy or filter cache –our approach is orthogonal to the hierarchy and can continue to be used for the largest last-level cache –need for interconnects between multiple physical cache structures eliminated –Non-uniform access model 25% more efficient than a filter cache model with similar capacities University of Utah 25

University of Utah 26 Outline Cache design background Technique I – Single low-swing bus Technique II – Multiple low-swing buses Technique III – Fully-pipelined low-swing bus Technique IV – Non-Uniform Power Access Technique V – Architectural mechanisms Evaluation Conclusion

University of Utah 27 Methodology SimpleScalar 3.0 OOO-simulator CACTI 6.0 for cache energy/delay computation 32nm process, 5GHz clock 32K each I- and D-L1, 2-way Unified 4MB L2 cache, 16-way 300 cycle main memory latency SPEC2k benchmark suite

Low-swing design points - Energy University of Utah 28

Low-swing design points - IPC University of Utah 29

Low-swing design points Clearly a trade-off between energy savings and performance drops ED 2 metric –Non-uniform model gives 5% improvement over baseline –Pipelined low-swing model is next best, with a 3% improvement over baseline –These are the two most compelling design points University of Utah 30

Architectural mechanisms University of Utah 31

Dynamic reconfiguration University of Utah 32

Sensitivity to cache size University of Utah 33

University of Utah 34 Outline Cache design background Technique I – Single low-swing bus Technique II – Multiple low-swing buses Technique III – Fully-pipelined low-swing bus Technique IV – Non-Uniform Power Access Technique V – Architectural mechanisms Evaluation Conclusion

University of Utah 35 Related Work Low-swing wires –“Smart memories” project, CACTI 6.0 Cache access energy –Drowsy cache, gated-ground cache, L0 instruction cache, non-uniformity in number of ways per set Ours is the first work to optimize the internal structure of the cache, and propose non-uniform power access within a cache bank

Key Contributions Study of the internal organization of large cache banks, identification of bottleneck Exploration the design space of low-swing wiring within large caches Introduction of the notion of Non-Uniform Power Access –Definition of the architectural mechanisms required to maximize the energy-saving potential of low-swing wires University of Utah 36

University of Utah 37 Thank you.. Questions?