Automatic Tuning of Two-Level Caches to Embedded Applications Ann Gordon-Ross and Frank Vahid* Department of Computer Science and Engineering University of California, Riverside *Also with the Center for Embedded Computer Systems, UC Irvine Nikil Dutt Center for Embedded Computer Systems School for Information and Computer Science University of California, Irvine This work was supported by the U.S. National Science Foundation, and by the Semiconductor Research Corporation

Introduction Memory access: 50% of embedded processor’s system power Caches are power hungry ARM920T(Segars 01) M*CORE (Lee/Moyer/Arends 99) Thus, the cache is a good candidate for optimizations Main Mem L2 Cache L1 Cache Processor 53%

Motivation Size Excess fetch and static energy if too large Tuning cache parameters to an application can save energy: 60% on average Balasubramonian’00, Zhang’03 Each application has different cache requirements One predetermined cache configuration can’t be best for all applications Size Excess fetch and static energy if too large Excess thrashing energy if too small L1 Cache

Motivation Line size Excess fetch energy if line size too large Tuning cache parameters to an application can save energy: 60% on average Balasubramonian’00, Zhang’03 Each application has different cache requirements One predetermined cache configuration can’t be best for all applications Line size Excess fetch energy if line size too large Excess stall energy if line size too small L1 Cache

} { Motivation Cache associativity Tuning cache parameters to an application can save energy: 60% on average Balasubramonian’00, Zhang’03 Each application has different cache requirements One predetermined cache configuration can’t be best for all applications { } Cache associativity Excess fetch energy per access if too high Excess miss energy if too low L1 Cache

Motivation By tuning these parameters, the cache can be customized to a particular application Choose lowest energy configuration Microprocessor L1 Cache Tuning Energy L2 Cache Possible Cache Configurations Main Memory

Related Work Tuning Configurable caches Configurable cache tuning Soft cores (ARM, MIPS, Tensillica, etc.) Even for hard processors (Motorola M*Core - Malik ISLPED’00; Albonesi MICRO’00; Zhang ISCA’03) Configurable cache tuning Mostly manually in practice Sub-optimal, time-consuming L1 automated methods Platune (Givargis TCAD’02, Palesi CODES’02) Zhang RSP’03 Two-level caches becoming popular More transistors on-chip available Bigger gap between on-chip and off-chip accesses Need automated tuning for L1+L2 Microprocessor Main Memory L1 Cache L2 Cache Tuning

Challenge for Two-Level Cache Tuning One level: 10s of configurations Two levels: 100s/1000s of configurations Need efficient heuristic Especially if used with simulation-based search Level 1 Level 2 - Total size - Line size - Associativity - Total size - Line size - Associativity 2500 configs * Say 50 configs. 50 configs.

Two-Level Cache Tuning Goal Develop fast, good-quality heuristic for tuning two-level caches to embedded applications for reduced energy consumption Presently focus on separate I and D cache in both levels Tune Instruction Cache Hierarchy Level 1 Caches Level 2 Caches I-cache I-cache Main Memory Microprocessor Tune Data Cache Hierarchy D-cache D-cache

Configurable Cache Architecture Our target configurable cache architecture is based on Zhang/Vahid/Najjar’s “Highly-Configurable Cache Architecture for Embedded Systems,” ISCA 2003 Way shutdown and way concatenation can be combined to offer a direct-mapped 4 KB cache 4 KB Level One Cache Way shutdown offers a 2-way 4 KB cache and a direct-mapped 2 KB cache 2 KB Level One Cache Way shutdown offers a 2-way 4 KB cache or a direct-mapped 2 KB cache 2 KB Level One Cache Base Level One Cache Way concatenation offers a 2-way or a directed-mapped variation 8 KB Way concatenation offers a 2-way or a directed-mapped variation 4 KB 2KB 2KB 2KB 2KB 8 KB cache consisting of 4 2KB banks that can operate as 4 ways

Configuration Space Cache parameters 432 possible configurations Size - L1 cache: 2, 4, and 8 KBytes. L2 cache: 16, 32, and 64 KBytes Line size (L1 or L2) - 16, 32, and 64 Bytes 16 byte physical base line size Associativity (L1 or L2) - Direct-mapped, 2-way, and 4-way 432 possible configurations For two levels, with separate I and D

Experimental Environment MediaBench EEMBC Chosen cache configuration Hit and miss ratios for each configuration SimpleScalar Exhaustive search Took days. For comparison purposes Cache exploration heuristic Cache energy - Cacti Main memory energy - Samsung memory CPU stall energy - 0.18 micron MIPS uP

First Heuristic: Tune Levels One-at-a-Time Tune L1, then L2 Initial L2: 64 KByte, 4-way, 64 byte line size For best L1 found, tune L2 cache Tuned each cache using Zhang’s heuristic for one-level cache tuning (RSP’03) Microprocessor L1 Cache L2 Cache Main Memory

First Heuristic: Tune Levels One-at-a-Time Zhang’s heuristic: Search parameters in order of importance (RSP’03) First search size Increase size to 4 KB. Level One Cache Finally, search associativity For the lowest energy line size, increase the associativity to 2 Level One Cache Next search line size If the increase in line size yields a decrease in energy, increase the line size to 64 Bytes Level One Cache First search size Begin with a 2 KByte, direct-mapped cache with a 16 Byte line size Level One Cache Next search line size For the lowest energy cache size, increase the line size to 32 Bytes Level One Cache Finally, search associativity If increasing the associativity yields a decrease in energy, increase the associativity to 4 Level One Cache First search size If the size increase yields energy improvements, increase the cache size to 8KB. Level One Cache

Results of First Heuristic Base cache configuration Level 1 - 8 KByte, 4-way, 32 byte line Level 2 - 64 KByte, 4-way, 64 byte line

First Heuristic Did not find optimal in most cases Sometimes 200% or 300% worse The two levels should not be explored separately Too much interdependence among L1 and L2 cache parameters E.g., high L1 associativity decreases misses and thus reduces need for large L2 Dozens of other such interdependencies

Improved Heuristic – Basic Interlacing To more fully explore the dependencies between the two levels, we interlaced the exploration of the level one and level two caches Determine the best size of level one cache Determine the best size of level two cache L1 Cache L2 Cache

Improved Heuristic – Basic Interlacing To more fully explore the dependencies between the two levels, we interlaced the exploration of the level one and level two caches Determine the best line size of level two cache Determine the best line size of level one cache L1 Cache L2 Cache

Improved Heuristic – Basic Interlacing To more fully explore the dependencies between the two levels, we interlaced the exploration of the level one and level two caches { } Determine the best associativity of level one cache { } Determine the best associativity of level two cache L1 Cache L2 Cache Basic interlacing performed better than the initial heuristic but there was still much room for improvement

Final Heuristic: Interlaced with Local Search Performed well, but some cases sub-optimal Manually examined those cases Determined small local search needed Final heuristic called: TCaT - The Two Level Cache Tuner 16KB 16KB 16KB However, the application may require the increased associativity. During the associativity search step, the cache size is allowed to increase so that larger associativities may be explored. Because of the bank arrangements, if a 16KB cache is determined to be the best size, the only associativity option is direct-mapped

TCaT Results: Energy Energy consumption (normalized to the base cache configuration) 53% energy savings in cache/memory access sub-system vs. base cache

TCaT Results: Performance Execution time for the TCaT cache configuration and the optimal cache configuration (normalized to the execution time of the benchmark running with the base cache configuration) TCaT finds near-optimal configuration, nearly 30% improvement over base cache

TCaT Exploration Time Improvements Searches only 28 of 432 possible configurations 6% of space Simulation-based approach 500 MHz Sparc 50 hrs vs. 3 hrs Hardware-based approach 434 sec vs. 28 sec

TCaT in Presence of Hw/Sw Partitioning Hardware/software partitioning may become common in SOC platforms On-chip FPGA Program kernels moved to FPGA Greatly reduces temporal and spatial locality of program Does TCaT still work well on programs with very low locality?

TCaT With Hardware/Software Partitioning Energy consumption (normalized to the base cache configuration) 55% energy savings in cache/memory access sub-system vs. base cache

Conclusions TCaT is an effective heuristic for two-level cache tuning Prunes 94% of search space for a given two-level configurable cache architecture Near-optimal performance results, 30% improvement vs. base cache Near-optimal energy results, 53% improvement vs. base cache Robust in presence of hw/sw partitioning Future work More cache parameters, unified 2L cache Even larger search space Dynamic in-system tuning Must avoid cache flushes