1. Automatic Tuning of Two-Level Caches to Embedded Applications
Ann Gordon-Ross and Frank Vahid*
Department of Computer Science and Engineering
University of California, Riverside
*Also with the Center for Embedded Computer Systems, UC Irvine
Nikil Dutt
Center for Embedded Computer Systems
School for Information and Computer Science
University of California, Irvine
This work was supported by the U.S. National Science Foundation, and by the Semiconductor Research Corporation

2. Introduction Memory access: 50% of embedded processor’s system power
Caches are power hungry
ARM920T(Segars 01)
M*CORE (Lee/Moyer/Arends 99)
Thus, the cache is a good candidate for optimizations
Main Mem
L2 Cache
L1 Cache
Processor
53%

3. Motivation Size Excess fetch and static energy if too large
Tuning cache parameters to an application can save energy: 60% on average
Balasubramonian’00, Zhang’03
Each application has different cache requirements
One predetermined cache configuration can’t be best for all applications
Size
Excess fetch and static energy if too large
Excess thrashing energy if too small
L1 Cache

4. Motivation Line size Excess fetch energy if line size too large
Tuning cache parameters to an application can save energy: 60% on average
Balasubramonian’00, Zhang’03
Each application has different cache requirements
One predetermined cache configuration can’t be best for all applications
Line size
Excess fetch energy if line size too large
Excess stall energy if line size too small
L1 Cache

5. } { Motivation Cache associativity
Tuning cache parameters to an application can save energy: 60% on average
Balasubramonian’00, Zhang’03
Each application has different cache requirements
One predetermined cache configuration can’t be best for all applications { }
Cache associativity
Excess fetch energy per access if too high
Excess miss energy if too low
L1 Cache

6. Motivation
By tuning these parameters, the cache can be customized to a particular application
Choose lowest energy configuration
Microprocessor
L1 Cache
Tuning
Energy
L2 Cache
Possible Cache Configurations
Main Memory

7. Related Work Tuning Configurable caches Configurable cache tuning
Soft cores (ARM, MIPS, Tensillica, etc.)
Even for hard processors (Motorola M*Core - Malik ISLPED’00; Albonesi MICRO’00; Zhang ISCA’03)
Configurable cache tuning
Mostly manually in practice
Sub-optimal, time-consuming
L1 automated methods
Platune (Givargis TCAD’02, Palesi CODES’02)
Zhang RSP’03
Two-level caches becoming popular
More transistors on-chip available
Bigger gap between on-chip and off-chip accesses
Need automated tuning for L1+L2
Microprocessor
Main Memory
L1 Cache
L2 Cache
Tuning

8. Challenge for Two-Level Cache Tuning
One level: 10s of configurations
Two levels: 100s/1000s of configurations
Need efficient heuristic
Especially if used with simulation-based search
Level 1
Level 2
- Total size
- Line size
- Associativity
- Total size
- Line size
- Associativity
2500 configs *
Say 50 configs.
50 configs.

9. Two-Level Cache Tuning Goal
Develop fast, good-quality heuristic for tuning two-level caches to embedded applications for reduced energy consumption
Presently focus on separate I and D cache in both levels
Tune Instruction Cache Hierarchy
Level 1 Caches
Level 2 Caches
I-cache
I-cache
Main Memory
Microprocessor
Tune Data Cache Hierarchy
D-cache
D-cache

10. Configurable Cache Architecture
Our target configurable cache architecture is based on Zhang/Vahid/Najjar’s “Highly-Configurable Cache Architecture for Embedded Systems,” ISCA 2003
Way shutdown and way concatenation can be combined to offer a direct-mapped 4 KB cache
4 KB
Level One Cache
Way shutdown offers a 2-way 4 KB cache and a direct-mapped 2 KB cache
2 KB
Level One Cache
Way shutdown offers a 2-way 4 KB cache or a direct-mapped 2 KB cache
2 KB
Level One Cache
Base Level One Cache
Way concatenation offers a 2-way or a directed-mapped variation
8 KB
Way concatenation offers a 2-way or a directed-mapped variation
4 KB
2KB
2KB
2KB
2KB
8 KB cache consisting of 4 2KB banks that can operate as 4 ways

11. Configuration Space Cache parameters 432 possible configurations
Size - L1 cache: 2, 4, and 8 KBytes. L2 cache: 16, 32, and 64 KBytes
Line size (L1 or L2) - 16, 32, and 64 Bytes
16 byte physical base line size
Associativity (L1 or L2) - Direct-mapped, 2-way, and 4-way
432 possible configurations
For two levels, with separate I and D

12. Experimental Environment
MediaBench
EEMBC
Chosen
cache configuration
Hit and miss ratios for each configuration
SimpleScalar
Exhaustive search
Took days.
For comparison purposes
Cache exploration heuristic
Cache energy - Cacti
Main memory energy - Samsung memory
CPU stall energy micron MIPS uP

13. First Heuristic: Tune Levels One-at-a-Time
Tune L1, then L2
Initial L2: 64 KByte, 4-way, 64 byte line size
For best L1 found, tune L2 cache
Tuned each cache using Zhang’s heuristic for one-level cache tuning (RSP’03)
Microprocessor
L1 Cache
L2 Cache
Main Memory

14. First Heuristic: Tune Levels One-at-a-Time
Zhang’s heuristic: Search parameters in order of importance (RSP’03)
First search size
Increase size to 4 KB.
Level One Cache
Finally, search associativity
For the lowest energy line size, increase the associativity to 2
Level One Cache
Next search line size
If the increase in line size yields a decrease in energy, increase the line size to 64 Bytes
Level One Cache
First search size
Begin with a 2 KByte, direct-mapped cache with a 16 Byte line size
Level One Cache
Next search line size
For the lowest energy cache size, increase the line size to 32 Bytes
Level One Cache
Finally, search associativity
If increasing the associativity yields a decrease in energy, increase the associativity to 4
Level One Cache
First search size
If the size increase yields energy improvements, increase the cache size to 8KB.
Level One Cache

15. Results of First Heuristic
Base cache configuration
Level KByte, 4-way, 32 byte line
Level KByte, 4-way, 64 byte line

16. First Heuristic Did not find optimal in most cases
Sometimes 200% or 300% worse
The two levels should not be explored separately
Too much interdependence among L1 and L2 cache parameters
E.g., high L1 associativity decreases misses and thus reduces need for large L2
Dozens of other such interdependencies

17. Improved Heuristic – Basic Interlacing
To more fully explore the dependencies between the two levels, we interlaced the exploration of the level one and level two caches
Determine the best size of level one cache
Determine the best size of level two cache
L1 Cache
L2 Cache

18. Improved Heuristic – Basic Interlacing
To more fully explore the dependencies between the two levels, we interlaced the exploration of the level one and level two caches
Determine the best line size of level two cache
Determine the best line size of level one cache
L1 Cache
L2 Cache

19. Improved Heuristic – Basic Interlacing
To more fully explore the dependencies between the two levels, we interlaced the exploration of the level one and level two caches { }
Determine the best associativity of level one cache { }
Determine the best associativity of level two cache
L1 Cache
L2 Cache
Basic interlacing performed better than the initial heuristic but there was still much room for improvement

20. Final Heuristic: Interlaced with Local Search
Performed well, but some cases sub-optimal
Manually examined those cases
Determined small local search needed
Final heuristic called: TCaT - The Two Level Cache Tuner
16KB
16KB
16KB
However, the application may require the increased associativity. During the associativity search step, the cache size is allowed to increase so that larger associativities may be explored.
Because of the bank arrangements, if a 16KB cache is determined to be the best size, the only associativity option is direct-mapped

21. TCaT Results: Energy
Energy consumption (normalized to the base cache configuration)
53% energy savings in cache/memory access sub-system vs. base cache

22. TCaT Results: Performance
Execution time for the TCaT cache configuration and the optimal cache configuration (normalized to the execution time of the benchmark running with the base cache configuration)
TCaT finds near-optimal configuration, nearly 30% improvement over base cache

23. TCaT Exploration Time Improvements
Searches only 28 of 432 possible configurations
6% of space
Simulation-based approach
500 MHz Sparc
50 hrs vs. 3 hrs
Hardware-based approach
434 sec vs. 28 sec

24. TCaT in Presence of Hw/Sw Partitioning
Hardware/software partitioning may become common in SOC platforms
On-chip FPGA
Program kernels moved to FPGA
Greatly reduces temporal and spatial locality of program
Does TCaT still work well on programs with very low locality?

25. TCaT With Hardware/Software Partitioning
Energy consumption (normalized to the base cache configuration)
55% energy savings in cache/memory access sub-system vs. base cache

26. Conclusions TCaT is an effective heuristic for two-level cache tuning
Prunes 94% of search space for a given two-level configurable cache architecture
Near-optimal performance results, 30% improvement vs. base cache
Near-optimal energy results, 53% improvement vs. base cache
Robust in presence of hw/sw partitioning
Future work
More cache parameters, unified 2L cache
Even larger search space
Dynamic in-system tuning
Must avoid cache flushes