1. Analyzing Behavior Specialized Acceleration
Tony Nowatzki
Karthikeyan Sankaralingam
My presentation today is about a new paradigm for building efficient general purpose cores, and a novel modeling technique to explore this paradigm.

2. Architectural Specialization
High-importance
Kernels/Domains
“General Purpose”
Codes
Deep Neural
Phone App.
Video Enc.
Web App
Speech Rec.
Server
Image Filter
Compiler
Dynamic Execution
Dynamic Execution
This all comes down to performing architectural specialization.
The way this normally works is that you find high-importance kernels/domains, and building specific hardware for execution of each workload. This gets very high performance/low-power by focusing the hardware resources on exactly that work that needs to be performed.
General purpose codes – don’t fit well into this paradigm
If we look at the phases of execution inside these codes, they exhibit broad behaviors that are common across applications and domains.
For example (data parallelism, control criticality, or memory regularity)
Suggests that maybe instead of doing
How to specialize?
Acc. 1
Acc. 2
Acc. 3
Acc. 4
Observation: Broad program behaviors span application domains
Too many applications
Too much code in each one
Code changes too quickly
Domain Specific
Acceleration

3. Behavior Specialization
Behaviors exploitable by specialized architectures
Exploitable behaviors cover majority of applications
Phone App.
Compiler
BSA 2
General Core
BSA 3
Web Server
Game AI
BSA 1
BSA 4
Analytics
Web App
Productivity
Graph Proc.
For behavior specialization to work, two things have to be true.
Behaviors should be exploitable by
(BSA: Behavior Specialized Accelerator)
ExoCore: Modular
Behavior Specialized Core
Dynamic Execution

4. Behavior Specialization in Practice
Domain-Specific
Accelerators
Shared Cache / NoC
ExoCore
Dynamic Program Execution
High-level
Program
Binary
Compiler
BSA 2
General Core
BSA 3
BSA 1
BSA 4
(BSA: Behavior Specialized Accelerator)
ExoCore: Modular
Behavior Specialized Core
ExoCore-Enabled
Multicore

5. Behavior Specialization Challenges
Methodological
How to study these in an effective way?
Months/years of modeling effort to days/weeks
Design/Analysis
How to compose and organize accelerators?
If we want to explore the inclusion of different possible accelerators, we need to build simulators and compilers for them, which is incredibly time consuming.
New modeling methodology that can reduce the months/years of compiler+simulator development to days or weeks.
An accelerator composition that significantly pushes speedup and energy efficiency without user involvement
× Speedup, × Energy Efficiency
without programmer involvement

6. What is a programmable accelerator?
Definition: Hardware which performs work, instead of the general purpose processor, to specialize elements of its execution.
General Purpose
Processor
Accelerator
Model all elements in the processor.
(events and dependencies)
Model specialization of execution.
(remove/transform events)

7. Background: μArch Dependence Graphs
[Fields, ISCA 2001]
Represent execution trace as a series of event nodes.
Edges between events describe μArch Dependencies.
Inorder Fetch/Dispatch/Commit F F F
Fetch F F
Fetch-Dispatch
Latency D D D
Dispatch D D
Dispatch Before
Execute E E E
Execute E E
Execution
Latency P P P
Complete P P
Complete Before
Commit C C C
Commit C C
Inst. 4
Inst. 1
Inst. 2
Inst. 3

8. Background: μArch Dependence Graphs
Data
Dependence
Branch Misprediction
Resource
Dependence F D E P C
bgz r1
r2 = ld[r3]
r4 = r2 * 2
r5 = r6 * r7 6 1
Critical Path  Execution Time F F F F D D D D E E E 1 E 6 P P P P C C C C
bgz r1
r2 = ld[r3]
r4 = r2 * 2
r5 = r6 * r7

9. Background: μArch Dependence Graphs
∑ μArch Events  Energy F F
I$ read F F D D D
RF read D
IW read E E E
ALU E
ALU
ALU
Rob
Write P P P P
ROB
Read C C C
ROB
read C
bgz r1
r2 = ld[r3]
r4 = r2 * 2
r5 = r6 * r7

10. What is a programmable accelerator?
Definition: Hardware which performs work, instead of the general purpose processor, to specialize elements its execution.
General Purpose
Processor
Accelerator
Model all elements in the processor.
(events and dependencies)
Model specialization of execution.
(remove/transform events)

11. Big Idea: Transformable Dependence Graph (TDG)
Accelerator+Core Dependence Graph D P 1 2 3 4
Graph
Transform 1 2 3
Core’s
Dep. Graph
Program IR
Model three things:
Accelerator’s compiler
Accelerator’s microarchitecture
Interaction b/t accelerator and general purpose core
Transformable Dependence Graph (TDG)

12. Example Accelerator: BERET [Micro 2011] (Bundled Execution of Recurring Traces)
Original Program
BERET Program
BERET Hardware
Comp. FU 0
ld (r0), r3
add r3, r0, r3
Comp. FU 1
assert r3 > r2
sub r2 r3 r2
assert r2 < r4 ld +
<<
BB1
ld (r0), r3
add r3, r0, r3
bge r3 r2
BB2
sub r2 r3 r2
bge r2 r4
BB3
add r2 r3 r2 T F 2%
98%
~0%
Configuration
Compound FU 0
Control
Compound FU 1
Simple accel – BERET
Goal -- executes program phases with hot-loop traces with very power efficient hardware.
Cfus connected to register file
To map a program onto beret, take hot loop trace, assign instructions to the CFU,
Compound FU 2
Reg. File
assert + ×
Compound FU 3
Hot Loop Trace

13. Control Mispeculation
Original TDG F D E P C F F F F F F F F F F F F F F F F D D D D D D D D D D D D D D D D ld + br - br ld + br - br ld + br + br P P P P P P P P P P P P P P P P C C C C C C C C C C C C C C C C
Iter 1
Iter 2
Iter 3
General Core
BB1
ld (r0), r3
add r3, r0, r3
bge r3 r2
BB2
sub r2 r3 r2
bge r2 r4
BB3
add r2 r3 r2 T F 2%
98%
Control Mispeculation
Program IR

14. BERET TDG TransformationP C F F F F F F F F F F F F F F F F D D D D D D D D D D D D D D D D ld + br - br ld + br - br ld + br + br P P P P P P P P P P P P P P P P C C C C C C C C C C C C C C C C
Iter 1
Iter 2
Iter 3
General Core
BERET + General Core
Iter 3 (Replay)
Iter 1
Iter 2
Iter 3 F F F F F F
CFU2
CFU2
CFU2 D
CFU1
CFU1
CFU1 D D D D D br br br ld + ld + ld + ld + br + br - br - br - br P P P P P P
Serialization Edges C C C C C C
Accelerator Entry

15. Accelerator+Core Dependence Graph
TDG Framework
gem5
Simulator D P
Dep.
Graph
Program IR D P
Dep.
Graph
TDG Analyzer & Transformer
(McPAT/CACTI)
TDG Constructor
(Accelerator Analysis +
Graph Transformation)
TDG: Transformable Dependence Graph
Accelerator+Core Dependence Graph
So that was an example, and our TDG modeling framework looks like this. We use gem5 to produce the original TDG, and we implement our own TDG analysis and transformation library.
This is easier than a compiler+simulator approach, because adding an accelerator is merely a matter of adding an accelerator analysis pass and graph transformation pass.
Accelerator+Core
Cycle Time & Power
Framework coming to a Docker container soon!

16. Does TDG-modeling work?
(compared to the Compiler+Simulator Approach)
Accelerator
Average Error in Speedup
Average Error in Energy Reduction
Conserv. Cores 5%
10%
BERET 8% 7%
SIMD
12%
DySER
15%
(compared to published numbers)

17. Behavior Specialization Challenges
Methodological
How to study these in an effective way?
Transformable Dependence Graph (TDG)
Design/Analysis
Transformable dependence graph modeling methodology for behavior specialized accelerators, now I want to talk about how we can create an effective core organization.
How to compose and organize accelerators?

18. Choosing Synergistic Accelerators
General Purpose Code
Data Parallel
Non-Data
Parallel
Low Control
Some
Control
High Control
High ILP
Low ILP
The intuitive way to choose accelerators is to classify program phases into different behaviors, and create an accelerator for each phase type. So in this slide I’ll describe one possible program phase classification, that we use in this work. …
SIMD
Separable Datapath
Non-critical Control
Repetitive Control
Unpredictable
Control
DP-CGRA
NS-DF
Trace Proc.

19. Compute slice to CGRA offloading
Target
properties:
Data-Parallel,
Low-control
Separable Datapath
Irregular Code,
Non-critical Control
Repetitive Control
Related
Publication:
DySER, HPCA 2011
SEED, ISCA 2015
BERET, MICRO 2011
(modified for dataflow)
Essential
Mechanism:
Short vector
instructions
Compute slice to CGRA offloading
Config. dataflow, no control speculation
Speculates trace path, re-execution
Error Checking
Sparse MM
Histogram
Graph Processing
Stencil
Dynamic Prog.
Mat. Mult.
Image Process.
Example
Workloads:
High level view of the architecture (general purpose processor and 4 accelerators integrated behind a private cache. That allows us to share data and switch between accelerators quickly)
Without getting into the architecture, I want to briefly describe these.
Most of us know what SIMD is, it targets data parallel workloads with low control
Data-parallel CGRA can efficiently execute data-parallel regions with control flow, and performance by offloading large compute datapaths onto a coarse grain reconfigurable architecture
Non-speculative dataflow processor is good for irregular code phases where the control decisions are not on the critical path.
Trace-processor is a dataflow-extension of the beret processor I showed earlier (it’s good at executing codes with very repetitive control decisions).

20. Evaluation Methodology
Workloads
Regular: Intel Microbenchmarks, Parboil
Semi-Regular: TPCH Queries, Mediabench, SpecFP
Irregular: SpecINT
General Core Configurations
Little
(IO2)
Medium
(OOO2)
Big
(OOO4)
Very Big
(OOO6)
Issue Width 2 4 6
ROB Size - 64
168
192
Instr. Window 32 48 52
Dcache Ports 1 3
FUs (ALU, Mul/Div,FP)
2,1,1
2,1,1,
3,2,2
4,2,3
We study this architecture with a wide variety of workloads and general purpose cores.

21. Evaluation Questions
Does ExoCore have high potential speedup or energy efficiency improvements?
Do program behaviors span multiple domains (and how much coverage)?
What new design-space opportunities does does an ExoCore approach enable?

22. ExoCore pushes frontier by 1.4-2.0× Speedup,
Overall Results
ExoCore
>2× Energy
Benefit
ExoCore pushes frontier by × Speedup,
× Energy Eff.

23. Per-Benchmark Execution Time
Execution Time Relative to OOO2
OOO2 + 4-BSA vs OOO2 Exec. Time
Parboil
Mediabench
TPCH
SPECFP
SpecINT
Intel µbench
2. Exploited program behaviors span across domains
(~80% Coverage of program execution time)

24. × × Design Space Cores Accelerators Metrics Speedup Energy Area
Tiny Inorder (IO2)
Small OOO (OOO2)
SIMD
(S)
DP-CGRA
(D)
Trace-P
(T)
NS-DF
(N)
Speedup
Energy
Area × ×
Medium OOO (OOO4)
Large OOO (OOO6)
Finally, the ExoCore organization opens up a huge design space in combining different combinations of accelerators and cores together. We have explored this design space across different metrics.
(4 Cores)
(16 Combinations)

25. 3. Many new design tradeoffs enabled
Design Space Results
Performance
Energy-Efficiency
Large
OOO
SIMD
DP-CGRA
Trace-P
NS-DF
Tiny IO
SIMD
DP-CGRA
NS-DF
Energy-Delay
Energy-Delay-Area
Without boring you with data, I’ll just summarize the results here.
Small/
Medium
OOO
SIMD
DP-CGRA
Trace-P
NS-DF
Small
OOO
DP-CGRA
NS-DF
3. Many new design tradeoffs enabled

26. Conclusions ExoCore Architecture Organization
Common behaviors spanning domains and program execution
Significant performance/energy advantages + new design space
Transformable Dependence Graph (TDG)
Common framework for accelerator modeling and comparisons
Higher abstraction while remaining detailed and accurate
BSA 2
General Core
BSA 3
BSA 1
BSA 4
ExoCore: Modular Behavior Specialized Core D P 1 2 3 4
ExoCore architecture organization is a promising way forward to improve performance/energy efficiency of general purpose processors without relying on technology scaling.
Transformable dependence graph is a modeling framework which greatly simplifies early design-space exploration of such designs. 1 2 3
Core’s
Dep. Graph
Program IR
Transformable Dependence Graph (TDG)