1 The Performance Potential for Single Application Heterogeneous Systems Henry Wong* and Tor M. Aamodt § *University of Toronto § University of British Columbia

2 Intuition suggests integrating parallel and sequential cores on a single chip should provide performance benefits by lowering communication overheads.

3 This work: Perform limit study of heterogeneous architecture performance when running a single general purpose application. Two main results: Single thread performance (read-after-write latency) of GPUs ought to improve for GPUs to accelerate a wider set of non-graphics workloads. Putting CPU and accelerator on single chip does not seem to improve performance “much” versus separate CPU and accelerator.

4 Outline Introduction Background: - GPU Computing / Heterogeneous - Barrel processing (relevant to GPUs) Limit Study Model - Sequential and Parallel Models - Dynamic programming algorithm - Modeling Bandwidth Results

5 Graphics Processing Unit (GPU) Polygons Textures Lights

6 Programmable GPU Rendering pipeline Polygons go in Pixels come out DX10 has 3 programmable stages

7 GPU/Stream Computing Use shader processors without rendering pipeline C-like high-level language for convenience

8 Separate GPU + CPU Off-chip latency Copy data between memory spaces

9 Single-Chip Lower latency Single memory address space: Share data, don't copy

10 Sequential Performance of Parallel Processor Contemporary GPUs have slow single thread performance. “Designed for cache miss” => use “barrel processing” to hide off-chip latency. This impacts minimum read-to-write latency for a single thread. Not an issue if you have 10 6 pixels each requiring 100 instruction long thread.

11 Sequential Performance of Parallel Processor GPUs can do many operations per clock cycle Nvidia G80 needs 3072 independent instructions every 24 clocks to keep pipelines filled Can model G80 as executing up to 3072 independent scalar instructions every 24 clocks For single thread CPU produces results ~100x faster: 2 IPC * 2 clock speed * 24 instruction latency Parallel Instruction Latency = ratio of read-to-write latency of dependent instructions on parallel processor (measured in CPU clock cycles) to CPU CPI.

12 Limit Study Optimistic abstract model of GPU and CPU “ILP limit study”-type trace analysis with optimistic assumptions. Assume constant CPI (=1.0) for sequential core. Parallel processor is ideal data flow processor, but with read- after-write latency some multiple of the sequential core clock. Parallel processor has unlimited parallelism Optimally schedule instructions on cores using dynamic programming algorithm.

13 Trace Analysis Assumptions Perfect branch prediction Perfect memory disambiguation Remove stack-pointer dependencies Remove induction variable dependencies by removing all instructions that depend (dynamically) only on compile time constants.

14 Scheduling a Trace

15 Dynamic Programming Switching between processors takes time Find optimal schedule by decomposing problem, using optimal solution to subproblem to create optimal solution to larger problem. Input: Trace of N instructions. Output: Optimum (minimum) number of cycles required to execute on abstract heterogeneous processor model. serial parallel serial parallel instructions

16 Optimal algorithm is quadratic in instruction trace length. Approximation: First, sort trace of instructions in dataflow order to uncover parallelism. Then, apply dynamic programming over traces of 30,000 instructions.

17 Bandwidth Latency of mode switch depends upon amount of data consumed on new processor produced by old processor. Use earliest-deadline-first scheduling. Simple model of bandwidth, e.g., max 32-bits every 8 cycles. Allow overlap of computation with communication. Iterative model: Use average mode switch latency from last iteration as fixed mode switch latency for next iteration. Results based upon actual implied latency of last iteration.

18 PTLSim (x86-64): micro-op traces SimPoint (phase classification): ~12 x 10M instruction segments. Benchmarks: Spec 2000, PhysicsBench, SimpleScalar (used as a benchmark), microbenchmarks. Experiment Setup

19 Average Parallelism As in prior ILP limit studies: lots of parallelism.

20 Instructions Scheduled on Parallel Cores As parallel processor’s sequential performance gets worse, more instructions scheduled on sequential core.

21 Parallelism on Parallel Processor As parallel processor’s sequential performance gets worse, work scheduled on parallel core needs to be more parallel.

22 Speedup over Sequential Core Applications exist with enough parallelism to fully utilize GPU function units. GPU

23 Speedup over Sequential Core “General Purpose” Workloads: Performance limited by sequential performance (read-after-write latency) of parallel cores. GPU

24 Slowdown of infinite communication cost (NoSwitch) Up to 5x performance improvement versus infinite cost. Communication cost matters most for GPU like parallel instruction latency. So, put on same chip?

25 Slowdown due to 100,000 cycles of mode-switch latency Can achieve 85% of the performance of single-chip with large (but not infinite) mode switch latency.

26 Mode Switches Number of mode switches decreases with increasing mode switch cost. More mode switches occur at intermediate values of parallel instruction latency. zero cycles 10 cycles 1000 cycles

27 PCI Express-like Bandwidth (and Latency) 1.07x to 1.48x performance improvement if reduce latency to zero and make bandwidth infinite. Less improvement if parallel instruction latency reduced--e.g. for better accelerator architecture.

28 Conclusions & Caveats GPUs could tackle more general-purpose applications if single thread performance was better. Performance improvement due to integrating CPU and accelerator on single chip (versus separate CPU and accelerator) does not appear staggering. Bandwidth has greater impact than latency. Caveats: It’s a limit study. Heterogeneous may still make sense for other reasons… e.g., if cheaper to add parallel cores than another chip sockets, power, etc…

29 Future Work Control dependence analysis Model interesting design points in more detail

30 Bandwidth sensitivity for GPU-like parallel instruction latency

31 Proportion of instructions on parallel processor

32 Slowdown of infinite communication Twophase shows strong sensitivity to communication latency for widely varying parallel instruction latency