1. Ahmad Lashgar, Amirali Baniasadi, Ahmad Khonsari ECE, University of Tehran, ECE, University of Victoria

2. This Work  Accelerators o Designed to maximize throughput o ILT: fetch the same instruction repeatedly o Wasted  Our solution: o Keep fetched instructions in small buffer, save energy  Key result: 19% front-end energy reduction Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 2

3. Outline  Background  Instruction Locality  Exploiting Instruction Locality o Decoded-Instruction Buffer o Row Buffer o Filter Cache  Case Study: Filter Cache o Organization o Experimental Setup o Experimental Results Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 3

4. Heterogonous Systems  Heterogonous system to achieve optimal performance/watt o Superscalar speculative out-of-order processor for latency-intensive serial workloads o Accelerator (Multi-threaded in-order SIMD processor) for High-throughput parallel workloads  6 of 10 Top500.org supercomputers today employ accelerators o IBM Power BQC 16C 1.60 GHz (1 st, 3 th, 8 th, and 9 th ) o NVIDIA Tesla (6 th and 7 th ) 4 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

5. GPUs as Accelerators  GPUs are most available accelerators o Class of general-purpose processors named SIMT o Integrated on same die with CPU (Sandy Bridge, etc)  High energy efficiency o GPU achieves 200 pJ/instruction o CPU achieves 2 nJ/instruction 5 [Dally’2010] Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

6. SIMT Accelerator  SIMT (Single-Instruction Multiple-Thread)  Goal is throughput  Deep-multithreaded  Designed for latency hiding  8- to 32-lane SIMD 6 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

7. Streaming Multiprocessor (SM), CTA & Warps  Threads of same thread-block (CTA) o Communicate through fast shared memory o Synchronized through fast synchronizer  A CTA is assigned to one SM  SMs execute in warp (group of 8-32 threads) granularity 7 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

8. Warping Benefits  Thousands of threads are scheduled zero-overhead o Context of threads are all on core  Concurrent threads are grouped into warps o Share control-flow tracking overhead o Reduce scheduling overhead o Improve utilization of execution units (SIMD efficiency) 8 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

9. Energy Reduction Potential in GPUs  Huge amount of context o Caches o Shared Memory o Register file o Execution units  To many inactive threads o Synchronization o Branch/Memory divergence  High Locality o Similar behavior by different threads 9 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

10. Baseline Pipeline Front-end  Modeled according to NVIDIA Patents  3-Stage Front-end o Instruction Fetch (IF) o Instruction Buffer (IB) o Instruction Dispatch (ID)  Energy breakdown o I-Cache second most energy consuming 10 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

11. SIMD Back-end SM Pipeline Front-end Example Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 11 W1↓↓↓↓ W2↓↓↓↓ Warp Scheduler I-Cache Instruction Buffer insnsrc1src2dest W1 W2 Scoreboard Field1Field2 W1 W2 Instruction Scheduler Operand Buffering lane1lane2lane3lane4 Register File W1 W2 12 Code sequence: 1: add r2 <- r0, r1 2: ld r3 <- [r2] 3: add r0 r1 r2 ld r2 -- r3 r2 r0 for all lanes r1 for all lanes r2 for all lanes r3 for all lanes r0 for all lanes r0 r0t0 r0t1 r0t2 r0t3 r1 r1t0 r1t1 r1t2 r1t3 r0t0 r1t0 r0t1 r1t1 r0t2 r1t2 r0t3 r1t3 r2t0 r2t1 r2t2 r2t3 r3 r2 r2t0 r2t1 r2t2 r2t3 r2t0 0 r2t1 0 r2t2 0 r2t3 0 r3t0 r3t1 r3t2 r3t3 1 3 r2 for all lanes r3 for all lanes 1 PC

12. Inter-Thread Instruction Locality (ITL)  Warps are likely to fetch/and decode same instruction  The percentage of instructions already fetched by other currently active warps recently: 12 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

13. Exploiting ITL  Toward performance improvement o Minor improvement by reducing the latency of arithmetic pipeline  Toward energy saving o Fetch/decode bypassing similar to loop buffering o Reducing accesses to I-Cache Row buffer Filter Cache (our case study) 13 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

14. Decoded- Instruction Buffer Fetch/Decode Bypassing Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 14 W1↓↓↓↓ W2↓↓↓↓ Warp Scheduler Instruction Buffer insnsrc1src 2 dest W1 W2 add r0 r1 r2 ld r2 -- r3 1 1 PC Decoded-insn I-Cache I-Cache tag I-Cache data No need to access I-Cache and decode logic if buffer hits PC Buffer can bypass 42% of instruction fetches

15. Row Buffer Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 15 W1↓↓↓↓ W2↓↓↓↓ Warp Scheduler I-Cache Instruction Buffer insnsrc1src2dest W1 W2 add r0 r1 r2 ld r2 -- r3 1 1 PC I-Cache tag I-Cache data MUX Buffer last accessed I-Cache line PC Row Buffer

16. Filter Cache (Our Case Study) Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 16 W1↓↓↓↓ W2↓↓↓↓ Warp Scheduler I-Cache Instruction Buffer insnsrc1src2dest W1 W2 add r0 r1 r2 ld r2 -- r3 1 1 PC Filter Cache I-Cache tag I-Cache data MUX Buffering last fetched instruction in a set-associative table PC

17. Filter Cache Enhanced Front-end  Bypass I-Cache accesses to save dynamic power  32-entry (256-byte) FC o FC hit rate Up to ~100% o Front-end Energy Saving Up to 19% o Front-end area overhead 4.7% o Front-end leakage overhead 0.7% 17 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

18. Methodology  Cycle-accurate simulation of CUDA workloads by GPGPU-sim o Configured to model NVIDIA Tesla architecture o 16 8-wide SMs o 1024 threads/SM o 48 KB D-L1$/SM o 4 KB I-L1$/SM (256-byte lines)  21 Workloads from: o RODINIA (Backprop, …) o CUDA SDK (Matrix Multiply, …) o GPGPU-sim (RAY,…) o Parboil (CP) o Third-party sequence alignment (MUMmerGPU++) Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 18

19. Methodology (2)  Energy evaluations under 32-nm technology using CACTI 19 Area (μm 2 ) Leakage (mW) Energy per R/W (pJ)Delay (ps) I-Cache tag2290.030.13115.94 I-Cache data182041.784.30221.20 Instruction Buf.26000.161.00137.59 Scoreboard69210.241.57162.17 Operand Buf.241730.534.16174.05 FC tag (32-entry)2660.030.14117.28 FC data (32-entry)22290.110.81161.76 FC tag (16-entry)1550.020.10105.47 FC data (16-entry)13370.050.57143.38 Modeled by a wide tag array Modeled by a data array Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs Scoreboard6921 Instruction Buf. Operand Buf.

20. Experimental Results  FC hit rate and energy saving o 32-entry FC o 1024-thread per SM o Round-robin warp scheduler  Sensitivity analysis under o FC size o Thread per SM o Warp scheduler 20 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs

21. FC Hit Rate and Energy Saving FC hit rate Baseline I-Cache energy (nJ) I-Cache + FC energy (nJ) Front-end energy-saving using FC CP100%16532.206616.157% HSPT 89%12076.705676.648% LPS83%14955.057625.978% MP30%161.40139.782% MTM95%347.37153.668% NN99%132820.8653780.9219% RAY76%10520.695945.776% SCN97%562.47240.709% Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 21 Low-Concurrent Warps, Divergent BranchHigh-Concurrent Warps, Coherent Branch MP30%161.40139.782% CP100%16532.206616.157%

22. Sensitivity Analysis  Filter Cache size o Larger FC provides higher hit-rate but has higher static/dynamic energy  Thread per SM o Higher thread per SM, higher the chance of instruction re-fetch  Warp Scheduling o Advanced warp schedulers (latency hiding or data cache locality boosters) may keeping the warps at the different paces Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 22 Round-robinTwo-level W0W1W0W1W0W1 Compute Memory Pending W0 W1 Time

23. Sensitivity to Multithreading-Depth 23 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 1024512Threads Per SM: ~ 1% hit reduction ~ 1% reduction in savings

24. Sensitivity to Warp Scheduling 24 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs Round-RobinTwo-LevelWarp Scheduler: ~1% hit reduction ~1% reduction in savings

25. Sensitivity to Filter Cache Size 25 Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 3216Number of entries in FC: ~5% hit reduction up to ~1% reduction in savings (due to lower hit rate ) Overall ~2% increase in savings (due to smaller FC)

26. Conclusion & Future Works  We have evaluated instruction locality among concurrent warps under deep-multithreaded GPU  The locality can be exploited for performance or energy- saving  Case Study: Filter cache provides 1%-19% energy-saving for the pipeline  Future Works: o Evaluating the fetch/decode bypassing o Evaluating concurrent kernel GPUs Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 26

27. Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 27 Thank you! Question?

28. Backup-Slides Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 28

29. References [Dally’2010] W. J. Dally, GPU Computing: To ExaScale and Beyond, SC 2010. Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 29

30. Workloads Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 30 Abbr. Name and Suite Grid SizeBlock Size#InsnCTA/SM BFSBFS Graph [2]16x(8)16x(512)1.4M1 BKPBack Propagation [2]2x(1,64)2x(16,16)2.9M4 CPCoulumb Poten. [19](8,32)(16,8)113M8 DYNDyn_Proc [2]13x(35)13x(256)64M4 FWALFast Wal. Trans. [18] 6x(32) 3x(16) (128) 7x(256) 3x(512) 11M2, 4 GASGaussian Elimin. [2]48x(3,3)48x(16,16)9M1 HSPTHotspot [2](43,43)(16,16)76M2 LPSLaplace 3D [1](4,25)(32,4)81M6 MP2MUMmer-GPU++ [8] big(196)(256)139M2 MPMUMmer-GPU++ [8] small(1)(256)0.3M1 MTMMatrix Multiply [18](5,8)(16,16)2.4M4 MU2MUMmer-GPU [2] big(196)(256)75M4

31. Workloads (2) Inter-Warp Instruction Temporal Locality in Deep-Multithreaded GPUs 31 Abbr. Name and Suite Grid SizeBlock Size#InsnCTA/SM MUMUMmer-GPU [2] small(1)(100)0.2M1 NNCNearest Neighbor [2]4x(938)4x(16)5.9M8 NNNeural Network [1] (6,28) (25,28) (100,28) (10,28) (13,13) (5,5) 2x(1) 68M5, 8 NQUN-Queen [1](256)(96)1.2M1 NWNeedleman-Wun. [2] 2x(1) … 2x(31) (32) 63x(16)12M2 RAYRay Tracing [1](16,32)(16,8)64M3 SCNScan [18](64)(256)3.6M4 SR1Speckle Reducing [2] big4x(8,8)4x(16,16)9.5M2, 3 SR2Speckle Reducing [2] small4x(4,4)4x(16,16)2.4M1