Warp-Level Divergence in GPUs: Characterization, Impact, and Mitigation Ping Xiang, Yi Yang, Huiyang Zhou 1 The 20th IEEE International Symposium On High Performance Computer Architecture, Orlando, Florida, USA

Outline Background Motivation Mitigation: WarpMan Experiments Conclusions 2

Register File Threads 3 Overview of GPU Architecture ALU Control ALU Cache ALU Warp DRAM Warp TB … … Shared Memory

Motivation: Typically large TB size (512, e.g.) –More efficient data sharing/communication within a TB –Limited total TB number Register File TB Unused Registers Resource Fragmentation 4

Motivation Warp-Level Divergence: …. TB Warp1 Warp2 Warp3 Warp4 Finished warps within the same TB don’t finish at the same time Resources cannot be released promptly Unused Resources 5

Outline Background Motivation –Characterization : Mitigation: WarpMan Experiments Conclusions 6

Characterization: Register File TB Unused Resources Spatial Resource underutilization Finished Temporal Resource underutilization 7

Spatial Resource Underutilization Register resource as an example 28%17% 46% 8

Temporal Resource Underutilization Case Study: Ray Tracing –6 warps per TB –Study TB0 as an example RTRU = 49.7% RTRU: ratio of temporal resource underutilization 9

Why There Is Temporal Resource Underutilization? Input-dependent workload imbalance –Same code, different input: “if(a < 123)” Program-dependent workload imbalance –Code like if(tid < 32) Memory divergence –Some warps experience more cache hits than others Warp scheduling policy –Scheduler prioritizes certain warps than others 10

Characterization: RTRU 11

Outline Background Motivation –Characterization : –Micro-benchmarking Mitigation: WarpMan Experiments Conclusions 12

Micro-benchmark Code runs on both GTX480 and GTX __global__ void TB_resource_kernel(…, bool call = false){ 2. if(call) bloatOccupancy(start, size); clock_t start_clock = clock(); 4. if(tid < 32){ //tid is the threadid within a TB 5 clock_offset = 0; 6. while( clock_offset < clock_count ) { 7. clock_offset = clock() - start_clock; 8. } 9. } 10. clock_t end_clock = clock(); 11. d_o[index] = start_clock; //index is the global thread id 12. d_e[index] = end_clock; 13.} 13

Micro-benchmarking Results > Using CUDA device [0]: GeForce GTX 480 > Detected Compute SM 3.0 hardware with 8 multi-processors. … CTA 250 Warp 0: start 80, end 81 CTA 269 Warp 0: start 80, end 81 CTA 272 Warp 0: start 80, end 81 CTA 283 Warp 0: start 80, end 81 CTA 322 Warp 0: start 80, end 81 CTA 329 Warp 0: start 80, end 81 … 14

Outline Background Motivation Mitigation: WarpMan Experiments Conclusions 15

WarpMan SM TB0 TB1 TB-level Resource Management Unused Resources Finished Warp2 Finished Warp1 Finished Warp0 TB2 16 Warp Level Resource Management cycle Workload TB0 TB2 TB1 Warp2 Warp0 Warp1

SM TB0 TB1 TB-level Resource ManagementWarpMan SM TB0 TB1 Unused Resources Finished Warp TB2 Warp0 From TB2 Warp1 From TB2 Finished Released Resource Warp2 From TB2 17 cycle Workload Warp0 and warp 1 WarpMan TB0 TB1 Warp2 warp2 Warp0 Warp1 Saved Cycle Warp Level Resource Management

WarpMan ---- Design Dispatch logic –Traditional TB-level dispatching logic –Add partial TB dispatch logic Workload buffer –Store the dispatched but not running partial TBs 18

Dispatching TB-level Resource Check Warp-level Resource Check Resources required for a TB Resources required for a Warp A full TB A partial TB Workload to be dispatched Shared memory Warp entries TB entries Registers 19 The shared memory is still allocated at the TB level

Workload Buffer Store the dispatched but not running TB –Hardware TB id (assigned by the hardware) –Software TB id (defined by the software) –Start warp id –End warp id –Valid bit bits 20

Workload Buffer Store the dispatched but not running TB TB120 WarpMan SM TB118 TB117 Unused Resources TB Num Warp0 From TB120 Warp1 From TB120 Start Warp ID End Warp ID Valid Workload buffer Finished Warp2 From TB

Outline Background Motivation: Mitigation: WarpMan Experiments Conclusions 22

Methodology Use GPUWattch for both timing and energy evaluation Baseline Architecture: (GTX480) –15 SMs, with SIMD size of 32, running at 1.4Ghz –Max TBs per SM is 8, Max threads per SM is 1536 –Scheduling policy: round robin / two level –16KB L1 cache, 48 KB shared memory. 128KB regs Applications from: Nvidia CUDA SDK Rodinia Benchmark Suit GPGPUsim 23

Performance Results: temp: allow early finished warps to release resource for new warps temp + spatial: resources are allocated /released at warp level The performance improvements can be as high as 71%/76% On average, 15.3% improvements 24

Energy Results The energy savings can be as high as over 20%, and 6% on average

A Software Alternative Change the software to have a smaller TB size Change the hardware to enable more concurrent TBs Inefficient shared memory usage / synchronization Decrease the data locality More as we proceed to the experimental results… a smaller TB size 26

Comparing to the Software Alternative CT and ST: software approach decreases L1 locality NN and BT: reduced total number of threads On average: 25% improvement VS 48% degradation 125% 52% 27

Related Work Resource underutilization due to branch divergence or thread- level divergence has been well studied. Yi Yang et al [Pact-21] targets at the shared memory resource management and is complementary to our proposed WarpMan scheme. D. Tarjan, et al [US Patent-2009], proposes to use virtual register table to manage physical register file to enable more concurrent TBs 28

Conclusion We highlight the limitations of TB-level resource management we characterize warp-level divergence and reveal the fundamental reasons for such divergent behavior; we propose WarpMan and show that it can be implemented with minor hardware changes we show that our proposed solution is highly effective and achieves significant performance improvements and energy savings Questions? 29