1. Muthu Baskaran 1 Uday Bondhugula 1 Sriram Krishnamoorthy 1 J Ramanujam 2 Atanas Rountev 1 P Sadayappan 1 1 Department of Computer Science & Engineering The Ohio State University 2 Department of Electrical and Computer Engineering Louisiana State University A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs * * Supported by NSF

2. Introduction  Emergence of many-core architectures ◦ High computation power ◦ E.g. GPUs  Development of high-performance codes in such architectures – Non-trivial!  CUDA parallel programming model for NVIDIA GPUs ◦ Good abstraction of the underlying architecture ◦ Not straight-forward to write a high-performance CUDA code A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

3. Introduction  Optimizations needed to address architectural challenges ◦ Memory access model ◦ Granularity and levels of parallelism in architecture  Solution: Compiler infrastructure to automatically generate efficient parallel programs  PLuTo compiler framework [PLDI08] recently developed for general-purpose multi-core targets ◦ Sequential C to OpenMP Parallel Tiled Code  Develop a framework to automatically generate parallel CUDA code A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

4. ◦ An algebraic framework for representing affine programs – statement domains, dependences, array access functions – and affine program transformations ◦ Regular affine programs ◦ Dense arrays ◦ Loop bounds – affine functions of outer loop variables, constants and program parameters ◦ Array access functions - affine functions of surrounding loop variables, constants and program parameters Polyhedral Model A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

5. for (i=1; i<=7; i++) for (j=2; j<=6; j++) S1: a[i][j] = a[j][i] + a[i][j-1]; ijij x S1 =. 0 -1 4 I S1 = ij1ij1 ≥ 0 -1 0 4 0 1 -2 1 0 -1 ₣ 1a (x S1 ) = 1 0 0 1. ijij + 0 ₣ 2a (x S1 ) = 0 1 1 0. ijij + 0 ₣ 3a (x S1 ) = 1 0 0 1. ijij + 0 -1 j i≥1 i≤7 i j≥2j≤6 Polyhedral Model Φ S (x S ) =. xSn1xSn1 CSCS A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

6. PLuTo Framework ◦ Available at http://sourceforge.net/projects/pluto-compiler A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

7. NVIDIA GPU Architecture  Two levels of parallelism ◦ Threads (Processor cores)  Grouped into SIMD warps ◦ Thread blocks (Multiprocessors)  Various memory units ◦ Different memory access model ◦ Cache and local store hierarchy  Partitioning (e.g. registers) and sharing of resources (e.g. shared memory) Off-chip memory... P1P2P8 Shared Memory Constant Cache Registers Texture Cache SM1 SM16 A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

8. Performance Characterization of NVIDIA GeForce 8800 GTX  Get insights into optimizations to be addressed by a compiler framework  Characterize key features of the machine architecture and their impact on different strategies: ◦ Global memory access ◦ Shared memory (scratchpad) access ◦ Concurrency and register pressure A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

9. Global Memory Access  Measured memory read bandwidth for ◦ Different data sizes ◦ Blocked and cyclic distribution of data access amongst the threads of a single thread block A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

10. Global Memory Access  Cyclic access has much higher bandwidth  Hardware optimization called global memory coalescing ◦ Access from consecutive threads of a (half) warp to consecutive locations coalesced ◦ Base address of (half) warp aligned to 4, 8 or 16 bytes A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

11. Optimizing Global Memory Access  Determine the extent of reuse of arrays  For arrays with sufficient reuse ◦ Copy from global memory to shared memory [PPoPP08]  For arrays with no reuse ◦ Find affine transformations enabling global memory coalescing ◦ If no suitable affine transformation enabling global memory coalescing  Copy to shared memory with possible global memory coalescing A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

12. Optimizing Global Memory Access j i ayx mv kernel: for (i=0;i<n;i++) { P: x [i]=0; for (j=0;j<n;j++) Q: x[i]+=a[i][j] * y[j]; } tmv kernel: for (i=0;i<n;i++) { S: x[i]=0; for (j=0;j<n;j++) T: x[i]+=a[j][i] * y[j]; } A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

13. Experimental Evaluation NDirect Global Copied to Shared 4K0.435.61 5K0.485.79 6K0.356.04 7K0.305.78 8K0.245.52 Performance comparison (in GFLOPS) of mv kernel A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

14. Experimental Evaluation NDirect Global Copied to Shared 4K0.435.61 5K0.485.79 6K0.356.04 7K0.305.78 8K0.245.52 NNon optimized Global Optimized Global 4K4.2225.21 5K3.0928.90 6K3.2433.47 7K3.7033.58 8K4.1334.93 Performance comparison (in GFLOPS) of mv kernel Performance comparison (in GFLOPS) of tmv kernel A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

15. Shared Memory Access  Shared memory organized into banks ◦ 16 banks in NVIDIA 8800 GTX ◦ Successive 32-bit words in successive banks  Bank conflicts in shared memory ◦ n threads access different address in same bank – n sequential requests (n-way conflict) ◦ Bandwidth of shared memory access inversely proportional to the degree of bank conflicts  Goal: To minimize shared memory bank conflicts A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

16. Optimizing Shared Memory Access  Strategy to minimize bank conflicts during shared memory access ◦ Pad the arrays copied into shared memory  Degree of bank conflicts = gcd (stride of array access across threads of a half warp, number of bank modules)  Cost of accessing a word in shared memory ◦ Linear function of degree of bank conflicts  Find padding factor that minimizes the cost considering all references to the array A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

17. Experimental Evaluation NNon optimized Share Optimized Share 4K5.6113.18 5K5.7913.87 6K6.0414.37 7K5.7813.86 8K5.5213.63 Performance comparison (in GFLOPS) of mv kernel A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

18. Parallelism vs Register Pressure  Performance enhancing approaches ◦ Reduction of number of loads/stores ◦ Increase in ILP ◦ Reduce dynamic instructions  Loop overhead reduction  Well-known optimization: Loop unrolling  Issues ◦ Increased register pressure ◦ Might reduce number of concurrent threads  Registers are partitioned among thread blocks A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

19. Parallelism vs Register Pressure  Higher thread-level parallelism needed to mask global memory access latency ◦ Threads scheduled in an interleaved manner to mask global memory access latency  Trade-off between ◦ number of active concurrent threads ◦ number of registers available for a thread in a thread block  Problem: Register allocation cannot be managed by any external compilation framework  Solution: Empirical evaluation to select an optimal choice A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

20. Model-driven Empirical Search  Need for empirical search ◦ Tight coupling  Program parameters – tile sizes, unroll factors  System parameters – threads, thread blocks  Resources - Number of registers, shared memory ◦ Lack of control on the registers (usage and allocation)  Model to estimate number of loads/stores ◦ Analytically in polyhedral model ◦ Empirically using ptx code  Register usage instrumentation ◦ Empirically using cubin object code A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

21. Model-driven Empirical Search ◦ Perform multilevel tiling (except register-level tiling) ◦ Generate optimal copy code ◦ Prune code versions by global memory traffic ◦ For all selected loop structures  do register-level tiling and explicit unrolling  instrument the register usage  discard those for which increased register pressure reduces concurrency to less than 25% of maximum possible concurrency ◦ In all selected code versions, pad the arrays in shared memory with optimal padding factor ◦ Search empirically among the remaining candidate loop structures  explicitly running them and timing the execution time and select the optimal one A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

22. Experimental Evaluation Performance of Matrix Kernels A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

23. Experimental Evaluation Performance of Matrix Kernels A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

24. ◦ Earlier GPU works  Automatic generation of pixel shader operations from a high- level data-parallel language – Tarditi et al. [ASPLOS’06]  Stream processing – Brook, RapidMind, PeakStream ◦ Considerable work on developing specific optimized algorithms and libraries for GPUs  E.g. CUDPP – CUDA Data Parallel Primitives ◦ Very little work on general compiler optimization strategies for GPUs  Performance metrics to prune the optimization search space on a pareto-optimality basis - by Ryoo et al. [CGO’08]  Optimize data communication between CPU and co-processor – by Gelado et al. [ICS’08] Related Work A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

25. Summary  Developed compiler optimizations to address key performance-influencing factors on NVIDIA GPUs ◦ Enable global memory coalescing in the polyhedral model for regular programs ◦ Reduce shared memory bank conflicts ◦ Determine optimized program parameters (unroll factors, tile sizes) through a model-driven empirical search A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

26. Ongoing and Future Work  Automatic thread-centric CUDA Code Generation in Polyhedral Model  Looking at data layout reordering to enhance memory accesses at various levels A Compiler Framework for Optimization of Affine Loop Nests for GPGPUs ICS’08

27. Thank You!

28.  Innermost core computations in many codes ◦ Dense linear algebra ◦ Image and signal processing ◦ Computational Electromagnetics (FDTD) ◦ Explicit PDE solvers (e.g. SWIM, SWEEP3D)  Likely to be more prevalent in future (esp. scientific) ◦ Codes with direct data access better than indirect data access: power & performance ◦ Structured-sparse (block sparse) better than arbitrary sparse (e.g. OSKI) ◦ Algorithms with sparse-outer but regular-inner structure may be more attractive for many-core processors, e.g. multi-block methods How prevalent are affine computations?