1. Performance Dominik G ö ddeke

2. 2Overview Motivation and example PDE –Poisson problem –Discretization and data layouts Five points of attack for GPGPU

3. 3 Example PDE: The Poisson Problem

4. 4 Discretization Grids Equidistant grids –Easy to implement –One array holds all the values –One array for right hand side –No matrix required, just a stencil

5. 5 Discretization Grids Tensorproduct grids –Reasonably easy to implement –Banded matrix, each band represented ass individual array N N Matrix i 2 Vectors N 12 2 GPU arrays 12 image courtesy of Jens Krüger

6. 6 Discretization Grids Generalized tensorproduct grids –Generality vs. efficient data structures tailored for GPU –Global unstructured macro mesh, domain decomposition –(an-)isotropic refinement into local tensorproduct meshes Efficient compromise –Hide anisotropies locally and exploit fast solvers on regular sub-problems: excellent numerical convergence –Large problems become viable

7. 7 Discretization Grids Unstructured grids –Bad performance for dynamic topology –Compact row storage format or similar –Challenging to implement: Indirection arrays Feedback loop to the vertex stage image courtesy of Jens Krüger

8. 8 Discretization Grids Adaptive grids –Handles coherent grid topology changes –Needs dynamic hash/tree structure and/or page table on the GPU –Actively being researched, see Glift project Physical Memory Virtual Domain Mipmap Page Table image courtesy of Aaron Lefohn

9. 9Overview Motivation and example PDE Five points of attack for GPGPU –Interpolation –On-chip bandwidth –Off-chip bandwidth –Overhead –Vectorization

10. 10 General Performance Tuning Traditional CPU cache-aware techniques –Blocking, reordering, unrolling etc. –Can not be applied directly No direct control of what actually happens –Hardware details are NDA‘ed, not public –Driver recompiles the code and might apply some SFCs or similar to arrange arrays in memory –Driver knows hardware details best, so let it work for you Only small cache, optimized for texture filtering –Prefetching of small local neighborhoods –Memory interface optimized for streaming sequential access

11. 11 Simplified GPU overview Vertex Processor (VP) Kernel changes index regions of input arrays Rasterizer Creates data streams from index regions Fragment Processor (FP) Kernel changes each datum independently, reads more input arrays CPU memory GPU memory

12. 12 First Point of Attack Vertex Processor (VP) Kernel changes index regions of input arrays Rasterizer Creates data streams from index regions Fragment Processor (FP) Kernel changes each datum independently, reads more input arrays CPU memory GPU memory take advantage of the interpolation hardware

13. 13Interpolation Recall how computation is triggered –Some geometry that covers the output region is drawn (more precisely, a quad with four vertices) –Index of each element in the output array is interpolated across the output region automatically already –This can be leveraged for all values that vary linearly over some region of the input arrays as well Example: Jacobi solver (cf. Session 1) -Typical domain sizes: 256x256, 512x512, 1024x1024 -Interpolate index math for neighborhood lookup as well

14. 14 float jacobi (float2 center : WPOS, uniform samplerRECT x, uniform samplerRECT b, uniform float one_over_h) : COLOR { float2 left = center – float2(1,0); float2 right = center + float2(1,0); float2 bottom = center – float2(0,1); float2 top = center + float2(0,1); float x_center = texRECT(x, center); float x_left = texRECT(x, left); float x_right = texRECT(x, right); float x_bottom = texRECT(x, bottom); float x_top = texRECT(x, top); float rhs = texRECT(b, center); float Ax = one_over_h * ( 4.0 * x_center - x_left - x_right – x_bottom – x_top ); float inv_diag = one_over_h / 4.0; return x_center + inv_diag*(rhs – Ax); } Interpolation Example void stencil (float4 position : POSITION, out float4 center: HPOS, out float2 left : TEXCOORD0, out float2 right : TEXCOORD1, out float2 bottom: TEXCOORD2, out float2 top : TEXCOORD3, uniform float4x4 ModelViewMatrix) { center = mul(ModelViewMatrix, position); left = center – float2(1,0); right = center + float2(1,0); bottom = center – float2(0,1); top = center + float2(0,1); } calculated 1024^2 times calculated 4 times extract offset calculation to the vertex processor float jacobi (float2 center : WPOS, uniform samplerRECT x, uniform samplerRECT b, in float2 left : TEXCOORD0, in float2 right : TEXCOORD1, in float2 bottom : TEXCOORD2, in float2 top : TEXCOORD3, uniform float one_over_h) : COLOR { float x_center = texRECT(x, center); float x_left = texRECT(x, left); float x_right = texRECT(x, right); float x_bottom = texRECT(x, bottom); float x_top = texRECT(x, top); float rhs = texRECT(b, center); float Ax = one_over_h * ( 4.0 * x_center - x_left - x_right – x_bottom – x_top ); float inv_diag = one_over_h / 4.0; return x_center + inv_diag*(rhs – Ax); } input vars after interpolation

15. 15 Interpolation Summary Powerful tool –Applicable to everything that varies linearly over some region –High level view: separate computation from lookup stencils Up to eight float4 interpolants available –On current hardware –Though using all 32 values might hurt in some applications Squeeze data into float4‘s –In this example, use 2 float4 instead of 4 float2

16. 16 Second Point of Attack Vertex Processor (VP) Kernel changes index regions of input arrays Rasterizer Creates data streams from index regions Fragment Processor (FP) Kernel changes each datum independently, reads more input arrays CPU memory GPU memory on-chip bandwidth

17. 17 Arithmetic Intensity Analysis of banded MatVec y=Ax, preassembled –Reads per component of y: 9 times into array x, once into each band –Operations per component of y: 9 multiply-adds Arithmetic intensity Operations per memory access Computation / bandwidth 18 reads 18 ops 18/18=1

18. 18 Precompute vs. Recompute Case 1: Application is compute-bound –High arithmetic intensity –Trade computation for memory access –Precompute as many values as possible and read in from additional input arrays Try to maintain spatial coherence –Otherwise, performance will degrade Rule of thumb –Need approx. 7 basic arithmetic ops to hide latency –Do not precompute x 2 if you read in x anyway

19. 19 Precompute vs. Recompute Case 2: Application is bandwidth-bound –Trade memory access for additional computation Example: Matrix assembly and Matrix-Vector multiplication –On-the-fly: recompute all entries in each MatVec Lowest memory requirement Good for simple entries or seldom use of matrix

20. 20 Precompute vs. Recompute –Partial assembly: precompute only few intermediate values Allows to balance computation and bandwidth requirements Good choice of precomputed results requires also little memory –Full assembly: precompute all entries of A Read entries during MatVec Good if other computations hide bandwidth problem Otherwise try to use partial assembly

21. 21 Third Point of Attack Vertex Processor (VP) Kernel changes index regions of input arrays Rasterizer Creates data streams from index regions Fragment Processor (FP) Kernel changes each datum independently, reads more input arrays CPU memory GPU memory off-chip bandwidth

22. 22 CPU - GPU Barrier Transfer is potential bottleneck –Often less than 1 GB/s via PCIe bus –Readback to CPU memory always implies a global syncronization point (pipeline flush) Easy case –Application directly visualizes results –Only need to transfer initial data to the GPU in a preprocessing step –No readback required –Examples: Interactive visualization and fluid solvers (cf. session 1)

23. 23 CPU - GPU Barrier Co-processor style computing –Readback to host is required –Don‘t want host or GPU idle: maximize throughput Interleaving computation with transfers –Apply some partitioning / domain decomposition –Simultaneously Prepare and transfer initial data for sub-problem i+1 Compute on sub-problem i Read back and postprocess result from sub-problem i-1 (causes pipeline flush but can‘t be avoided) –Good if input data is much larger than output data

24. 24 Fourth Point of Attack Vertex Processor (VP) Kernel changes index regions of input arrays Rasterizer Creates data streams from index regions Fragment Processor (FP) Kernel changes each datum independently, reads more input arrays CPU memory GPU memory overhead

25. 25 Playing it big Typical performance behaviour –CPU: Opteron 252, highly optimized cache-aware code –GPU: GeForce 7800 GTX, straight-forward, incl. transfers –saxpy, dot, MatVec CPU wins –Small problems –In-cache GPU wins –Large problems –Hide overhead + transfers

26. 26 Playing it big Nice analogy: Memory hierarchies –GPU memory is fast, comparable to in-cache on CPUs –Consider offloading to the GPU as manual prefetching –Always choose that type of memory that is fastest for the given chunk of data Lots of parallel threads in flight –Need lots of data elements to compute on –Otherwise, PEs won‘t be saturated Worst case and best case –Offload saxpy for small N individually to the GPU –Offload whole solvers for large N to the GPU (e.g. a full MG cycle)

27. 27 Fifth Point of Attack Vertex Processor (VP) Kernel changes index regions of input arrays Rasterizer Creates data streams from index regions Fragment Processor (FP) Kernel changes each datum independently, reads more input arrays CPU memory GPU memory instruction-level parallelism

28. 28Vectorization GPUs are designed to process 4-tupels of data –Same cost to compute on four float values as on one –Take advantage of co-issueing over the four components Swizzles –Swizzling components of the 4-tupels is free (no MOVs) –Example: data=(1,2,3,4) yields data.zzyx=(3,3,2,1) –Very useful for index math and storing values in float4‘s Problem –Challenging task to map data into RGBA –Very problem-specific, no rules of thumb

29. 29Conclusions Be aware of potential bottlenecks –Know hardware capabilities –Analyze arithmetic intensity –Check memory access patterns –Run existing benchmarks, e.g. GPUbench (Stanford) –Minimize number of pipeline stalls Adapt algorithms –Try to work around bottlenecks –Reformulate algorithm to exploit the hardware more efficiently