1. GPU Programming: eScience or Engineering? Henri Bal COMMIT/ msterdam Vrije Universiteit

2. Graphics Processing Units ● GPUs and other accelerators take top-500 by storm ● Many application success stories ● But GPUs are very difficult to program and optimize http://www.nvidia.com/object/ tesla-case-studies.html

3. Example: convolution ● About half a Ph.D. thesis Naive Fully optimized

4. Parallel Programming Lab course ● Lab course for MSc students (next to lectures) ● CUDA: ● Simple image processing application on 1 node ● MPI: ● Parallel all pairs shortest path algorithms ● CUDA: 11 out of 21 passed (52 %) ● MPI: 17 out of 21 passed (80 %)

5. Questions ● Why are accelerators so difficult to program? ● What are the challenges for Computer Science? ● What role do applications play?

6. Background ● Netherlands eScience Center ● Bridge between ICT and applications ● Climate modeling, astronomy, water management, digital forensics, … ● COMMIT: (100 M€) public-private ICT program ● http://www.commit-nl.nl/ ● Distributed ASCI Supercomputer (DAS) ● Testbed for Computer Science (Euro-Par 2014 keynote) COMMIT/

7. Cluster computing Zoo (1994), Orca Wide-area computing DAS-1 (1997), Albatross Grid computing DAS-2 (2002), Manta, Satin eScience & optical grids DAS-3 (2006), Ibis Hybrid computing DAS-4 (2010), Glasswing, MCL My background

8. Background (team) Ph.D. students ● Ben van Werkhoven ● Alessio Sclocco ● Ismail El Hewl ● Pieter Hijma Staff ● Rob van Nieuwpoort (NLeSC) ● Ana Varbanescu (UvA) Scientific programmers ● Rutger Hofman ● Ceriel Jacobs

9. Agenda Application case studies Multimedia kernel (convolution) Astronomy kernel (dedispersion) Climate modelling: optimizing multiple kernels Lessons learned: why is GPU programming hard? Programming methodologies ‘’Stepwise refinement for performance’’ methodology Glasswing: MapReduce on accelerators

10. Application case study 1: Convolution operations Image I of size I w by I h Filter F of size F w by F h Thread block of size B w by B h Naïve CUDA kernel: 1 thread per output pixel Does 2 arithmetic operations and 2 loads (8 bytes) Arithmetic Intensity (AI) = 0.25

11. Hierarchy of concurrent threads

12. Grid Global Memory Block (0, 0) Shared Memory Thread (0, 0) Registers Thread (1, 0) Registers Block (1, 0) Shared Memory Thread (0, 0) Registers Thread (1, 0) Registers Constant Memory Memory optimizations for tiled convolution Filter (small) goes into constant memory Threads within a block cooperatively load entire area they need into a small (e.g. 96KB) shared memory

13. Tiled convolution 16x16 thread block processing an 11x 7 filter ● Arithmetic Intensity:

14. Analysis ● If filter size increases: ● Arithmetic Intensity increases: ● Kernel shifts from memory-bandwidth bound to compute-bound ● Amount of shared memory needed increases → fewer thread blocks can run concurrently on each SM

15. Tiling ● Each thread block computes 1xN tiles in horizontal direction +Increases amount of work per thread +Saves loading overlapping borders +Saves redundant instructions -More shared memory, fewer concurrent thread blocks No shared memory bank conflicts

16. Adaptive tiling ● Tiling factor is selected at runtime depending on the input data and the resource limitations of the device ● Highest possible tiling factor that fits within the shared memory available (depending on filter size) ● Plus loop unrolling, memory banks, search optimal configuration Ph.D. thesis Ben van Werkhoven, 27 Oct. 2014 + FGCS journal, 2014

17. Lessons learned ● Everything must be in balance to obtain high performance ● Subtle interactions between resource limits ● Runtime decision system (adaptive tiling), in combination with standard optimizations ● Loop unrolling, memory bank conflicts

18. Application case study 2: Auto-tuning Dedispersion ● Used for searching pulsars in radio astronomy data ● Pulsar signals get dispersed: lower radio frequencies arrive progressively later ● Can be reversed by shifting in time the signal’s lower frequencies (dedispersion) Alessio Sclocco et al.: Auto-Tuning Dedispersion for Many-Core Accelerators, IPDPS 2014

19. Auto-tuning ● Using auto-tuning to find optimal configuration for: ● Different many-core platforms ● NVIDIA & AMD GPUs, Intel Xeon Phi ● Different observational scenarios ● LOFAR, Apertif ● Different number of Dispersion Measures (DMs) ● Represents number of free electrons between source & receiver ● Measure of distance between emitting object & receiver ● Parameters: ● Number of threads per sample or DM, thread block size, number of registers per thread, ….

20. Auto-tuning: number of threads per thread block LOFAR Apertif

21. Histogram of achieved GFLOP/s ● 396 configurations, the winner is an outlier

22. Lessons learned ● Auto-tuning allows algorithms to adapt to different platforms and scenarios ● Auto-tuning has large impact on dedispersion ● Guessing a good configuration without auto-tuning is difficult

23. Application case study 3: Global Climate Modeling ● Understand future local sea level changes ● Needs high-resolution simulations ● Combine two approaches: ● Distributed computing (multiple resources) ● GPUs COMMIT/

24. Distributed Computing ● Use Ibis to couple different simulation models ● Land, ice, ocean, atmosphere ● Wide-area optimizations similar to Albatross project (16 years ago), like hierarchical load balancing

25. Enlighten Your Research Global award EMERALD (UK) KRAKEN (USA) STAMPEDE (USA) SUPERMUC (GER) #7 #10 10G CARTESIUS (NLD) 10G

26. GPU Computing ● Offload expensive kernels for Parallel Ocean Program (POP) from CPU to GPU ● Many different kernels, fairly easy to port to GPUs ● Execution time becomes virtually 0 ● New bottleneck: moving data between CPU & GPU CPU host memory GPU device memory Host Device PCI Express link

27. Different methods for CPU-GPU communication ● Memory copies (explicit) ● No overlap with GPU computation ● Device-mapped host memory (implicit) ● Allows fine-grained overlap between computation and communication in either direction ● CUDA Streams or OpenCL command-queues ● Allows overlap between computation and communication in different streams ● Any combination of the above

28. Problem ● Problem: ● Which method will be most efficient for a given GPU kernel? Implementing all can be a large effort ● Solution: ● Create a performance model that identifies the best implementation: ● What implementation strategy for overlapping computation and communication is best for my program? Ben van Werkhoven, Jason Maassen, Frank Seinstra & Henri Bal: Performance models for CPU-GPU data transfers, CCGrid2014 (nominated for best-paper-award)

29. MOVIE

30. Example result MeasuredModel

31. Different GPUs (state kernel)

32. Different GPUs (buoydiff)

33. Comes with spreadsheet

34. Lessons learned ● PCIe transfers can have a large performance impact for applications with many small kernels ● Several methods for transferring data and overlapping computation & communication exist ● Performance modelling helps to select the best mechanism

35. Why is GPU programming hard? ● Mapping algorithm to architecture is difficult, especially as the architecture is difficult: ● Many levels of parallelism ● Limited resources (registers, shared memory) ● Less of everything than CPU (except parallelism), especially per thread, makes problem-partitioning difficult ● Everything must be in balance to obtain performance

36. Why is GPU programming hard? ● Many crucial high-impact optimizations needed: ● Data reuse ● Use shared memory efficiently ● Limited by #registers per thread, shared memory per thread block ● Memory access patterns ● Shared memory bank conflicts, global memory coalescing ● Instruction stream optimization ● Control flow divergence, loop unrolling ● Moving data to/from the GPU ● PCIe transfers

37. Why is GPU programming hard? ● Portability ● Optimizations are architecture-dependent, and the architectures change frequently ● Optimizations are often input dependent ● Finding the right parameters settings is difficult ● Need better performance models ● Like Roofline and our I/O model

38. Why is GPU programming hard? ● Bottom line: tension between ● control over hardware to achieve performance ● higher abstraction level to ease programming ● Programmers need understandable performance ● Old problem in Computer Science, but now in extreme form (1989)

39. Agenda Application case studies Multimedia kernel (convolution) Astronomy kernel (dedispersion) Climate modelling: optimizing multiple kernels Lessons learned: why is GPU programming hard? Programming methodologies ‘’Stepwise refinement for performance’’ methodology Glasswing: MapReduce on accelerators

40. Programming methodology: stepwise refinement for performance ● Methodology: ● Programmers can work on multiple levels of abstraction ● Integrate hardware descriptions into programming model ● Performance feedback from compiler, based on hardware description and kernel ● Cooperation between compiler and programmer P. Hijma et al., Stepwise-refinement for Performance: a methodology for many-core programming,” Concurrency and Computation: Practice and Experience (accepted)

41. MCL: Many-Core Levels ● MCL program is an algorithm mapped to hardware ● Start at a suitable abstraction level ● E.g. idealized accelerator, NVIDIA Kepler GPU, Xeon Phi ● MCL compiler guides programmer which optimizations to apply on given abstraction level or to move to deeper levels

42. MCL ecosystem

43. Convolution example

44. Compiler feedback

45. Performance (GTX480, 9×9 filters) 380 GFLOPS MCL: 302 GFLOPS Compiler +

46. Performance evaluation Compared to known, fully optimized versions (* measured on a C2050, ** using a different input).

47. Current work on MCL: Heterogeneous many-core clusters ● New GPUs become available frequently, but older- generation GPUs often still are fast enough ● Clusters become heterogeneous and contain different types of accelerators ● VU DAS-4 cluster: ● NVIDIA GTX480 GPUs (22) ● NVIDIA K20 GPUs (8) ● Intel Xeon Phi (2) ● NVIDIA C2050 (2), Titan, GTX680 GPU ● AMD HD7970 GPU

48. Cashmere ● Integration MCL + Satin divide-and-conquer system ● Satin [ACM TOPLAS 2010] does: ● Load-balancing (cluster-aware random work-stealing) ● Latency hiding ● MCL allows kernels to be written and optimized for each type of hardware ● Cashmere does integration, application logic, mapping, and load balancing for multiple GPUs/node

49. Cashmere skeleton

50. Kernel performance (GFLOP/s)

51. K-Means on a homogeneous GTX480 cluster scalability absolute performance

52. Heterogeneous performance Homogeneous: efficiency on 16 GTX480 Heterogeneous: efficiency over total combined hardware

53. Lessons learned ● MCL ● Enables us to develop many optimized many-core kernels ● Key: stepwise refinement + multiple abstraction levels ● Cashmere ● High performance and automatic load balancing even when the many-core devices differ widely ● Efficiency >90% in 3 out of 4 applications in heterogeneous executions

54. Agenda Application case studies Multimedia kernel (convolution) Astronomy kernel (dedispersion) Climate modelling: optimizing multiple kernels Lessons learned: why is GPU programming hard? Programming methodologies ‘’Stepwise refinement for performance’’ methodology Glasswing: MapReduce on accelerators

55. Other approaches that deal with performance vs abstraction ● Domain specific languages ● Patterns, skeletons, frameworks ● Berkeley Dwarfs

56. Glasswing: Rethinking MapReduce ● Use accelerators (OpenCL) as mainstream feature ● Massive out-of-core data sets ● Scale vertically & horizontally ● Maintain MapReduce abstraction Ismail El Helw, Rutger Hofman, Henri Bal [HPDC’2014, SC’2014]

57. Glasswing Pipeline ● Overlaps computation, communication & disk access ● Supports multiple buffering levels

58. GPU optimizations ● Glasswing framework does: ● Memory management ● Some shared memory optimizations ● Data movement, data staging ● Programmer: ● Focusses on the map and reduce kernels (using OpenCL) ● Can do kernel optimizations if needed ● Coalescing, memory banks, etc.

59. Glasswing vs. Hadoop 64-node CPU Infiniband cluster

60. Glasswing vs. Hadoop 16-Node GTX480 GPU Cluster

61. Performance K-Means Hadoop Glasswing GPU Glasswing CPU GPMR compute

62. Compute Device Comparison

63. Lessons learned ● Scalable MapReduce framework combining coarse- grained and fine-grained parallelism ● Handles out-of-core data, sticks with MapReduce model ● Overlaps kernel executions with memory transfers, network communication and disk access ● Outperforms Hadoop by 1.2 – 4x on CPUs and 20 – 30x on GPUs

64. Discussion ● eScience applications help us to ● Understand the complexity of GPU programming ● Validate our ideas and software ● Give inspiration for new CS research ● Applications do need performance of GPUs ● Next in line: SKA, digital forensics, water management … ● GPU programming and optimization is too time- consuming for real applications

65. Discussion ● Dealing with performance ● GPU programs need many complex optimizations to obtain high performance ● Auto-tuning, performance modelling, machine learning, compiler-based reasoning ● How to deal with the tension between abstraction- level and control? ● New programming methodologies that allow a choice ● Frameworks that do separation of concerns

66. Questions?