1 Typical performance bottlenecks and how they can be found Bert Wesarg ZIH, Technische Universität Dresden

2 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: –Finding load imbalances in OpenMP codes Case II: –Finding communication and computation imbalances in MPI codes Outline

3 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: Sparse Matrix Vector Multiplication foreach row r in A y[r.x] = 0 foreach non-zero element e in row y[r.x] += e.value * x[e.y]

4 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Naïve OpenMP Algorithm: Distributes the rows of A evenly across the threads in the parallel region The distribution of the non-zero elements may influence the load balance in the parallel application Case I: Sparse Matrix Vector Multiplication #pragma omp parallel for foreach row r in A y[r.x] = 0 foreach non-zero element e in row y[r.x] += e.value * x[e.y]

5 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Measuring the static OpenMP application Case I: Load imbalances in OpenMP codes % cd ~/Bottlenecks/smxv % make PREP=scorep scorep gcc -fopenmp -DLITTLE_ENDIAN \ -DFUNCTION_INC='"y_Ax-omp.inc.c"' -DFUNCTION=y_Ax_omp \ -o smxv-omp smxv.c -lm scorep gcc -fopenmp -DLITTLE_ENDIAN \ -DFUNCTION_INC='"y_Ax-omp-dynamic.inc.c"‘ \ -DFUNCTION=y_Ax_omp_dynamic -o smxv-omp-dynamic smxv.c -lm % OMP_NUM_THREADS=8 scan –t./smxv-omp yax_large.bin

6 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Two metrics which indicate load imbalances: –Time spent in OpenMP barriers –Computational imbalance Open prepared measurement on the LiveDVD with Cube Case I: Load imbalances in OpenMP codes: Profile % cube ~/Bottlenecks/smxv/scorep_smxv-omp_large/trace.cubex [CUBE GUI showing trace analysis report]

7 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: Time spent in OpenMP barriers These threads spent up to 20% of there running time in the barrier

8 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: Computational imbalance Master thread does 66% of the work

9 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Improved OpenMP Algorithm Distributes the rows of A dynamically across the threads in the parallel region Case I: Sparse Matrix Vector Multiplication #pragma omp parallel for schedule(dynamic,1000) foreach row r in A y[r.x] = 0 foreach non-zero element e in row y[r.x] += e.value * x[e.y]

10 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Two metrics which indicate load imbalances: –Time spent in OpenMP barriers –Computational imbalance Open prepared measurement on the LiveDVD with Cube: Case I: Profile Analysis % cube ~/Bottlenecks/smxv/scorep_smxv-omp-dynamic_large/trace.cubex [CUBE GUI showing trace analysis report]

11 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: Time spent in OpenMP barriers All threads spent similar time in the barrier

12 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: Computational imbalance Threads do nearly equal work

13 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Open prepared measurement on the LiveDVD with Vampir: Case I: Trace Comparison % vampir ~/Bottlenecks/smxv/scorep_smxv-omp_large/traces.otf2 [Vampir GUI showing trace]

14 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: Time spent in OpenMP barriers Improved runtime Less time in OpenMP barrier

15 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: Computational imbalance Great imbalance for time spent in computational code

16 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case I: –Finding load imbalances in OpenMP codes Case II: –Finding communication and computation imbalances in MPI codes Outline

17 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Case II: Heat Conduction Simulation

18 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Application uses MPI for boundary exchange and OpenMP for computation Simulation grid is distributed across MPI ranks Case II: Heat Conduction Simulation

19 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Ranks need to exchange boundaries before next iteration step Case II: Heat Conduction Simulation

20 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering MPI Algorithm Building and measuring the heat conduction application: Open prepared measurement on the LiveDVD with Cube Case II: Profile Analysis foreach step in [1:nsteps] exchangeBoundaries computeHeatConduction % cd ~/Bottlenecks/heat % make PREP=‘scorep --user’ [... make output...] % scan –t mpirun –np 16./heat-MPI % cube ~/Bottlenecks/heat/scorep_heat-MPI_16/trace.cubex [CUBE GUI showing trace analysis report]

21 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Step 1: Compute heat in the area which is communicated to your neighbors Case II: Hide MPI communication with computation

22 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Step 2: Start communicating boundaries with your neighbors Case II: Hide MPI communication with computation

23 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Step 3: Compute heat in the interior area Case II: Hide MPI communication with computation

24 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Improved MPI Algorithm Measuring the improved heat conduction application: Open prepared measurement on the LiveDVD with Cube Case II: Profile Analysis foreach step in [1:nsteps] computeHeatConductionInBoundaries startBoundaryExchange computeHeatConductionInInterior waitForCompletionOfBoundaryExchange % scan –t mpirun –np 16./heat-MPI-overlap % cube ~/Bottlenecks/heat/scorep_heat-MPI-overlap_16/trace.cubex [CUBE GUI showing trace analysis report]

25 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Open prepared measurement on the LiveDVD with Vampir: Case II: Trace Comparison % vampir ~/Bottlenecks/heat/scorep_heat-MPI_16/traces.otf2 [Vampir GUI showing trace]

26 SC’13: Hands-on Practical Hybrid Parallel Application Performance Engineering Thanks to Dirk Schmidl, RWTH Aachen, for providing the sparse matrix vector multiplication code Acknowledgments