1. Code Tuning and Parallelization on Boston Universitys Scientific Computing Facility Doug Sondak sondak@bu.edu Boston University Scientific Computing and Visualization

2. Outline Introduction Timing Profiling Cache Tuning Timing/profiling exercise Parallelization

3. Introduction Tuning –Where is most time being used? –How to speed it up Often as much art as science Parallelization –After serial tuning, try parallel processing –MPI –OpenMP

4. Timing

5. When tuning/parallelizing a code, need to assess effectiveness of your efforts Can time whole code and/or specific sections Some types of timers –unix time command –function/subroutine calls –profiler

6. CPU or Wall-Clock Time? both are useful for parallel runs, really want wall-clock time, since CPU time will be about the same or even increase as number of procs. is increased CPU time doesnt account for wait time wall-clock time may not be accurate if sharing processors –wall-clock timings should always be performed in batch mode

7. Unix Time Command easiest way to time code simply type time before your run command output differs between c-type shells (cshell, tcshell) and Bourne-type shells (bsh, bash, ksh)

8. tcsh results Unix time Command (contd) twister:~ % time mycode 1.570u 0.010s 0:01.77 89.2% 75+1450k 0+0io 64pf+0w user CPU time (s) system CPU time (s) wall-clock time (s) (u+s)/wc avg. shared + unshared text space input + output operations page faults + no. times proc. was swapped

9. Unix Time Command (3) bsh results $ time mycode Real 1.62 User 1.57 System 0.03 wall-clock time (s) user CPU time (s) system CPU time (s)

10. Function/Subroutine Calls often need to time part of code timers can be inserted in source code language-dependent

11. cpu_time intrinsic subroutine in Fortran returns user CPU time (in seconds) –no system time is included 0.01 sec. resolution on p-series real :: t1, t2 call cpu_time(t1)... do stuff to be timed... call cpu_time(t2) print*, 'CPU time = ', t2-t1, ' sec.'

12. system_clock intrinsic subroutine in Fortran good for measuring wall-clock time on p-series: –resolution is 0.01 sec. –max. time is 24 hr.

13. system_clock (contd) t1 and t2 are tic counts count_rate is optional argument containing tics/sec. integer :: t1, t2, count_rate call system_clock(t1, count_rate)... do stuff to be timed... call system_clock(t2) print*,'wall-clock time = ', & real(t2-t1)/real(count_rate), sec

14. times can be called from C to obtain CPU time 0.01 sec. resolution on p-series can also get system time with tms_stime #include void main(){ int tics_per_sec; float tic1, tic2; struct tms timedat; tics_per_sec = sysconf(_SC_CLK_TCK); times(&timedat); tic1 = timedat.tms_utime; … do stuff to be timed … times(&timedat); tic2 = timedat.tms_utime; printf("CPU time = %5.2f\n", (float)(tic2-tic1)/(float)tics_per_sec); }

15. gettimeofday can be called from C to obtain wall-clock time sec resolution on p-series #include void main(){ struct timeval t; double t1, t2; gettimeofday(&t, NULL); t1 = t.tv_sec + 1.0e-6*t.tv_usec; … do stuff to be timed … gettimeofday(&t, NULL); t2 = t.tv_sec + 1.0e-6*t.tv_usec; printf(wall-clock time = %5.3f\n", t2-t1); }

16. MPI_Wtime convenient wall-clock timer for MPI codes sec resolution on p-series

17. MPI_Wtime (contd) Fortran C double precision t1, t2 t1 = mpi_wtime()... do stuff to be timed... t2 = mpi_wtime() print*,'wall-clock time = ', t2-t1 double t1, t2; t1 = MPI_Wtime();... do stuff to be timed... t2 = MPI_Wtime(); printf(wall-clock time = %5.3f\n,t2-t1);

18. omp_get_wtime convenient wall-clock timer for OpenMP codes resolution available by calling omp_get_wtick() 0.01 sec. resolution on p-series

19. omp_get_wtime (contd) Fortran C double precision t1, t2, omp_get_wtime t1 = omp_get_wtime()... do stuff to be timed... t2 = omp_get_wtime() print*,'wall-clock time = ', t2-t1 double t1, t2; t1 = omp_get_wtime();... do stuff to be timed... t2 = omp_get_wtime(); printf(wall-clock time = %5.3f\n,t2-t1);

20. Timer Summary CPUWall Fortrancpu_timesystem_clock Ctimesgettimeofday MPIMPI_Wtime OpenMPomp_get_time

21. Profiling

22. Profilers profile tells you how much time is spent in each routine various profilers available, e.g. –gprof (GNU) –pgprof (Portland Group) –Xprofiler (AIX)

23. gprof compile with -pg file gmon.out will be created when you run gprof executable > myprof for multiple procs. (MPI), copy or link gmon.out.n to gmon.out, then run gprof

24. gprof (contd) ngranularity: Each sample hit covers 4 bytes. Time: 435.04 seconds called/total parents index %time self descendents called+self name index called/total children 0.00 340.50 1/1.__start [2] [1] 78.3 0.00 340.50 1.main [1] 2.12 319.50 10/10.contrl [3] 0.04 7.30 10/10.force [34] 0.00 5.27 1/1.initia [40] 0.56 3.43 1/1.plot3da [49] 0.00 1.27 1/1.data [73]

25. gprof (3) ngranularity: Each sample hit covers 4 bytes. Time: 435.04 seconds % cumulative self self total time seconds seconds calls ms/call ms/call name 20.5 89.17 89.17 10 8917.00 10918.00.conduct [5] 7.6 122.34 33.17 323 102.69 102.69.getxyz [8] 7.5 154.77 32.43.__mcount [9] 7.2 186.16 31.39 189880 0.17 0.17.btri [10] 7.2 217.33 31.17.kickpipes [12] 5.1 239.58 22.25 309895200 0.00 0.00.rmnmod [16] 2.3 249.67 10.09 269 37.51 37.51.getq [24]

26. pgprof compile with Portland Group compiler –pgf95 (pgf90, etc.) –pgcc – –Mprof=func similar to –pg –run code pgprof –exe executable pops up window with flat profile

27. pgprof (contd)

28. pgprof (3) line-level profiling – –Mprof=line optimizer will re-order lines –profiler will lump lines in some loops or other constructs –may want to compile without optimization, may not in flat profile, double-click on function

29. pgprof (4)

30. xprofiler AIX (twister) has a graphical interface to gprof compile with -g -pg -Ox –Ox represents whatever level of optimization youre using (e.g., O5) run code –produces gmon.out file type xprofiler mycode –mycode is your code run comamnd

31. xprofiler (contd)

32. xprofiler (3) filled boxes represent functions or subroutines fences represent libraries left-click a box to get function name and timing information right-click on box to get source code or other information

33. xprofiler (4) can also get same profiles as from gprof by using menus –report flat profile –report call graph profile

34. Cache

35. Cache is a small chunk of fast memory between the main memory and the registers secondary cache registers primary cache main memory

36. Cache (contd) Variables are moved from main memory to cache in lines –L1 cache line sizes on our machines Opteron (katana cluster) 64 bytes Power4 (p-series) 128 bytes PPC440 (Blue Gene) 32 bytes Pentium III (linux cluster) 32 bytes If variables are used repeatedly, code will run faster since cache memory is much faster than main memory

37. Cache (contd) Why not just make the main memory out of the same stuff as cache? –Expensive –Runs hot –This was actually done in Cray computers Liquid cooling system

38. Cache (contd) Cache hit –Required variable is in cache Cache miss –Required variable not in cache –If cache is full, something else must be thrown out (sent back to main memory) to make room –Want to minimize number of cache misses

39. Cache example … x[0] x[1] x[2] x[3] x[4] x[5] x[6] x[7] x[8] x[9] Main memory mini cache holds 2 lines, 4 words each for(i=0; i<10; i++) x[i] = i a b …

40. Cache example (contd) … x[0] x[1] x[2] x[3] x[4] x[5] x[6] x[7] x[8] x[9] for(i=0; i<10; i++) x[i] = i x[0] x[1] x[2] x[3] We will ignore i for simplicity need x[0], not in cache cache miss load line from memory into cache next 3 loop indices result in cache hits a b …

41. Cache example (contd) … x[0] x[1] x[2] x[3] x[4] x[5] x[6] x[7] x[8] x[9] for(i=0; i<10; i++) x[i] = i x[0] x[1] x[2] x[3] need x[4], not in cache cache miss load line from memory into cache next 3 loop indices result in cache hits x[4] x[5] x[6] x[7] a b …

42. Cache example (contd) … x[0] x[1] x[2] x[3] x[4] x[5] x[6] x[7] x[8] x[9] for(i==0; i<10; i++) x[i] = i x[8] x[9] a b need x[8], not in cache cache miss load line from memory into cache no room in cache! replace old line x[4] x[5] x[6] x[7] a b …

43. Cache (contd) Contiguous access is important In C, multidimensional array is stored in memory as a[0][0] a[0][1] a[0][2] …

44. Cache (contd) In Fortran and Matlab, multidimensional array is stored the opposite way: a(1,1) a(2,1) a(3,1) …

45. Cache (contd) Rule: Always order your loops appropriately –will usually be taken care of by optimizer –suggestion: dont rely on optimizer! for(i=0; i<N; i++){ for(j=0; j<N; j++){ a[i][j] = 1.0; } do j = 1, n do i = 1, n a(i,j) = 1.0 enddo CFortran

46. Tuning Tips

47. Some of these tips will be taken care of by compiler optimization –Its best to do them yourself, since compilers vary

48. Tuning Tips (contd) Access arrays in contiguous order –For multi-dimensional arrays, rightmost index varies fastest for C and C++, leftmost for Fortran and Matlab Bad Good for(i=0; i<N; i++){ for(j=0; j<N; j++{ a[i][j] = 1.0; } for(j=0; j<N; j++){ for(i=0; i<N; i++{ a[i][j] = 1.0; }

49. Tuning Tips (3) Eliminate redundant operations in loops Bad: Good: for(i=0; i<N; i++){ x = 10; } … x = 10; for(i=0; i<N; i++){ } …

50. Tuning Tips (4) Eliminate if statements within loops They may inhibit pipelining for(i=0; i<N; i++){ if(i==0) perform i=0 calculations else perform i>0 calculations }

51. Tuning Tips (5) Better way perform i=0 calculations for(i=1; i<N; i++){ perform i>0 calculations }

52. Tuning Tips (6) Divides cost far more than multiplies or adds –Often order of magnitude difference! Bad: Good: for(i=0; i<N; i++) x[i] = y[i]/scalarval; qs = 1.0/scalarval; for(i=0; i<N; i++) x[i] = y[i]*qs ;

53. Tuning Tips (7) There is overhead associated with a function call Bad: Good: for(i=0; i<N; i++) myfunc(i); myfunc ( ); void myfunc(x){ for(int i=0; i<N; i++){ do stuff }

54. Tuning Tips (8) There is overhead associated with a function call Minimize calls to math functions Bad: Good: for(i=0; i<N; i++) z[i] = log(x[i]) * log(y[i]); for(i=0; i<N; i++){ z[i] = log(x[i] + y[i]);

55. Tuning Tips (9) There is overhead associated with a function call recasting may be costlier than you think Bad: Good: sum = 0.0; for(i=0; i<N; i++) sum += (float) i isum = 0; for(i=0; i<N; i++) isum += i; sum = (float) isum

56. Parallelization

57. Introduction MPI & OpenMP Performance metrics Amdahls Law

58. Introduction Divide and conquer! –divide operations among many processors –perform operations simultaneously –if serial run takes 10 hours and we hit the problem with 5000 processors, it should take about 7 seconds to complete, right? not so easy, of course

59. Introduction (contd) problem – some calculations depend upon previous calculations –cant be performed simultaneously –sometimes tied to the physics of the problem, e.g., time evolution of a system want to maximize amount of parallel code –occasionally easy –usually requires some work

60. Introduction (3) method used for parallelization may depend on hardware proc 0 proc 1 proc 2 proc 3 mem 0 mem 1 mem 2 mem 3 distributed memory proc 0 proc 1 proc 2 proc 3 mem shared memory proc 0 proc 1 proc 2 proc 3 mem 0 mem 1 mixed memory

61. Introduction (4) distributed memory –e.g., katana, Blue Gene –each processor has own address space –if one processor needs data from another processor, must be explicitly passed shared memory –e.g., p-series IBM machines –common address space –no message passing required

62. Introduction (5) MPI –for both distributed and shared memory –portable –freely downloadable OpenMP –shared memory only –must be supported by compiler (most do) –usually easier than MPI –can be implemented incrementally

63. MPI Computational domain is typically decomposed into regions –One region assigned to each processor Separate copy of program runs on each processor

64. MPI (contd) Discretized domain to solve flow over airfoil System of coupled PDEs solved at each point

65. MPI (3) Decomposed domain for 4 processors

66. MPI (4) Since points depend on adjacent points, must transfer information after each iteration This is done with explicit calls in the source code

67. MPI (5) Diminishing returns –Sending messages can get expensive –Want to maximize ratio of computation to communication

68. OpenMP Usually loop-level parallelization An OpenMP directive is placed in the source code before the loop –Assigns subset of loop indices to each processor –No message passing since each processor can see the whole domain for(i=0; i<N; i++){ do lots of stuff }

69. OpenMP (contd) Cant guarantee order of operations for(i = 0; i < 7; i++) a[i] = 1; for(i = 1; i < 7; i++) a[i] = 2*a[i-1]; ia[i] (serial)a[i] (parallel) 011 122 244 388 4162 5324 6648 Proc. 0 Proc. 1 Parallelize this loop on 2 processors Example of how to do it wrong!

70. Quantify performance Two common methods –parallel speedup –parallel efficiency

71. Parallel Speedup S n = parallel speedup n = number of processors T 1 = time on 1 processor T n = time on n processors

72. Parallel Speedup (2)

73. Parallel Efficiency n = parallel efficiency T 1 = time on 1 processor T n = time on n processors n = number of processors

74. Parallel Efficiency (2)

75. Parallel Efficiency (3) What is a reasonable level of parallel efficiency? depends on –how much CPU time you have available –when the paper is due can think of (1- ) as wasted CPU time my personal rule of thumb ~60%

76. Parallel Efficiency (4) Superlinear speedup –parallel efficiency > 1.0 –sometimes quoted in the literature –generally attributed to cache issues subdomains fit entirely in cache, entire domain does not this is very problem-dependent be suspicious!

77. Amdahls Law let fraction of code that can execute in parallel be denoted p let fraction of code that must execute serially be denoted s let T = time, n = number of processors

78. Amdahls Law (2) Noting that p = (1-s) parallel speedup is (dont confuse S n with s ) Amdahls Law

79. Amdahls Law (3) can also be expressed as parallel efficiency by dividing by n Amdahls Law

80. suppose s = 0 ; => linear speedup Amdahls Law (4)

81. Amdahls Law (5) suppose s = 1 ; => no speedup

82. Amdahls Law (6)

83. Amdahls Law (7) Should we despair? –No! –bigger machines bigger computations smaller value of s if you want to run on a large number of processors, try to minimize s

84. Recommendations

85. Add timers to your code –As you make changes and/or run new cases, they may give you an indication of a problem Profile your code –Sometimes results are surprising –Review tuning tips –See if you can speed up functions that are consuming the most time Try highest levels of compiler optimization

86. Recommendations (contd) Once youre comfortable that youre getting reasonable serial performance, parallelize If portability is an issue, MPI is a good choice If youll always be running on a shared-memory machine (e.g., multicore PC), consider OpenMP For parallel code, plot parallel efficiency vs. number of processors –Choose appropriate number of processors