Experiences parallelising the mixed C-Fortran Sussix BPM post-processor H. Renshall, BE Dept associate, Jan 2012 Using appendix material from CERN-ATS-Note MD (July 2011) 23/01/20121 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX

23/01/2012 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX 2 The Problem: SUSSIX is a FORTRAN program for the post processing of turn-by-turn BeamPositionMonitor data, which computes the frequency, amplitude, and phase of tunes and resonant lines to a high degree of precision through the use of an interpolated FFT. Analysis of such data represents a vital component of many linear and non-linear dynamics measurements. For analysis of LHC BPM data a specific version sussix4drive, run through the C steering code Drive God lin, has been implemented in the CCC by the beta-beating team. Analysis of all LHC BPMs, however, represents a major real time computational bottleneck in the control room, which has prevented truly on-line study of the BPM data. In response to this limitation an effort has been underway to decrease the real computational time, with a factor of 10 as the target, of the C and Fortran codes by parallelizing them.

23/01/2012 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX 3 Solutions considered: Since the application is run on dedicated servers in the CCC the obvious technique is to profit from the current multi-core hardware: 24/48 cores are now typical. The first idea was to use a parallelised FFT from the NAG fsl6i2dcl library for SMP and multicore together with the intel 64-bit Fortran compiler and the intel maths kernel library recommended by NAG. As a learning exercise various NAG installation validation examples of enhanced routines were run, including multi-dimensional FFTs, and all took about the same real time, but increasing user cpu time, as the number of cores was increased on a fairly idle 16-core lxplus machine. Not surprising since the examples only take msec, comparable to the overhead to launch a new thread.

23/01/2012 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX 4 The Sussix application calls cfft (D704 in the CERN program library) which maps onto NAG c06ecf which has not yet been enhanced. c06ecf was 10% slower than cfft on a simple test case giving the same numerical results, probably due to extra housekeeping and extra numerical controls. At the same time profiling the Sussix application (with gprof) showed that only 7.5% of the total cpu time was spent in cfft and with less than 10 msec per individual call hence one could expect little or no real-time speedup by using a parallelised version. The profile showed that 70% of the cpu time was spent in a function calcr searching for the maxima of the fourier spectra with large numbers of executions of a compact reverse inner loop over the number of turns of bpm data.

23/01/2012 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX 5 This inverse loop over maxd, the number of LHC turns measured by an individual bpm, could not be improved. In a real case maxd is typically 1000 and this loop is executed 10 million times: double complex zp,zpp,zv zpp=zp(maxd) do np=maxd-1,1, -1 zpp=zpp*zv+zp(np) enddo It was decided to try and parallelise using, like NAG, the OPENMP implementation supported by the Intel compiler and examining the granularity revealed that the highest level of independent code execution was over the processing of individual BPM data.

23/01/2012 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX 6 The pure FORTRAN offline version was parallelised first by adding OPENMP parallelisation directives round the main bpm loop. Each bpm data is in a separate file: !$OMP PARALLEL DO PRIVATE(n,iunit,filename,nturn) !$OMP& SHARED (isix,ntot,iana,iconv,nt1,nt2,narm,istune,etune,tunex, tuney,tunez,nsus,idam,ntwix,ir,imeth,nrc,eps,nline, lr,mr,kr,idamx,ifin,isme,iusme,inv,iinv,icf,iicf) do n=1,ntot ! Parallel loop over all bpm (typically 500) call datspe(iunit,idam,ir,nt1,nt2,nturn,imeth,narm,iana) call ordres(eps,narm,nrc,idam,n,nturn) enddo !$OMP END PARALLEL DO In addition !$OMP THREADPRIVATE directives were added for all non-shareable variables in the called subroutine trees.

23/01/2012 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX 7 This gave good scaling up to 10 cores on a non-dedicated 16-core lxplus machine (reported at the 24 th ICE section meeting of 2011) so was worth extending to the target mixed C and Fortran version to be run in the control room. The bpm data is read into memory from a single file then a bpm loop is called from C code with a different but similar OPENMP syntax to give the same scaling result: #pragma omp parallel private(i,ii,ij,kk) #pragma omp for for (i=pickstart; i<=maxcounthv ; i++){ sussix4drivenoise_(&doubleToSend[0], &tune[0], &amplitude[0]) #pragma omp critical /* here I/O C-code in the loop needing sequential execution */ } The Fortran datspe and ordres call trees were unchanged.

23/01/2012 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX 8 The OPENMP directives multi-thread the code and the threads then map onto physical CPUs in a multi-core machine. The run- time environment variable OMP_NUM_THREADS instructs OPENMP how many threads, hence cores, it can use for an execution and enables easy measurement of the scaling. Since the order of processing of individual BPMs is arbitrary the results file is post-processed by a unix sort as part of the application to give the same results as a non-parallel execution. A test case of real 1000 turn LHC BPM data, analysed to find 160 lines, was performed on a reserved 24 core machine cs- ccr-spareb7 in the CCC. A normal run of this test case takes about 50 seconds on this machine. The observed wall-time speedup of C-Fortran Sussix as a function of the number of cores (from E. Maclean) is shown on the final slide.

23/01/2012 BE-NAG Meeting: Parallelisation of C-Fortran SUSSIX 9 About a factor of 10 improvement in the real computation time has been realised for this test case saturating at 12 cores, probably due to memory bandwidth limits. For the study of amplitude detuning reported in CERN-ATS- Note the parallelized C-Fortran SUSSIX was utilised within the beta- beat GUI and the target tenfold real-time reduction was verified in practice. This technique could be of interest to other applications.