N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER Comparing Cray Tasking and OpenMP NERSC User Services Overview of Cray Tasking Overview of OpenMP Directives Comparison Mixing OpenMP with MPI

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 2 Cray Shared-Memory Multiprocessors 1982: X-MP –1, 2, or 4 processors 1985: Cray-2 –4 processors 1988: Y-MP –8 processors “Cray for the 90s”: C90 –16 processors

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 3 Cray Tasking Microtasking –Parallel loops Macrotasking –Parallel regions Autotasking –Emphasis on “auto”

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 4 History of OpenMP Defined by OpenMP Architecture Review Board –Compaq/DEC, HP, IBM, Intel, SGI/Cray, Sun, etc. Endorsed by software and application vendors –Absoft, The Portland Group, etc. –ANSYS, Fluent, Livermore Software, NAG, etc. Fortran, C, and C++ bindings –Fortran specification released late 1997 –C and C++ specification released late 1998 See for complete details

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 5 Scope of OpenMP Shared-memory explicit parallelism –No automatic parallelism –Some research into extending to distributed-memory environments Directives, environment variables, runtime library routines

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 6 Cray Tasking Syntax sentinel directive [argument …] [, directive …] sentinel –cdir$ or !dir$ ( cmic$ and !mic$ are obsolete) Fixed format source –“normal” Fortran column rules Free format source –sentinel begins anywhere on a line, but must be the first text –sentinel followed by anything other than space implies continuation Multiple directives are comma-delimited –if a directive has required arguments, no other directives allowed

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 7 OpenMP Syntax prefix directive [clause[[,] clause]…] Fixed format source –prefix: !$OMP, C$OMP, or *$OMP –“normal” Fortran source rules Free format source –prefix: !$OMP –sentinel begins anywhere on a line, but must be the first text –“normal” Fortran source rules

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 8 Directive Correspondence Autotasking DOALL GUARD ENDGUARD PARALLEL ENDPARALLEL DOPARALLEL ENDDO CASE ENDCASE OpenMP PARALLEL DO CRITICAL END CRITICAL PARALLEL END PARALLEL DO END DO SECTIONS END SECTIONS

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 9 Simple Parallel Loop !MIC$ DOALL PRIVATE(I) SHARED(X,Y,Z) !$OMP PARALLEL DO DO I = 1, N Z(I)=LOG(SIN(X(I))**2+COS(Y(I))**4) END DO !$OMP END PARALLEL DO

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 10 Directive Correspondence Autotasking DOALL GUARD ENDGUARD PARALLEL ENDPARALLEL DOPARALLEL ENDDO CASE ENDCASE OpenMP PARALLEL DO CRITICAL END CRITICAL PARALLEL END PARALLEL DO END DO SECTIONS END SECTIONS

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 11 Parallel Loop With Critical Section !MIC$ DOALL PRIVATE(I) SHARED(X,Y,Z,T) !$OMP PARALLEL DO DO I = 1, N Z(I) = LOG(SIN(X(I))**2 + COS(Y(I))**4) !MIC$ GUARD !$OMP CRITICAL T = T + Z(I) !MIC$ ENDGUARD !$OMP ENDCRITICAL END DO !$OMP END PARALLEL DO

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 12 Directive Correspondence Autotasking DOALL GUARD ENDGUARD PARALLEL ENDPARALLEL DOPARALLEL ENDDO CASE ENDCASE OpenMP PARALLEL DO CRITICAL END CRITICAL PARALLEL END PARALLEL DO END DO SECTIONS END SECTIONS

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 13 Parallel Regions !MIC$ PARALLEL PRIVATE(X) SHARED(Y) !$OMP PARALLEL PRIVATE(X) !$OMP DO X = 3.0 !MIC$ GUARD !$OMP CRITICAL Y = Y !MIC$ ENDGUARD !$OMP ENDCRITICAL !MIC$ ENDPARALLEL !$OMP END DO !$OMP END PARALLEL

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 14 Directive Correspondence Autotasking DOALL GUARD ENDGUARD PARALLEL ENDPARALLEL DOPARALLEL ENDDO CASE ENDCASE OpenMP PARALLEL DO CRITICAL END CRITICAL PARALLEL END PARALLEL DO END DO SECTIONS END SECTIONS

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 15 Generalized Parallel Loop SUM = 0.0 !MIC$ PARALLEL PRIVATE(XSUM,I) SHARED(SUM,A,N) XSUM = 0.0 !MIC$ DOPARALLEL DO I = 1, N XSUM = XSUM + A(I) ENDDO !MIC$ GUARD SUM = SUM + XSUM !MIC$ ENDGUARD !MIC$ ENDDO !MIC$ ENDPARALLEL

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 16 Generalized Parallel Loop (cont.) SUM = 0.0 !$OMP PARALLEL PRIVATE(XSUM) XSUM = 0.0 !$OMP DO DO I = 1, N XSUM = XSUM + A(I) ENDDO !$OMP CRITICAL SUM = SUM + XSUM !$OMP END CRITICAL !$OMP END DO !$OMP END PARALLEL

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 17 Directive Correspondence Autotasking DOALL GUARD ENDGUARD PARALLEL ENDPARALLEL DOPARALLEL ENDDO CASE ENDCASE OpenMP PARALLEL DO CRITICAL END CRITICAL PARALLEL END PARALLEL DO END DO SECTIONS END SECTIONS

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 18 Parallel Blocks !MIC$ PARALLEL !$OMP PARALLEL !MIC$ CASE !$OMP SECTIONS !$OMP SECTION CALL SUBA !MIC$ CASE !$OMP SECTION CALL SUBB !MIC$ ENDCASE !$OMP END SECTIONS !MIC$ ENDPARALLEL !$OMP END SECTIONS

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 19 Mixing OpenMP and MPI INCLUDE ’mpi.h' CALL MPI_INIT(IERR) CALL MPI_COMM_RANK(MPI_COMM_WORLD,MYID,IERR) CALL MPI_COMM_SIZE(MPI_COMM_WORLD,NP,IERR) PRINT *,'PROCESS ',MYID,' OF ',NP,' IS ALIVE' !$OMP PARALLEL PRIVATE(NTHREADS, TID) TID = OMP_GET_THREAD_NUM() PRINT *,'HI;THREAD=',TID,' PROCESS=',MYID IF (TID.EQ. 0) THEN NTHREADS = OMP_GET_NUM_THREADS() PRINT *,'NTHREADS=',NTHREADS,’MYID=',MYID END IF !$OMP END PARALLEL CALL MPI_FINALIZE(IERR) END

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER June, 2000Cray Tasking vs. OpenMP - User Services Group 20 Mixing OpenMP and MPI (cont.) % mpxlf_r smp.f -qsmp ** hi === End of Compilation 1 === Compilation successful for file smp.f. % setenv XLSMPOPTS "parthds=2" %./a.out -nodes 2 Process 1 of 2 is alive Process 0 of 2 is alive HI;THREAD=0 PROCESS=0 NTHREADS=2 MYID=0 HI;THREAD=1 PROCESS=0 HI;THREAD=0 PROCESS=1 NTHREADS=2 MYID=1 HI;THREAD=1 PROCESS=1