1. gLite MPI Job Amina KHEDIMI (a.khedimi@dtri.cerist.dz) CERIST
27/05/2018
gLite MPI Job
Amina KHEDIMI
CERIST
Africa 6 -Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting
Rabat, June 8, 2011
Rabat

2. Outline MPI and its implementations Basic Structures of MPI Programs
27/05/2018
MPI and its implementations
Basic Structures of MPI Programs
Structure of a MPI job in the Grid without mpi-start
Wrapper script and Hooks for mpi-start
Structure of a MPI job in the Grid with mpi-start
Defining the job and executable
Running the MPI job
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting
Rabat

3. MPI and its implementations
The Message Passing Interface (MPI) is a de-facto standard for writing parallel application.
There are two versions of MPI, MPI-1 and MPI-2;
Two implementations of MPI-1:
LAM;
MPICH.
Two implementations of MPI-2:
OpenMPI;
MPICH2.
Each version has different implementations:
Some implementation are hardware related:
E.g.: InfiniBand networks require MVAPICH v.1 or v.2 libraries.
Individual sites may chose to support only a subset of these implementations, or none at all.

4. Goals of the MPI standard
MPI prime goals are:
To provide source-code portability
To allow efficient implementation across a range of architectures
A great deal of functionality
the user need not cope with communication failures. Such failures are dealt with by the underlying communication subsystem.
a “master” node starts some processes “slaves” by establishing SSH sessions
all processes can share a common workspace and/or exchange data based on send() and receive() routines

5. A bit of history ...
The Message Passing Interface (MPI) is a standard developed by the Message Passing Interface Forum (MPIF). It specifies a portable interface APIs for writing message-passing programs in Fortran, C and C++
MPIF ( with the participation of more than 40 organizations, started working on the standard in 1992.
The first draft (Version 1.0), which was published in 1994, was strongly influenced by the work at the IBM T. J. Watson Research Center.
MPIF has further enhanced the first version to develop a second version (MPI-2) in The latest release of the first version (Version 1.2) is offered as an update to the previous release and is contained in the MPI-2 document.

6. Basic Structures of MPI Programs
Header files
Initializing MPI
MPI Communicator
MPI Function format
Communicator Size
Process Rank
Finalizing MPI

7. 1.Header files
All sub-programs that contains calls to MPI subroutine MUST include the MPI HEADER file C:
#include <mpi.h>
Fortran:
#include ‘mpi.h’
The header file contains definitions of MPI constants, MPI types and functions

8. 2.Initializing MPI
The first MPI routine called in any MPI program must be the initialisation routine MPI_INIT.
Every MPI program must call this routine once, before any other MPI routines.
Making multiple calls to MPI_INIT is erroneous.
The C version of the routine accepts argc and argv as arguments : int MPI_Init(int &argc, char &argv);
The Fortran version takes no arguments other than the error code: MPI_INIT(IERROR)

9. 3.MPI Communicator
The Communicator is a variable identifying a group of processes that are allowed to communicate with each other
There is a default communicator MPI_COMM_WORLD which identify the group of all the processes.
The processes are ordered and numbered consecutively from 0 (in both Fortran and C), the number of each process being known as its rank
The rank identifies each process within the communicator.
The predefined communicator MPI_COMM_WORLD for 7 processes The numbers indicate the ranks of each process.

10. How many processes are associated with a communicator ?
5.Communicator Size
How many processes are associated with a communicator ?
C :
MPI_Comm_size (MPI_Comm comm, int *SIZE);
Fortran :
INTEGER COMM, SIZE, IERR
CALL MPI_COMM_SIZE (COMM,SIZE,IERR)
Output SIZE

11. 6.Process Rank
What is the ID of a process in a group ? C:
MPI_Comm_rank (MPI_Comm comm, int *RANK);
Fortran:
INTEGER COMM, RANK, IERR
CALL MPI_COMM_RANK (COMM, RANK, IERR)
Output : RANK

12. 7.Finalizing MPI
An MPI program should call the MPI routine MPI_FINALIZE when all communications have completed. This routine cleans up all MPI data-structures, etc.
Once this routine has been called, no other calls can be made to MPI routines
Finalizing the MPI environment C:
int MPI_Finalize ();
Fortran:
INTEGER IERR
CALL MPI_FINALIZE (IERR)

13. Get information from the Grid
Find the sites that support MPICH and MPICH2
~]$ lcg-info --vo eumed --list-ce --query 'Tag=MPICH‘
CE: ce-grid.narss.sci.eg:8443/cream-pbs-eumed
CE: ce-grid.obspm.fr:2119/jobmanager-pbs-eumed
CE: ce0.m3pec.u-bordeaux1.fr:2119/jobmanager-pbs-eumed
CE: ce01.grid.cynet.ac.cy:8443/cream-pbs-eumed
CE: cream-ce-grid.obspm.fr:8443/cream-pbs-eumed
~]$ lcg-info --vo eumed --list-ce --query 'Tag=MPICH2‘
CE: ce-grid.narss.sci.eg:8443/cream-pbs-eumed
CE: ce-grid.obspm.fr:2119/jobmanager-pbs-eumed
CE: ce01.grid.arn.dz:8443/cream-pbs-eumed
CE: ce-02.roma3.infn.it:8443/cream-pbs-eumed
CE: ce01.grid.hiast.edu.sy:8443/cream-pbs-eumed
CE: ce01.grid.um.edu.mt:8443/cream-pbs-eumed
CE: ce03.grid.arn.dz:8443/cream-pbs-eumed
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting 13

14. Get information from the Grid
27/05/2018
~]$ lcg-info --vo eumed --list-ce --query 'Tag=MPICH' --attrs 'CE,FreeCPUs,TotalCPUs’
- CE: ce-grid.narss.sci.eg:8443/cream-pbs-eumed
- CE ce-grid.narss.sci.eg:8443/cream-pbs-eumed
- FreeCPUs
- TotalCPUs
- CE: ce-grid.obspm.fr:2119/jobmanager-pbs-eumed
- CE ce-grid.obspm.fr:2119/jobmanager-pbs-eumed
- FreeCPUs
- TotalCPUs
- CE: ce0.m3pec.u-bordeaux1.fr:2119/jobmanager-pbs-eumed
- CE ce0.m3pec.u-bordeaux1.fr:2119/jobmanager-pbs-eumed
- FreeCPUs
- TotalCPUs
- CE: ce01.grid.cynet.ac.cy:8443/cream-pbs-eumed
- CE ce01.grid.cynet.ac.cy:8443/cream-pbs-eumed
- FreeCPUs
- TotalCPUs
- CE: cream-ce-grid.obspm.fr:8443/cream-pbs-eumed
- CE cream-ce-grid.obspm.fr:8443/cream-pbs-eumed
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting 14
Rabat

15. Get information from the Grid
Find the sites who have shared home MPI directory
What happens when there is a MPI shared directory in the site?
The file is compiled in one WN, then the file read by the other WNs in the same site.
What happens when is not MPI shared directory?
The file is compiled in one WN, then the file must be copied to the others WNs.
Tag=MPI-start
~]$ lcg-info --vo eumed --list-ce --query 'Tag=MPI_SHARED_HOME‘
CE: ce-grid.narss.sci.eg:8443/cream-pbs-eumed
CE: ce-grid.obspm.fr:2119/jobmanager-pbs-eumed
CE: ce0.m3pec.u-bordeaux1.fr:2119/jobmanager-pbs-eumed
CE: ce01.grid.arn.dz:8443/cream-pbs-eumed
CE: ce2.cnrst.magrid.ma:8443/cream-pbs-eumed
CE: ce01.grid.um.edu.mt:8443/cream-pbs-eumed
CE: ce03.grid.arn.dz:8443/cream-pbs-eumed
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

16. Requierements to submit jobs
JobType attribute in jdl-file needs to be set to “Normal” in order to run MPI jobs via MPI-START scripts.
"CPUNumber" needs to corresponds to the number of desired nodes.
"Executable" attribute has to point to wrapper script (mpi-start-wrapper.sh in this case).
"Arguments" are MPI binary and the MPI flavour that it uses.
Note that Executable + Arguments form the command line on the WN
MPI-START allows user defined extensions via hooks
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting 16

17. Structure of a MPI job in the Grid without mpi-start
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

18. mpi-start-wrapper.sh Wrapper script and Hooks for mpi-start
Users typically use a script that sets up paths and other internal settings to initiate the mpi-start processing. The following script (named "mpi-start-wrapper.sh") is generic and should not need to have significant modifications made to it.
The script first sets up the environment for the chosen flavor of MPI using environment variables supplied by the system administrator.
It then defines the executable, arguments, MPI flavor, and location of the hook scripts for mpi-start.
Lastly, the wrapper invokes mpi-start itself.
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

19. mpi-hooks.sh Wrapper script and Hooks for mpi-start
The user may write a script that is called before and after the MPI executable is run. The pre-hook can be used, for example, to compile the executable itself or download data. The post-hook can be used to analyze results or to save the results on the grid.
The pre- and post-hooks may be defined in separate files, but the names of the functions must be named exactly "pre_run_hook" and "post_run_hook“.
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

20. Structure of a MPI job in the Grid with mpi-start
mpi-start-wrapper.sh
mpi-Hooks.sh
JDL File
MPI file
JobType = "Normal";
CPUNumber = 16;
Executable = "mpi-start-wrapper.sh";
Arguments = "mpi-test MPICH2";
StdOutput = "mpi-test.out";
StdError = "mpi-test.err";
InputSandbox = {"mpi-start-wrapper.sh","mpi-hooks.sh","mpi-test.c"};
OutputSandbox = {"mpi-test.err","mpi-test.out"};
Requirements =
Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment)
&& Member("MPICH2", other.GlueHostApplicationSoftwareRunTimeEnvironment);
InputSandbox
mpi-test.out
mpi-test.err
OutputSandbox
Sets up the environment
for an specific MPI implementation
mpi-hooks.sh
Used before and after the execution of MPI program.
- Pre-hook: Download data and compile MPI.c
- Post-hook: Analyze and save data
mpi.c
MPI code
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

21. Structure of a MPI job in the Grid with mpi-start
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

22. Structure of a MPI job in the Grid with mpi-start
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

23. MPI Program
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

24. Output job
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

25. MPI-START Documentation
EGEE Mpi guide
MPI-START Documentation
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

26. Hands-on
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting

27. Questions …
Rabat, Joint CHAIN/EPIKH/EUMEDGRID Support event in School on Application Porting