1. MPI Applications with the Grid Engine
The EPIKH Project
(Exchange Programme to advance e-Infrastructure Know-How)
MPI Applications with the Grid Engine
Riccardo Rotondo Joint CHAIN/EUMEDGRID- Support/EPIKH School to Science Gateways Amman, Jordan

2. Outline What is MPI ? MPI Challenges on GRID MPI Interface
MPI embedded in your GRID Engine
MPI Portlet Development
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 2 2

3. Message Passing Interface
What is MPI ?
A standard defining a syntax and semantics useful for righting message-passing appications.
Why MPI ?
Heavy usage of CPU power in HPC;
Development of portable and scalable large parallel applications.
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 3 3

4. MPI in GRID There is no standard way of starting an MPI application
No common syntax for mpirun, mpiexec support optional
The cluster where the MPI job is supposed to run doesn't have a shared file system
How to distribute the binary and input files?
How to gather the output?
Different clusters over the Grid are managed by different Local Resource Management Systems (PBS, LSF, SGE,…)
Where is the list of machines that the job can use?
What is the correct format for this list?
How to compile MPI program?
How can a physicist working on Windows workstation compile his code for/with an Itanium MPI implementation?
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 4 4

5. MPI-Start
Specifies a unique interface to the upper layer to run a MPI job
Allow the support of new MPI implementations without modifications in the Grid middleware
Support of “simple” file distribution
Provide some support for the user to help manage his data
Grid Middleware
MPI-START
MPI
Resources
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 5 5

6. MPI-Start
mpi-start is a recommended solution to hide the implementation details for jobs submission.
The design of mpi-start was focused in making the MPI job submission as transparent as possible from the cluster details!
Using the mpi-start system requires the user to define a wrapper script that set the environment variables and a set of hooks.
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 6 6

7. MPI-Start design goals
Portable
The program must be able to run under any supported operating system
Modular and extensible architecture
Plugin/Component architecture
Relocatable
Must be independent of absolute path, to adapt to different site configurations
Remote “injection” of mpi-start along with the job
“Remote” debugging features
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 7 7

8. MPI-Start Architecture
CORE
Scheduler
Execution
Hooks
It was developed inside the Int.EU.Grid project
PBS/Torque
SGE
LSF
Open MPI
MPICH2
LAM
PACX
Local
User
Compiler
File Dist.
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 8 8

9. MPI-Start Flow START Scheduler Plugin Execution Plugin Hooks PluginsNO
Do we have a scheduler plugin for the current environment?
Ask Scheduler plugin for a machinefile in default format NO
Execution Plugin
Do we have a plugin for the selected MPI?
It was developed inside the Int.EU.Grid project
Activate MPI Plugin
Prepare mpirun
Trigger pre-run hooks
Start mpirun
Trigger post-run hooks
Hooks Plugins
Dump Env
EXIT
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 9 9

10. Using MPI-Start Interface with environment variables:
I2G_MPI_APPLICATION:
The executable
I2G_MPI_APPLICATION_ARGS:
The parameters to be passed to the executable
I2G_MPI_TYPE:
The MPI implementation to use (e.g openmpi, ...)
I2G_MPI_VERSION:
Specifies which version of the the MPI implementation to use. If not defined the default version will be used
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 10 10

11. Using MPI-Start More variables: I2G_MPI_PRECOMMAND
Specifies a command that is prepended to the mpirun (e.g. time).
I2G_MPI_PRE_RUN_HOOK
Points to a shell script that must contain a “pre_run_hook” function.
This function will be called before the parallel application is started (usage: compilation of the executable)
I2G_MPI_POST_RUN_HOOK
Like I2G_MPI_PRE_RUN_HOOK, but the script must define a “post_run_hook” that is called after the parallel application finished (usage: upload of results).
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 11 11

12. Using MPI-Start MPI commands are transparent to the user
The script:
The submission (in SGE):
The StdOut:
The StdErr:
~]$ cat test2mpistart.sh
#!/bin/sh
# This is a script to show how mpi-start is called
# Set environment variables needed by mpi-start
export I2G_MPI_APPLICATION=/bin/hostname
export I2G_MPI_APPLICATION_ARGS=
export I2G_MPI_NP=2
export I2G_MPI_TYPE=openmpi
export I2G_MPI_FLAVOUR=openmpi
export I2G_MPI_JOB_NUMBER=0
export I2G_MPI_STARTUP_INFO=/home/imain179
export I2G_MPI_PRECOMMAND=time
export I2G_MPI_RELAY=
export I2G_MPI_START=/opt/i2g/bin/mpi-start
# Execute mpi-start
$I2G_MPI_START
~]$ qsub -S /bin/bash -pe openmpi 2 -l allow_slots_egee=0 ./test2mpistart.sh
~]$ cat test2mpistart.sh.o114486
Scientific Linux CERN SLC release 4.5 (Beryllium)
lflip30.lip.pt
lflip31.lip.pt
It was developed inside the Int.EU.Grid project
[lflip31] /home/imain179 > cat test2mpistart.sh.e114486
Scientific Linux CERN SLC release 4.5 (Beryllium)
real 0m0.731s
user 0m0.021s
sys 0m0.013s
MPI commands are transparent to the user
No explicit mpiexec/mpirun instruction
Start the script via normal LRMS submission
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 12 12

13. Wrapper Script for MPI Start
#!/bin/bash
# Pull in the arguments.
MY_EXECUTABLE=`pwd`/$1
MPI_FLAVOR=$2
# Convert flavor to lowercase for passing to mpi-start.
MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'`
# Pull out the correct paths for the requested flavor.
eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH`
# Ensure the prefix is correctly set. Don't rely on the defaults.
eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH
export I2G_${MPI_FLAVOR}_PREFIX
# Touch the executable.
#It exist must for the shared file system check.
# If it does not, then mpi-start may try to distribute the executable
# when it shouldn't.
touch $MY_EXECUTABLE
# Setup for mpi-start.
export I2G_MPI_APPLICATION=$MY_EXECUTABLE
export I2G_MPI_APPLICATION_ARGS=
export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER
export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh
# If these are set then you will get more debugging information.
export I2G_MPI_START_VERBOSE=1
#export I2G_MPI_START_DEBUG=1
# Invoke mpi-start.
$I2G_MPI_START
The script takes 2 argument the executable name and the MPI flavor
It was developed inside the Int.EU.Grid project
The script refers to the Hooks scripts
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 13 13

14. Wrapper Script for MPI Start
#!/bin/bash
# Pull in the arguments.
MY_EXECUTABLE=`pwd`/$1
MPI_FLAVOR=$2
# Convert flavor to lowercase for passing to mpi-start.
MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'`
# Pull out the correct paths for the requested flavor.
eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH`
# Ensure the prefix is correctly set. Don't rely on the defaults.
eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH
export I2G_${MPI_FLAVOR}_PREFIX
# Touch the executable.
#It exist must for the shared file system check.
# If it does not, then mpi-start may try to distribute the executable
# when it shouldn't.
touch $MY_EXECUTABLE
# Setup for mpi-start.
export I2G_MPI_APPLICATION=$MY_EXECUTABLE
export I2G_MPI_APPLICATION_ARGS=
export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER
export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh
# If these are set then you will get more debugging information.
export I2G_MPI_START_VERBOSE=1
#export I2G_MPI_START_DEBUG=1
# Invoke mpi-start.
$I2G_MPI_START
The script takes 2 argument the executable name and the MPI flavor
It was developed inside the Int.EU.Grid project
The script refers to the Hooks scripts
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 14 14

15. Outline Resource Manager worker node
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 15 15

16. MPI Portlet Develop MPI Portlet
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 16 16

17. cpunumber in mpi_portlet.java
A global variable representing the number or the cpu can be set by user
String cpunumber; // Number of cpu will execute the mpi script in parallel
Reading the CPU number from the user request
It was developed inside the Int.EU.Grid project
public void getInputForm(ActionRequest request) {
. . .
List items = upload.parseRequest(request);
cpunumber=item.getString();
// or
cpunumber=(String)request.getParameter("cpunumber"); }
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 17 17

18. cpunumber in mpi_portlet.java
Using the cpunumber submitting the job:
public void submitJob() {
. . .
String arguments="mpi-start-wrapper.sh cpi mpich2";
JSagaJobSubmission tmpJSaga = new JSagaJobSubmission();
tmpJSaga.setTotalCPUCount(cpunumber); }
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 18 18

19. cpunumber input.jsp Using the cpunumber submitting the job:
<form enctype="multipart/form-data" action="<portlet:actionURL portletMode="view"><portlet:param name="PortletStatus" value="ACTION_SUBMIT"/></portlet:actionURL>" method="post">
. . .
<p>Insert <b>cpu number</b></p>
<textarea id="cpunumber" rows="1" cols="60%" name="cpunumber">4</textarea>
</form>
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 19 19

20. mpi-start, mpi-hooks, mpi-app
Setting the location of the mpi scripts and mpi application.
public void submitJob() {
. . .
String executable="/bin/sh";
String arguments="mpi-start-wrapper.sh cpi mpich2";
String inputSandbox= appServerPath+"/WEB-INF/job/pilot_script.sh”
+","+appServerPath+"/WEB-INF/job/cpi.c"
+","+appServerPath+"/WEB-INF/job/mpi-hooks.sh"
+","+appServerPath+"/WEB-INF/job/mpi-start-wrapper.sh"
+","+inputSandbox_inputFile; }
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 20 20

21. Grid Option in portlet.xml
In portlet.xml can set some default preferences that can be recall in the java code:
<portlet>
<portlet-name>mpi-portlet</portlet-name>
<display-name>mpi-portlet</display-name>
<portlet-class>it.infn.ct.mpi_portlet</portlet-class>
. . .
<init-param>
<name>init_JobRequirements</name>
<value>
Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment);
Member("MPICH", other.GlueHostApplicationSoftwareRunTimeEnvironment)
</value>
</init-param>
</portlet>
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 21 21

22. Grid Option in mpi-portlet.java
First we need to get the value from portlet preferences:
private void getPreferences( ActionRequest actionRequest, RenderRequestrenderRequest) {
. . .
pref_JobRequirements=prefs.getValue("pref_JobRequirements",init_JobRequirements); }
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 22 22

23. Grid Option in mpi-portlet.java
Now set the Job Requirements in java code to submit the job:
public void submitJob() {
. . .
String jdlRequirements[] = pref_JobRequirements.split(";");
int numRequirements=0;
for(int i=0; i<jdlRequirements.length; i++) {
if(!jdlRequirements[i].equals("")) {
jdlRequirements[numRequirements] = "JDLRequirements=("+jdlRequirements[i]+")";
numRequirements++;
tmpJSaga.setJDLRequirements(jdlRequirements); }
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 23 23

24. Grid Option in mpi-portlet.java
References
Gilda Trainin Material:
Science Gateway developer page:
/wiki/Main/Science%20Gateway%20Developer%20P ages
MPI Standalone wiki:
/wiki/Main/GridEngineMPIStandaloneCode
It was developed inside the Int.EU.Grid project
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways, 24 24

25. Thank you for your attention
Special thanks to Mario Reale
Amman, Jordan - Joint CHAIN/EUMEDGRID-Support/EPIKH School for Application Porting, 25 25