Advanced gLite job management Paschalis Korosoglou, AUTH/GRNET EPIKH Application Porting School 2011 Beijing, China Paschalis Korosoglou, AUTH/GRNET EPIKH Application Porting School 2011 Beijing, China

EPIKH Application Porting School, May 2011, Beijing, China Workload Management System 2

EPIKH Application Porting School, May 2011, Beijing, China For the hands-on session you will need 3 $ svn co $ svn update $ svn co $ svn update

EPIKH Application Porting School, May 2011, Beijing, China 4 Job collection Simultaneous submission Asynchronous execution As many jobs as we like $ glite-wms-job-submit -a --collection -o JobID JDLs/

EPIKH Application Porting School, May 2011, Beijing, China Parametric jobs Submission of multiple jobs each time varying the value of a parameter Parameter values defined either through an incremental process or through a predefined list Parameters may be integers, floats, characters or character strings 5

EPIKH Application Porting School, May 2011, Beijing, China Examples of parametric JDL files Example bash script: 6 JobType = "Parametric"; Executable = ”run.sh"; Arguments = "_PARAM_"; InputSandbox = ”run.sh"; Parameters = 8; ParameterStep = 2; ParameterStart = 0; StdOutput = "std_PARAM_.out”; StdError = "std.err"; OutputSandbox = {"std_PARAM_.out", "std.err”}; JobType = "Parametric"; Executable = ”run.sh"; Arguments = "_PARAM_"; InputSandbox = ”run.sh"; Parameters = 8; ParameterStep = 2; ParameterStart = 0; StdOutput = "std_PARAM_.out”; StdError = "std.err"; OutputSandbox = {"std_PARAM_.out", "std.err”}; JobType = "Parametric"; Executable = ”run.sh"; Arguments = "_PARAM_"; InputSandbox = ”run.sh"; Parameters = {0,2,4,6,8,10,12,14}; StdOutput = "std_PARAM_.out”; StdError = "std.err"; OutputSandbox = {"std_PARAM_.out", "std.err”}; JobType = "Parametric"; Executable = ”run.sh"; Arguments = "_PARAM_"; InputSandbox = ”run.sh"; Parameters = {0,2,4,6,8,10,12,14}; StdOutput = "std_PARAM_.out”; StdError = "std.err"; OutputSandbox = {"std_PARAM_.out", "std.err”}; #!/bin/bash param=$1 sleep $param echo $param #!/bin/bash param=$1 sleep $param echo $param Incremental switchVariables list

EPIKH Application Porting School, May 2011, Beijing, China Parametric jobs Simultaneous submission Asynchronous execution Excellent for 1 dimensional parametric studies 7 $ glite-wms-job-submit -a -o JobID param.jdl

EPIKH Application Porting School, May 2011, Beijing, China DAG jobs 8 DAG is an acronym for Directed Acyclic Graph In terms of job management DAG is used for defining interdependencies among jobs Characteristics: Job submission is asynchronous Job execution is asynchronous

EPIKH Application Porting School, May 2011, Beijing, China A dag job example 9 [ Type = "dag"; Nodes = [ peter = [ description = [ JobType="Normal"; Executable = "/bin/echo"; Arguments = "Hi! My name is Peter"; StdOutput = "peter.out"; StdError = "peter.err"; OutputSandbox = {"peter.out", "peter.err"}; ]; anna = [ description = [ JobType="Normal"; Executable = "reply.sh"; InputSandbox = {"reply.sh”,root.Nodes.peter.description.OutputSandbox[0]}; StdOutput = "anna.out"; StdError = "anna.err"; OutputSandbox = {"anna.out", "anna.err"}; ]; Dependencies = { { peter, anna } }; ] [ Type = "dag"; Nodes = [ peter = [ description = [ JobType="Normal"; Executable = "/bin/echo"; Arguments = "Hi! My name is Peter"; StdOutput = "peter.out"; StdError = "peter.err"; OutputSandbox = {"peter.out", "peter.err"}; ]; anna = [ description = [ JobType="Normal"; Executable = "reply.sh"; InputSandbox = {"reply.sh”,root.Nodes.peter.description.OutputSandbox[0]}; StdOutput = "anna.out"; StdError = "anna.err"; OutputSandbox = {"anna.out", "anna.err"}; ]; Dependencies = { { peter, anna } }; ] $ glite-wms-job-submit -a -o JobID dag.jdl

EPIKH Application Porting School, May 2011, Beijing, China MPI Jobs MPI is an acronym for Message Passing Interface MPI has become a very popular approach for writing parallel software/applications especially in the HPC realm In MPI parallel processes are synced or communicate through messages There are two types of messages One process to/from one process (point-to-point) One/all process to/from all/one processes (collective) 10

EPIKH Application Porting School, May 2011, Beijing, China Hello worlds example 11 #include using namespace std; int main(int argc, char* argv[]) { int rank,size; MPI_Init( &argc, &argv ); MPI_Comm_rank( MPI_COMM_WORLD, &rank ); MPI_Comm_size( MPI_COMM_WORLD, &size ); cout << "Hello World! I am process " << rank << " out of " << size << "." << endl; MPI_Finalize(); return 0; } #include using namespace std; int main(int argc, char* argv[]) { int rank,size; MPI_Init( &argc, &argv ); MPI_Comm_rank( MPI_COMM_WORLD, &rank ); MPI_Comm_size( MPI_COMM_WORLD, &size ); cout << "Hello World! I am process " << rank << " out of " << size << "." << endl; MPI_Finalize(); return 0; } Mpi library functions definitions are included MPI_Comm_rank assigns to each process a unique ID MPI_Comm_size assigns to each process the total number of processes Each process prints the values of its rank and size variables Notice that in an MPI programme variables are not shared. They are unique per process (distributed memory scheme).

EPIKH Application Porting School, May 2011, Beijing, China MPI in gLite There are more than one MPI implementations to choose from mpich, mpich2, lam-mpi, openmpi, mvapich etc gLite supports several MPI implementations through MPI-START Bootstrapping of mpi processes is done via a wrapper script (usually mpi-start.sh) An additional wrapper (mpi-hooks.sh) may be used for pre and post processing within the same job Within the JDL file clearly state mpi-start mechanism will be used by the job 12

EPIKH Application Porting School, May 2011, Beijing, China Hello worlds using gLite (wrappers) 13 #!/bin/bash MY_EXECUTABLE=`pwd`/$1 MPI_FLAVOR=$2 MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'` eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH` eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH export I2G_${MPI_FLAVOR}_PREFIX touch $MY_EXECUTABLE export I2G_MPI_APPLICATION=$MY_EXECUTABLE export I2G_MPI_APPLICATION_ARGS= export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh chmod +x $MY_EXECUTABLE $I2G_MPI_START #!/bin/bash MY_EXECUTABLE=`pwd`/$1 MPI_FLAVOR=$2 MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'` eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH` eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH export I2G_${MPI_FLAVOR}_PREFIX touch $MY_EXECUTABLE export I2G_MPI_APPLICATION=$MY_EXECUTABLE export I2G_MPI_APPLICATION_ARGS= export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh chmod +x $MY_EXECUTABLE $I2G_MPI_START pre_run_hook () { mpicxx -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.cpp return 0 } pre_run_hook () { mpicxx -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.cpp return 0 }

EPIKH Application Porting School, May 2011, Beijing, China Hello worlds using gLite (JDL file) 14 JobType = "Normal"; CpuNumber = 4; Executable = "mpi-start.sh"; Arguments = "hello MPICH2"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"mpi-start.sh","mpi-hooks.sh","src/hello.cpp"}; OutputSandbox = {"std.err","std.out"}; Requirements = Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment) && Member("MPICH2", other.GlueHostApplicationSoftwareRunTimeEnvironment); JobType = "Normal"; CpuNumber = 4; Executable = "mpi-start.sh"; Arguments = "hello MPICH2"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"mpi-start.sh","mpi-hooks.sh","src/hello.cpp"}; OutputSandbox = {"std.err","std.out"}; Requirements = Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment) && Member("MPICH2", other.GlueHostApplicationSoftwareRunTimeEnvironment); JobType is Normal A new attribute (CpuNumber) must be defined Arguments are picked up by the wrapper script(s) Clearly state that we need mpi-start at runtime and the mpi implementation of our choice

EPIKH Application Porting School, May 2011, Beijing, China That’s all folks… 15