The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) Advanced Job Riccardo Rotondo National Institute of Nuclear Physics Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators Cairo,

2 Outline Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, Job Collection Parametric Job Direct Acyclic Graph Job Message Passing Interface Job

3 Collection Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, [ Type = “Collection"; nodes={ [ Executable = "/bin/hostname"; Arguments = “-f"; StdOutput = "hostname.out"; StdError = "hostname.err"; OutputSandbox = {"hostname.err","hostname.out"}; ], [ Executable = "/bin/sh"; Arguments = "start_povray_valve.sh"; StdOutput = “povray.out"; StdError = “povray.err"; InputSandbox = {“start_povray_valve.sh"}; OutputSandbox = {“povray.err",“povray.out"}; Requirements = Member (“POVRAY-3,5”, other.GlueHostApplicationSoftwareRunTimeEnvironment); ] }; ] a c b d e

4 Parametric Job Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, Jobs which are the same excepeting for a parameter It’s possible to trace only one job… … all together as a parametric job. A special variable (_PARAM_) refers to the particular parameters of the job. _PARAM_ can be a number automatically generated or string taken from a list.

5 Parametric Job Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, Parameters = an integer set the number of steps or a string list (in that case a different string refers to a different steps) ParameterStart, ParameterStep = ParameterStart the first value ParameterStep increment between to possibly value _PARAM_ NodesCollocation = If it’s true all the parametric job are sent to the same CE

6 Parametric job: an example Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, JobType = "Parametric"; Executable = “/bin/echo"; Arguments = “_PARAM_”; StdOutput = "myoutput_PARAM_.txt"; StdError = "myerror_PARAM_.txt"; Parameters = 3; ParameterStep = 1; ParameterStart = 1; OutputSandbox = {“myoutput_PARAM_.txt”};

7 DAG Job Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, It’s a set of job where the input, the output or the execution of a job comes after one or more job. –Jobs are nodes of a trees. Every path identify a dependency between one or more job. –dependencies = –A list that describes dependencies, strings are the node of the trees. – format:{ { a, b }, { a, c }, { a, d }, { c, e }, { d, e } } examples: –dependencies = { { A, {B, C, D} }; // nodes “B”, “C” and “D” depends form “A” { B, F } { {C, D}, E }; { E, F }; }; –max_nodes_running = An integer > 0 set the maximum job that ca be submitted but the DAGman ABCDEF

Type = "dag"; max_nodes_running = 5; InputSandbox = {"/foo/*.exe", "/home/larocca/bar", "gsiftp://neo.datamat.it:5678/tmp/cms_sim.exe ", "file:///tmp/myconf"}; InputSandboxBaseURI = "gsiftp://matrix.datamat.it:5432/tmp"; nodes = [ nodeA = [ description = [ JobType = "Normal"; Executable = "a.exe"; InputSandbox = { "/home/larocca/myfile.txt", root.InputSandbox}; ]; nodeF = [description = [ JobType = "Normal"; Executable = "b.exe"; Arguments = "1 2 3"; OutputSandbox = {"myoutput.txt", "myerror.txt" }; ]; nodeD = [description = [ JobType = "Checkpointable"; Executable = "b.exe"; Arguments = "1 2 3"; InputSandbox = { "file:///home/larocca/data.txt", root.nodes.nodeF.description.OutputSandbox[0] }; ]; nodeC = [ file = "/home/larocca/nodec.jdl"; ]; nodeB = [ file = "foo.jdl"; ]; ]; dependencies = { { nodeA, nodeB }, { nodeA, nodeC }, {nodeA, nodeF }, { { nodeB, nodeC, nodeF }, nodeD } }; 8 DAG Job Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, a cf d b

9 Job Type Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, Possible value of JDL Type you can set: –JOB: A simple job (DEFAULT) –DAG: A Direct Acyclic Graph of dependent job –COLLECTION: A set of independent job with common attributes –PARAMETRIC: they are the same except for a parameter JOB A DAG a cf d b Collection a c b d e Parametic acbde _PARAM_

10 Job MPI Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, This kind of Job use MPI Framework on Computing Element WMProxy: –Select all CE which support this framework –Select again only the CE with enough CPU core Attributes –JobType=“MPICH” –CPUNumber=10; // for example this job nees 10 cores to run. [ Type = "Job"; JobType = "MPICH"; Executable = "cpi"; NodeNumber = 2; StdOutput = "cpi.out"; StdError = "cpi.err"; InputSandbox = {"cpi"}; OutputSandbox = {"cpi.err","cpi.out"}; RetryCount = 0; ]

11 References Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, WMProxy User’s guidehttps://edms.cern.ch/file/674643/1/EGEE- JRA1-TEC WMPROXY-guide-v0-3.pdfhttps://edms.cern.ch/file/674643/1/EGEE- JRA1-TEC WMPROXY-guide-v0-3.pdf JDL Attributes Specification Attributes-v0-8.pdf gLite 3.1 user’s guide v0-8.pdf

12 References Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, Submission with WMProxy – SimpleJobSubmissionhttps://grid.ct.infn.it/twiki/bin/view/GILDA/ SimpleJobSubmission Submission with DAGs, collections and parametric jobs – WmProxyUsehttps://grid.ct.infn.it/twiki/bin/view/GILDA/ WmProxyUse

13 Questions? Cairo, Africa Joint EUMEDGRID-Support/EPIKH School for Grid Site Administrators, Thank you for your attention