Job Description Language (JDL) Valeria Ardizzone INFN EGEE User Tutorial Johannesburg, 28-29 March 2008

The JDL language The Job Description Language (JDL) describes jobs for execution on Grid. The JDL adopted within the gLite middleware is based upon Condor’s CLASSified Advertisement language (ClassAd). A ClassAd is a record-like structure composed of a finite number of attributes separated by semi-colon (;) A ClassAd is highly flexible and can be used to represent arbitrary services The JDL file is processed by the “Match-making process” to select the best resource that satisfy the job’s requirements

The JDL language The JDL file lines have the format : Attribute = expression; 2 categories of attributes: Job Attributes define the job itself Resources indicate the job constraints in terms of: Computing Resource Data and Storage resources Comments are indicated by # or // The JDL is sensitive to blank characters and tabs. No blank characters or tabs should follow the semicolon at the end of a line.

JDL : basic attributes In a JDL, some attributes are mandatory while others are optional. An “essential” JDL is the following: Type -> is a string representing the type of the request described by the JDL JobType -> is a string representing the type of the job described by the JDL Executable -> is a string representing the executable/command name Arguments -> is a string containing all the job command line arguments StdInput -> is a string representing the standard input of the job. StdOutput and -> is a string representing the standard output and error of the job. StdError InputSandbox -> is a string or a list of strings identifying the list of files on the UI local file system or on an accessible gridFTP server OutputSandbox -> is a string or a list of strings identifying the list of files generated by the job on the WN at runtime, which the user wants to retrieve Requirements -> it represents job requirements on resources. Rank -> is a ClassAd Floating-Point expression that states how to rank CEs that have already met the Requirements expression. RetryCount -> is an integer representing the maximum number of deep job re-submissions to be done in case of failure

JobType JobType = < string > “Normal” a simple job “Interactive” a job whose standard streams are forwarded to the submitting client “MPICH” a MPI parallel job “Partitionable” a job composed by a set of independent steps / iterations for parallel execution “Checkpointable” a job able to save its state in order to be suspended and resumed from the same point “Parametric” a job with parametric attributes in its JDL to submit many similar instances with a single command (only parameterized attributes vary)

Example: Job Normal $> cat hostname.jdl Type = “Job”; JobType = “Normal”; Executable = “/bin/sh/”; Arguments = “start_hostname.sh”; StdError = “stderr.log”; StdOutput = “stdout.log”; InputSandbox = “start_hostname.sh”; OutputSandbox = {“stderr.log”,”stdout.log”}; RetryCount = 7; $> cat start_hostname.sh #!/bin/sh sleep 5 hostname –f

JDL : Requirements Requirements (mandatory) Job requirements on Grid resources (CE,SE,…) Evaluation performed by the Match Maker Specified using attributes published by the Information Service If not specified, the default value is: Requirements = other.GlueCEStateStatus == "Production“; Examples: Requirements = other.GlueCEUniqueID == “adc006.cern.ch:2119/jobmanager-pbs-infinite” Requirements = Member(“ALICE-3.07.01”, other.GlueHostApplicationSoftwareRunTimeEnviron ment); Requirements = other.GlueCEInfoTotalCPUs > 2 && other.GlueCEPolicyMaxRunningJobs < 2;

JDL : Rank Rank (mandatory) Floating-Point expression used to rank CEs that have already met the Requirements expression. can contain attributes that describe the CE in the Information System (IS). evaluation performed by the Resource Broker (RB) during the match-making phase. A higher numeric value equals a better rank. If not specified, the default value is: Rank = -other.GlueCEStateEstimatedResponseTime; E.g.: Rank = other.GlueCEStateFreeCPUs;

Example: sphere.jdl $> cat sphere.jdl #author: giuseppe.larocca@ct.infn.it Type = "Job"; JobType = "Normal"; Executable = "/bin/sh"; MyProxyServer="lxshare0207.cern.ch"; StdOutput = "sphere.out"; StdError = "sphere.err"; InputSandbox = {"start_sphere.sh","sphere1.pov","sphere1.ini"}; OutputSandbox = {"sphere.out","sphere.err","final_sphere.gif"}; RetryCount = 7; Arguments = "start_sphere.sh"; Requirements = Member("POVRAY-3.5",other.GlueHostApplicationSoftwareRunTimeEnvironment);

Example: pds2jpg-ASAR-Demo.jdl $> cat pds2jpg-ASAR-Demo.jdl [ VirtualOrganisation = "gilda"; Executable = "/bin/bash"; Arguments = "pds2jpg_ASAR_install.sh ASA_APG_1PXPDE20020819_093043_000000152008_00394_02452_0000"; StdOutput = "pds2jpg_ASAR.out"; StdError = "pds2jpg_ASAR.err"; OutputSandbox = { "ASA_APG_1PXPDE20020819_093043_000000152008_00394_02452_0000-b1.jpg", "ENVISAT_Product_courtesy_of_European_Space_Agency", "pds2jpg_ASAR.out", "pds2jpg_ASAR.err“ }; RetryCount = 3; JobType = "normal"; Type = "Job"; InputSandbox = {"./pds2jpg_ASAR_install.sh","./beam20.tar.gz"}; rank = (-other.GlueCEStateEstimatedResponseTime); requirements = (other.GlueCEStateStatus=="Production") ]

Parametric Jobs (1/2) Job = “Parametric” results in the submission of sets of identical jobs except for a parameter varying from a job to another each job receives a different jobID so… it is possible to trace it separately from the others but… parametric job handle allows a common treating a special variable (_PARAM_) marks variable items amongst JDL attributes _PARAM_ assumes numerical values or a list of declared values (strings)

Parametric Jobs (1/2) Parameters = < Integer | list of strings > an Integer to indicate the number of steps or a list of strings (each of them is the name of a step) ParameterStart, ParameterStep = < Integer > ParameterStart indicates the initial step ParameterStep indicates the amount of increment between two subsequent values of _PARAM_ NodesCollocation = < Boolean > if true all the job instances are sent to the same CE

a Job a DAG a Collection a a a b c d b c d e node dependency e Request types Type = < string > “Job” a simple job (default) “DAG” a Direct Acyclic Graph of dependent jobs “Collection” a set of independent jobs a Job a DAG a Collection a a a b c d b c d e node dependency e

Collections A job Collection is a set of independent jobs that for some reasons (known to the user) have to be submitted, monitored and controlled as a single request. A B C [ Type = "collection"; InputSandbox = { "input_common1.txt","input_common2.txt" }; nodes = { [ JobType = "Normal"; NodeName = "node1"; Executable = "/bin/sh"; Arguments = "script_node1.sh"; InputSandbox = {"script_node1.sh", root.InputSandbox[0] }; StdOutput = "myoutput1"; StdError = "myerror1"; OutputSandbox = {"myoutput1","myerror1"}; ], [ JobType = "Normal"; . . . ] }; ]

DAG (1/2) A DAG (directed acyclic graph) represents a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs. The jobs are nodes (vertices) in the graph and the edges (arcs) identify the dependencies. Type=“dag”; nodes = [ a= [ description = [ JobType=“Normal”; Executable=“e.exe”; InputSandBox={…}; ] ] b=[ file=node.jdl; ] ] A B D C E F

DAG (2/2) max_running_nodes = < intero > dependencies = < lista di liste of stringhe > is a list of lists representing the dependencies between the nodes of the DAG described in the nodes attribute. syntax: { { a, b }, { a, c }, { a, d } , { c, e }, { d, e } } example: dependencies = { { A, {B, C, D} }; // i nodi “B”, “C” e “D” dipendono da “A” { B, F } { {C, D}, E }; { E, F }; }; A B D C E F max_running_nodes = < intero > is an integer greater than zero representing the maximum number of nodes of a DAG that can be submitted by DAGMan at a given time.

References EGEE User Guide https://edms.cern.ch/file/722398//gLite-3- UserGuide.pdf JDL Attributes https://edms.cern.ch/file/555796/1/EGEE-JRA1-TEC- 555796-JDL-Attributes-v0-9.pdf Exercises on GILDA Wiki: