1. Job Description Language (JDL)
Valeria Ardizzone
INFN
EGEE User Tutorial
Johannesburg, March 2008

2. The JDL language
The Job Description Language (JDL) describes jobs for execution on Grid.
The JDL adopted within the gLite middleware is based upon Condor’s CLASSified Advertisement language (ClassAd).
A ClassAd is a record-like structure composed of a finite number of attributes separated by semi-colon (;)
A ClassAd is highly flexible and can be used to represent arbitrary services
The JDL file is processed by the “Match-making process” to select the best resource that satisfy the job’s requirements

3. The JDL language The JDL file lines have the format :
Attribute = expression;
2 categories of attributes:
Job Attributes define the job itself
Resources indicate the job constraints in terms of:
Computing Resource
Data and Storage resources
Comments are indicated by # or //
The JDL is sensitive to blank characters and tabs.
No blank characters or tabs should follow the semicolon at the end of a line.

4. JDL : basic attributes
In a JDL, some attributes are mandatory while others are optional. An “essential” JDL is the following:
Type -> is a string representing the type of the request described by the JDL
JobType -> is a string representing the type of the job described by the JDL
Executable -> is a string representing the executable/command name
Arguments -> is a string containing all the job command line arguments
StdInput -> is a string representing the standard input of the job.
StdOutput and -> is a string representing the standard output and error of the job.
StdError
InputSandbox -> is a string or a list of strings identifying the list of files on the UI local file system or on an accessible gridFTP server
OutputSandbox -> is a string or a list of strings identifying the list of files generated by the job on the WN at runtime, which the user wants to retrieve
Requirements -> it represents job requirements on resources.
Rank -> is a ClassAd Floating-Point expression that states how to rank CEs that have already met the Requirements expression.
RetryCount -> is an integer representing the maximum number of deep job re-submissions to be done in case of failure

5. JobType JobType = < string > “Normal” a simple job
“Interactive” a job whose standard streams are forwarded
to the submitting client
“MPICH” a MPI parallel job
“Partitionable” a job composed by a set of independent steps /
iterations for parallel execution
“Checkpointable” a job able to save its state in order to be suspended and resumed from the same point
“Parametric” a job with parametric attributes in its JDL to submit many similar instances with a single command (only parameterized attributes vary)

6. Example: Job Normal $> cat hostname.jdl Type = “Job”;
JobType = “Normal”;
Executable = “/bin/sh/”;
Arguments = “start_hostname.sh”;
StdError = “stderr.log”;
StdOutput = “stdout.log”;
InputSandbox = “start_hostname.sh”;
OutputSandbox = {“stderr.log”,”stdout.log”};
RetryCount = 7;
$> cat start_hostname.sh
#!/bin/sh
sleep 5
hostname –f

7. JDL : Requirements Requirements (mandatory)
Job requirements on Grid resources (CE,SE,…)
Evaluation performed by the Match Maker
Specified using attributes published by the Information Service
If not specified, the default value is:
Requirements = other.GlueCEStateStatus == "Production“;
Examples:
Requirements = other.GlueCEUniqueID == “adc006.cern.ch:2119/jobmanager-pbs-infinite”
Requirements = Member(“ALICE ”, other.GlueHostApplicationSoftwareRunTimeEnviron ment);
Requirements = other.GlueCEInfoTotalCPUs > 2 && other.GlueCEPolicyMaxRunningJobs < 2;

8. JDL : Rank Rank (mandatory)
Floating-Point expression used to rank CEs that have already met the Requirements expression.
can contain attributes that describe the CE in the Information System (IS).
evaluation performed by the Resource Broker (RB) during the match-making phase.
A higher numeric value equals a better rank.
If not specified, the default value is:
Rank = -other.GlueCEStateEstimatedResponseTime;
E.g.: Rank = other.GlueCEStateFreeCPUs;

9. Example: sphere.jdl $> cat sphere.jdl
#author:
Type = "Job";
JobType = "Normal";
Executable = "/bin/sh";
MyProxyServer="lxshare0207.cern.ch";
StdOutput = "sphere.out";
StdError = "sphere.err";
InputSandbox = {"start_sphere.sh","sphere1.pov","sphere1.ini"};
OutputSandbox = {"sphere.out","sphere.err","final_sphere.gif"};
RetryCount = 7;
Arguments = "start_sphere.sh";
Requirements =
Member("POVRAY-3.5",other.GlueHostApplicationSoftwareRunTimeEnvironment);

10. Example: pds2jpg-ASAR-Demo.jdl
$> cat pds2jpg-ASAR-Demo.jdl [
VirtualOrganisation = "gilda";
Executable = "/bin/bash";
Arguments = "pds2jpg_ASAR_install.sh
ASA_APG_1PXPDE _093043_ _00394_02452_0000";
StdOutput = "pds2jpg_ASAR.out";
StdError = "pds2jpg_ASAR.err";
OutputSandbox = {
"ASA_APG_1PXPDE _093043_ _00394_02452_0000-b1.jpg",
"ENVISAT_Product_courtesy_of_European_Space_Agency",
"pds2jpg_ASAR.out", "pds2jpg_ASAR.err“ };
RetryCount = 3;
JobType = "normal";
Type = "Job";
InputSandbox = {"./pds2jpg_ASAR_install.sh","./beam20.tar.gz"};
rank = (-other.GlueCEStateEstimatedResponseTime);
requirements = (other.GlueCEStateStatus=="Production") ]

11. Parametric Jobs (1/2) Job = “Parametric”
results in the submission of sets of identical jobs except for a parameter varying from a job to another
each job receives a different jobID so…
it is possible to trace it separately from the others but…
parametric job handle allows a common treating
a special variable (_PARAM_) marks variable items amongst JDL attributes
_PARAM_ assumes numerical values or a list of declared values (strings)

12. Parametric Jobs (1/2) Parameters = < Integer | list of strings >
an Integer to indicate the number of steps or a list of strings (each of them is the name of a step)
ParameterStart, ParameterStep = < Integer >
ParameterStart indicates the initial step
ParameterStep indicates the amount of increment between two subsequent values of _PARAM_
NodesCollocation = < Boolean >
if true all the job instances are sent to the same CE

13. a Job a DAG a Collection a a a b c d b c d e node dependency e
Request types
Type = < string >
“Job” a simple job (default)
“DAG” a Direct Acyclic Graph of dependent jobs
“Collection” a set of independent jobs
a Job
a DAG
a Collection a a a b c d b c d e
node
dependency e

14. Collections
A job Collection is a set of independent jobs that for some reasons (known to the user) have to be submitted, monitored and controlled as a single request. A B C
[ Type = "collection"; InputSandbox = { "input_common1.txt","input_common2.txt" }; nodes = { [ JobType = "Normal"; NodeName = "node1"; Executable = "/bin/sh"; Arguments = "script_node1.sh"; InputSandbox = {"script_node1.sh", root.InputSandbox[0] }; StdOutput = "myoutput1"; StdError = "myerror1"; OutputSandbox = {"myoutput1","myerror1"}; ], [ JobType = "Normal"; ] }; ]

15. DAG (1/2)
A DAG (directed acyclic graph) represents a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs.
The jobs are nodes (vertices) in the graph and the edges (arcs) identify the dependencies.
Type=“dag”; nodes = [ a= [ description = [ JobType=“Normal”; Executable=“e.exe”; InputSandBox={…}; ] ]
b=[ file=node.jdl; ] ] A B D C E F

16. DAG (2/2) max_running_nodes = < intero >
dependencies = < lista di liste of stringhe >
is a list of lists representing the dependencies between the
nodes of the DAG described in the nodes attribute.
syntax: { { a, b }, { a, c }, { a, d } , { c, e }, { d, e } }
example:
dependencies = {
{ A, {B, C, D} }; // i nodi “B”, “C” e “D” dipendono da “A” { B, F } { {C, D}, E }; { E, F }; }; A B D C E F
max_running_nodes = < intero >
is an integer greater than zero representing the maximum number of
nodes of a DAG that can be submitted by DAGMan at a given time.

17. References EGEE User Guide
UserGuide.pdf
JDL Attributes
JDL-Attributes-v0-9.pdf
Exercises on GILDA Wiki: