1. Workload Management System (WMS) & Job Description Language (JDL)
Marcello Iacono Manno
INFN Catania
PI2S2 First Tutorial
Messina,

2. Sommario WMS (Workload Management System)
is a part of the Grid Middleware
collects the requests for resources from the jobs
collects information about the status of the infrastructure
matches requests with offers
specifically chose a CE (Computing Element) for execution
JDL (Job Description Language)
is a language for the description of the job request to the Grid
specifies input, output , executable files
indicates the kind of job (normal, collection, DAG, etc…)
gives user preferences about matchmaking
Catania, PI2S2 Tutorial,

3. Summary
Catania, PI2S2 Tutorial,

4. cancellation) expressed
WMS’s Architecture
Job management
requests (submission,
cancellation) expressed
via a Job Description
Language (JDL)
Catania, PI2S2 Tutorial,

5. WMS’s Architecture Keeps submission requests Requests are kept
for a while
if no matching
resources available
Catania, PI2S2 Tutorial,

6. The Task Queue Task Queue Non-matching requests
possibility to keep a submission request for a while if no resources are immediately available that match the job requirements
technique used by the AliEn and Condor systems
Non-matching requests
will be retried either periodically
eager scheduling approach
or as soon as notifications of available resources appear in the ISM (Information Super Market)
lazy scheduling approach
Catania, PI2S2 Tutorial,

7. WMS’s Scheduling Policies
WM can adopt:
eager scheduling (“push” model)
a job is bound to a resource as soon as possible and, once the decision has been taken, the job is passed to the selected resource for execution
lazy scheduling (“pull” model)
foresees that the job is held by the WM until a resource becomes available, at which point that resource is matched against the submitted jobs
the job that fits best is passed to the resource for immediate execution.
Catania, PI2S2 Tutorial,

8. WMS’s Architecture Finds an appropriate CE for each submission
request, taking into account
job requests and preferences,
Grid status, utilization policies
on resources
Catania, PI2S2 Tutorial,

9. WMS’s Architecture Performs the actual job submission and monitoring
Catania, PI2S2 Tutorial,

10. WMS’s Architecture Repository of resource information
available to matchmaker
Updated via notifications
and/or active
polling on sources
Catania, PI2S2 Tutorial,

11. The Information Supermarket
Information SuperMarket (ISM)
has been inherited by EU DataGrid (EDG) project
decoupling between the collection of information concerning resources and its use
allows flexible application of different policies
basically consists of a repository of resource information that is available in read only mode to the matchmaking engine
the update is the result of
the arrival of notifications
active polling of resources
some arbitrary combination of both
can be configured so that certain notifications can trigger the matchmaking engine
improve the modularity of the software
support the implementation of lazy scheduling policies
Catania, PI2S2 Tutorial,

12. Job Submission Services
WMS components handling the job during its lifetime and performing the submission
Job Adapter
is responsible for
making the final touches to the JDL expression for a job, before it is passed to CondorC for the actual submission
creating the job wrapper script that creates the appropriate execution environment in the CE worker node
transfer of the input and of the output sandboxes
CondorC
responsible for
performing the actual job management operations
job submission, job removal
DAGMan
meta-scheduler
purpose is to navigate the graph
determine which nodes are free of dependencies
follow the execution of the corresponding jobs.
instance is spawned by CondorC for each handled DAG
Log Monitor
watching the CondorC log file
intercepting interesting events concerning active jobs
events affecting the job state machine
triggering appropriate actions.
Catania, PI2S2 Tutorial,

13. Job Logging & Bookkeeping
L&B tracks jobs in terms of events
important points of job life
submission, finding a matching CE, starting execution etc
gathered from various WMS components
events are passed to a physically close component of the L&B infrastructure
locallogger
avoid network problems
stores them in a local disk file and takes over the responsibility to deliver them further
destination of an event is one of bookkeeping servers
assigned statically to a job upon its submission
processes the incoming events to give a higher level view on the job states
Submitted, Running, Done
various recorded attributes
JDL, destination CE name, job exit code
retrieval of both job states and raw events is available via legacy (EDG) and WS querying interfaces
user may also register for receiving notifications on particular job state changes
Catania, PI2S2 Tutorial,

14. Job State Machine (1/9)
Submitted: job is entered by the user to the User Interface but not yet transferred to Network Server for processing
Catania, PI2S2 Tutorial,

15. Job State Machine (2/9)
Waiting: job was accepted by NS and is waiting for Workload Manager processing or being processed by WMHelper modules.
Catania, PI2S2 Tutorial,

16. Job State Machine (3/9)
Ready: job processed by WM and its Helper modules (CE found) but not yet transferred to the CE (local batch system queue) via JC and CondorC. This state does not exists for a DAG as it is not subjected to matchmaking (the nodes are) but passed directly to DAGMan.
Catania, PI2S2 Tutorial,

17. Job State Machine (4/9)
Scheduled: job waiting in the queue on the CE. This state also does not exists for a DAG as it is not directly sent to a CE (the nodes are).
Catania, PI2S2 Tutorial,

18. Job State Machine (5/9)
Running: job is running. For a DAG this means that DAGMan has started processing it.
Catania, PI2S2 Tutorial,

19. Job State Machine (6/9)
Done: job exited or considered to be in a terminal state by CondorC (e.g., submission to CE has failed in an unrecoverable way).
Catania, PI2S2 Tutorial,

20. Job State Machine (7/9)
Aborted: job processing was aborted by WMS (waiting in the WM queue or CE for too long, over-use of quotas, expiration of user credentials).
Catania, PI2S2 Tutorial,

21. Job State Machine (8/9)
Cancelled: job has been successfully canceled on user request.
Catania, PI2S2 Tutorial,

22. Job State Machine (9/9) Cleared: output sandbox was transferred to
the user or removed due to the timeout.
Catania, PI2S2 Tutorial,

23. DAG State Machine
Catania, PI2S2 Tutorial,

24. input files interface output files GRID JDL executable job resources
JDL introduction
JDL describes a request to the WMS
allowing the submission of one or more jobs
the single items are called attributes
job descriptors are mandatory attributes
resource descriptors are supplied by the Information Service (IS) following the Glue (gLite) schema
input files
output files
JDL
GRID
job
executable
interface
resources
Catania, PI2S2 Tutorial,

25. JDL generalities JDL describes a request to the WMS
JDL allows submission, cancellation, status query, output retrieval with either CLI or GUI
two versions:
legacy Network Server interface (socket)
WMProxy (web service)
job description: Condor classified advertisements (classads)
descriptors are called attributes
mandatory attributes: job fundamental descriptors
resource attributes are related to the status and characteristics of grid resources Information Service (IS) schema (Glue for gLite)
Catania, PI2S2 Tutorial,

26. JDL format a list of entries enclosed by [ ] (each terminated by ;)
entry: <attribute> = <value> | < list of values >;
attribute: a string with the name of the attribute
value:
string “abc” a double-quoted string
Integer
Floating Point 12.34
Boolean “true”,”false”,expression (see GLUESchema)
classads (see nodes)
list of values: enclosed by <{ }> separated by <,>
{ “abc” , “bcd” , “def” }
Catania, PI2S2 Tutorial,

27. a Job a DAG a Collection a a a b c d b c d e node dependency e
Request types
Type = < string >
“Job” a simple job (default)
“DAG” a Direct Acyclic Graph of dependent jobs
“Collection” a set of independent jobs
a Job
a DAG
a Collection a a a b c d b c d e
node
dependency e
Catania, PI2S2 Tutorial,

28. JobType JobType = < string > “Normal” a simple job
“Interactive” a job whose standard streams are forwarded
to the submitting client
“MPICH” a MPI parallel job
“Partitionable” a job composed by a set of independent steps /
iterations for parallel execution
“Checkpointable” a job able to save its state in order to be suspended and resumed from the same point
“Parametric” a job with parametric attributes in its JDL to submit many similar instances with a single command (only parameterized attributes vary)
Catania, PI2S2 Tutorial,

29. Executable Executable = < string >
the path for the command/exec file(*)
“/usr/bin/java/j2sdk1.5.0/bin/java”, “/home/user/executable.exe”
environment variables accepted(*) “$JAVA/bin/java”
local or absolute paths accepted(*) “executable.exe”
requires an identical file entry in the Input Sandbox
remote files can be specified by gsiftp (local path exec)
mandatory for all jobs
wild cards not allowed
arguments are reported in a dedicated attribute
(*) on the executing WN
Catania, PI2S2 Tutorial,

30. Arguments Arguments = < string >
the arguments for the executable file
“-out outputfile.dat”
together with: Executable = “execprog”;
originates on the WN the command line:
$ execprog -out outputfile.dat
quotas (“”) have to follow a backslash(\)
“ -a \”quoted string\” -bcd” becomes (with the above executable)
$ execprog -a ”quoted string” -bcd
Catania, PI2S2 Tutorial,

31. StdInput, StdOutput, StdError
StdInput, StdOutput, StdError = < string >
the paths for the I/O/err file
same rules as the ‘Executable’ attribute
require identical entries in the Input/Output Sandboxes
StdOutput and StdError can be the same file
StdInput not required for Interactive jobs
examples:
StdInput = “/home/iacono/config.dat”;
StdOutput = “gsiftp://grid999.ct.infn.it:1234/tmp/file.out”;
Catania, PI2S2 Tutorial,

32. InputSandbox & OutputSandbox
InputSandbox, OutputSandbox = < string | list of strings >
identifying …
the input file(s) to be copied from the local UI (or a gridFTP server) to the WN before starting execution
the output file(s) to be prepared for downloading from WN after job execution completion (transfer: glite-job-output)
wild cards admitted if solved locally
LFN files not admitted (InputData or script–copy required)
files in InputSandbox must not exceed 10 MB length
different filenames required (same destination directory)
examples:
InputSandbox= { “myinp1.dat“ , ”data/myinp2.dat” };
Catania, PI2S2 Tutorial,

33. InputData InputData = < string | list of strings >
identify LFNs, GUIDs, LDs and/or queries
query the related Data Catalog to retrieve PFNs
input queried files into WN current directory
influences the WMS matchmaking decision
example:
InputData = { “lfn:/grid/gilda/isospin.dat”,
“guid:135b7b23-4a6a-87e7-9d101f8c8b70”,
“lds:testfile.inp” // LDS catalog
“query: select my_files”, // LDS catalog
“si-lfn:/file.inp” /* StorageIndex catalog */ };
when catalog is not specified (first two entries), if StorageIndex attribute is declared,
then the pointed catalog is used, otherwise WMS tries first RLS and then DLI
Catania, PI2S2 Tutorial,

34. StorageIndex & DataCatalog
StorageIndex = < string > (*)
the endpoint URL of the StorageIndex (SI) service to resolve file names
for si-lfn: or si-guid: files in InputData when StorageIndex is not specified, the VO default SI is used
example:
StorageIndex = " catalog-service-fr-mysql/services/SEIndex"
DataCatalog = < string > (*)
the endpoint URL of the RLS or DLI service to be used to resolve file names
example:
DataCatalog = “ service-fr-mysql/services/FiremanCatalog";
(*) for usage with InputData only
Catania, PI2S2 Tutorial,

35. OutputSE & OutputData OutputSE = < string >
a string representing the URL of a SE for output data storing
influence the matchmaking decision by the RB
example:
OutputSE = “grid009.ct.infn.it”;
OutputData = < list of classads > (*)
list of classads describing output files
similar to DataRequirements
automatic upload output files upon job completion
(*) not yet supported
Catania, PI2S2 Tutorial,

36. VirtualOrganisation, RetryCount & ShallowRetryCount
VirtualOrganisation = < string >
the name of the VO
has to match with the WMS default
overwritten by –vo option in glite-job-submit or edg-job-submit
example:
VirtualOrganisation = “cometa”;
RetryCount, ShallowRetryCount = < Integer >
indicate how many times the job must be re-submitted upon a failure due to some grid resources
not valid for DAG / Collection
limited by the MaxRetryCount parameter of WMS
resubmission is ‘shallow’ if user job aborted before running
‘deep’ resubmission resets shallow retry count
Catania, PI2S2 Tutorial,

37. LBAddress & VOMSProxyServer
LBAddress = < string >
the address of LB
format: <host>[:<port>]
default taken from WMS configuration (port = 9000)
example:
LBAddress = “lb-grid.ct.infn.it“;
MyProxyServer = < string >
the address of the proxy server
automatic renewal of proxy certificate (long jobs)
port defaults to 7512
example:
MyProxyServer = “grid001.ct.infn.it:7512”;
Catania, PI2S2 Tutorial,

38. NodeNumber & ListenerPort
NodeNumber = < Integer >
an integer >1 specifying how many CPUs are needed for a MPICH job
mandatory for JobType = “MPICH”
example:
NodeNumber = 3;
ListenerPort = < Integer >
the port number where condor_console_shadow listens for job standard streams
for usage with JobType = ‘Interactive’
example:
ListenerPort = 44000;
Catania, PI2S2 Tutorial,

39. ListenerHost & ListenerPipeName
ListenerHost = < string >
the host name on which the condor_console_shadow listens for job standard streams
for usage with JobType = “Interactive”
it is used when submission and interactive session are on different machines
ListenerPipeName = < string >
the absolute path of the pipes where job standard streams are located
example:
ListenerPipeName = “/tmp/pipe”;
means: stdin=/tmp/pipe.in, stdout=/tmp/pipe.out
(default=/tmp/listener/<jobID unique string>)
Catania, PI2S2 Tutorial,

40. JobSteps JobSteps = < Integer | list of strings >
either an integer representing the number of steps to run
or a string list with the labels associated to the steps of a
partitionable or checkpointable job
the main stepper is a part of the user job (not WMS) that links it to the run time checkpointing library
if also specified in JobState classad, this definition prevails
examples:
JobSteps = 1000; (runs 1000 steps of main stepper)
JobSteps = { “a”,”b”,”c” }; (runs sections “a”, “b”, “c” of main stepper)
Catania, PI2S2 Tutorial,

41. CurrentStep & JobState
CurrentStep = < Integer >
an integer number (>0) indicating the step number to be
taken as the initial one when submitting a checkpointable
or partitionable job
example:
CurrentStep = 2; (default=0)
JobState = < classad >
a job checkpoint state to start with when submitting a checkpointable job
example:
JobState = [ JobSteps = 1000; CurrentStep = 350; UserData = [
DumpPath=“gsiftp://grid999.ct.infn.it:1234/tmp/dumpfile” ] ]
Catania, PI2S2 Tutorial,

42. GLUESchema GLUESchema (Grid Logical Uniform Environment)
an information model to describe grid sites and services
independent from specific implementation
syntax < entity >.< property >
examples:
entity = < Site | Service | SubCluster | StorageElement | Host | ... | Other >
property = < UniqueID | NameUniqueID >
Requirements = other.GlueCEUniqueID == "grid010.ct.infn.it:2119/jobmanager-lcgpbs-infinite"
Requirements = other.GlueCEInfoLRMSType == "PBS" || other.GlueCEInfoLRMSType == "LSF"
Catania, PI2S2 Tutorial,

43. Requirements & UserTags
Requirements = < Boolean expression built with classads >
a Boolean classad expression with C-like operators
describe job requirement on resources (CE)
attributes according to the GlueSchema
CE requirements attributes in the IS begin with prefix “other.”
example:
Requirements = other.GlueCEInfoTotalCPUs > 2 &&
other.GlueCEPolicyMaxRunningJobs < 2;
UserTags = < classad >
a classad attribute that allows the user to specify user-defined key, value pairs (values must be strings)
tags can be used to query the LB
Catania, PI2S2 Tutorial,

44. Rank & FuzzyRank
Rank = < Floating Point expression built with classads >
a classad Floating Point expression showing the rank of each CE matching the Requirements expression
the highest rank CE will be selected for job execution
examples:
Rank = other.GlueCEPolicyMaxRunningJobs
other.GlueCEStateRunningJobs;
(CE with the greatest number of free slots)
Rank = - other.GlueCEStateEstimateResponseTime
(CE with the estimated shortest travel time through the local batch system queue)
FuzzyRank = < Boolean >
a Boolean attribute for the fuzzyfication of ranking attribution
defaults to false
Catania, PI2S2 Tutorial,

45. Nodes (1/2) Nodes = < list of classads >
classads describe nodes and their dependencies
example:
nodes = [
a = [ /* node “a” */
description = [ description: a classad
JobType = “Normal”; containing a JDL file
Executable = “a.exe”; to describe a node
InputSandbox = {…}; ];
b = [ /* node “b” */
file = node_b,jdl; file: a string indicating
]; the absolute path of a
… JDL file describing a node
Catania, PI2S2 Tutorial,

46. Parametric Jobs Job = “Parametric”
results in the submission of sets of identical jobs except for a parameter varying from a job to another
each job receives a different jobID so…
it is possible to trace it separately from the others but…
parametric job handle allows a common treating
a special variable (_PARAM_) marks variable items amongst JDL attributes
_PARAM_ assumes numerical values or a list of declared values (strings)
Catania, PI2S2 Tutorial,

47. ParameterStart, ParameterStep, Parameters, NodesCollocation
Parameters = < Integer | list of strings >
an Integer to indicate the number of steps or a list of strings (each of them is the name of a step)
ParameterStart, ParameterStep = < Integer >
ParameterStart indicates the initial step
ParameterStep indicates the amount of increment between two subsequent values of _PARAM_
NodesCollocation = < Boolean >
if true all the job instances are sent to the same CE
Catania, PI2S2 Tutorial,

48. Job Collections Job = “Collection”
a set of independent jobs that must be submitted, monitored and controlled as a single request
similar to a DAG, but without dependencies
all the clauses for DAGs are extended to Collections
nodes are treated by classads
attributes are referred to the whole Collection
an inherit mechanism is also present
Catania, PI2S2 Tutorial,

49. GangMatching Requirements = anyMatch (<list of classads>) |
whichMatch(<list of classads>) |
allMatch (<list of classads>)
RB uses classads to perform matchmacking
job and CE are usually the only involved entities
if also SE is to be considered, a more general mechanism is provided with new classad built-in functions
example:
Requirements = anyMatch( other.storage.CloseSEs,
target.GlueSAStateAvailableSpace > 200)
forces RB to select a CE close to a SE with >200 MBs of available space
Catania, PI2S2 Tutorial,

50. Bibliography bibliography WMS JDL attributes Simple job submission
JDL attributes
Simple job submission
More on JDL ( includes MPI jobs )
Job requesting data
Catania, PI2S2 Tutorial,

51. Questions…
Catania, PI2S2 Tutorial,