1. Workload Management System
Mike Mineter
Footer

2. Contents What is the Workload Management System (WMS)?
How do you use it?
Further information
Footer

3. Without WMS… Without the WMS, need direct interaction with nodes
User
Nodes
Without the WMS, need direct interaction with nodes
Need to know resource addresses, capabilities
Usually want a higher level abstraction – submit a job “to a Grid” not “to a CE”
Footer

4. Which CE do you want to use?
Without the WMS, use the Information System to see what’s available, then choose… lcg-infosites --vo gilda ce
#CPU Free Total Jobs Running Waiting ComputingElement
grid011f.cnaf.infn.it:2119/jobmanager-lcgpbs-short
grid011f.cnaf.infn.it:2119/jobmanager-lcgpbs-long
grid011f.cnaf.infn.it:2119/jobmanager-lcgpbs-infinite
grid010.ct.infn.it:2119/jobmanager-lcgpbs-short
grid010.ct.infn.it:2119/jobmanager-lcgpbs-long
grid010.ct.infn.it:2119/jobmanager-lcgpbs-infinite
…….[30% shown].
WMS does this for you!
chooses CE for each job, balances workload, manages jobs and their files
Footer

5. With WMS User WMS Compute Elements WMS manages jobs on users’ behalf
User doesn’t decide where jobs are run
User defines the job and its requiremements, WMS matches this with available CEs
Effect:
Easier submission
Users insulated from change in Compute elements
WMS – can optimise your jobs – e.g. which CE?
Footer

6. Basics Why does the Workload Management System exist? Grids have
Many users
Running many jobs – a “job” = an executable / script you want to run
many compute nodes are available
WMS makes running jobs easier for the user AND optimises use of available resources
It builds on the basic grid services
Authorisation, Authentication, Security, Information Systems, Job submission
Terminology: “Compute element”: defined as a batch queue - One cluster can have many queues
Footer

7. WMS User describes job in text file using Job Description Language
Local Workstation
User describes job in text file using Job Description Language
Submits job to WMS using (usually) the command-line interface
ssh UI
UI (user interface) has preinstalled client software
WMS
Workload Management System
CEs
Footer

8. Using WMS Jobs run in batch mode on grids.
Steps in running a job on a gLite grid with WMS:
Create a text file in “Job Description Language”
Optional check: list the compute elements that match your requirements (“list match” command)
Submit the job
Non-blocking - Each job is given an id.
Occasionally check the status of your job
When “Done” retrieve output
Footer

9. Example JDL file Executable = “gridTest”; StdError = “stderr.log”;
StdOutput = “stdout.log”;
InputSandbox = {“/home/joda/test/gridTest”};
OutputSandbox = {“stderr.log”, “stdout.log”};
Requirements = other.GlueCEPolicyMaxCPUTime > 480;
ShallowRetryCount = 3;
Example JDL file – InputData – from SE using logical file name.
Footer

10. Job states
Flag
Meaning
SUBMITTED
submission logged in the Logging & Bookkeeping service
WAIT
job match making for resources
READY
job being sent to executing CE
SCHEDULED
job scheduled in the CE queue manager
RUNNING
job executing on a Worker Node of the selected CE queue
DONE
job terminated without grid errors
CLEARED
job output retrieved
ABORT
job aborted by middleware, check reason
Footer

11. WMS: role of WMProxy UI WMProxy
Client on the UI communicates with the “WM Proxy”
On UI run: glite-wms-…commands
WMProxy acts on your behalf in using the WM – it needs a “delegated proxy” – hence “-a” option on commands
Local
Workstation UI
UI (user interface) has preinstalled client software
WMProxy
Workload
Manager
CEs
Footer

12. More about WMProxy UI WMProxy Workload Manager Local
Workstation
WMPProxy can manage complex jobs
Before WMProxy, user had to script or create software to manage these on the UI UI
UI (user interface) has preinstalled client software
WMProxy
Workload
Manager
CEs
Footer

13. Principales Commandes
glite-wms-job-submit (edg-job-submit) Soumets un job Retourne le jobID
glite-wms-job-status (edg-job-status)
Donne le statut du job
glite-wms-job-output (edg-job-get-output)
Récupère les fichiers spécifiés dans l’attribut OutputSandbox
glite-wms-job-cancel (edg-job-cancel)
Annule un job
glite-wms-job-list-match (edg-job-list-match)
Liste les ressources compatible avec la description du job
Effectue le matchmaking sans soumettre le job
glite-wms-job-logging-info (edg-job-get-logging-info)
Donne des informations de logging sur les jobs soumis (tout les événements répertoriés par les divers composants du WMS)
Très utile pour débuguer
Footer

14. WMS commands glite-wms-job-submit glite-wms-job-status
Submit a job
returns jobID – this is used in subsequent commands
glite-wms-job-status
Check status of job
glite-wms-job-output
Obtain result files
glite-wms-job-cancel
Cancel a job
glite-wms-job-list-match
List resourcews that can accept this job
Does the matchmaking against resource characteristics for the job
glite-wms-job-logging-info
Retrieves information from the logging service
Important for debugging
Footer

15. Further information gLite Users Guide GILDA wiki
Follow and “Documentation”
GILDA wiki
We are using some of these pages
EGEE Digital Library
Footer

16. What next? Practical to show basic use of the WMS
Then next talk shows more complex jobs
And another practical to run these
Footer