1. 1 DIRAC Job submission A.Tsaregorodtsev, CPPM, Marseille LHCb-ATLAS GANGA Workshop, 21 April 2004

2. 2 DIRAC Job structure  Job consists of one or more steps; can be split in subjobs corresponding to steps.  Step consists of one or more modules; is indivisible w.r.t to the job scheduling;  Module is the least unit of the execution:  Use standard modules in production;  User defined modules in analysis can be used. SoftwareInstallation module GaussApplication module BookkeepingUpdate module Gauss Step SoftwareInstallation module BooleApplication module BookkeepingUpdate module Boole+Brunel Step BrunelApplication module Production Job

3. 3 Job class diagram Job Step Module n n TypedParameter n n n ScriptletScriptShellApplication ApplicationFactory

4. 4 Job object structure  Three-level structure as a result of quite a lot of experimentation:  Need for very complex workflows for production jobs;  Can be reduced to a single step, single module job for analysis  Simple Job Class structure allows for easy formalization:  For example – simple XML schema for persistent representation

5. 5 Job object contract (subset)  Job(‘job.xml’) - constructor from XML file  addParameter()  addOption() - just a special type parameter  addStep()  getPackages() – packages required by all the steps  inputData()  outputData()  execute() - recursively executes all the steps, modules  toXML() – save as XML file  toJDL() – generate JDL file suitable for submission

6. 6 Job API example module=Module() module.addParameter(‘NAME’,’PARAM’,’DaVinci’) module.addParameter(‘TYPE’,’PARAM’,’APPLICATION’) module.addParameter(‘VERSION’,’PARAM’,’@{DAVINCI_VERSION}’) opt = ‘={DATAFILE=”@{INPUT_FILE}” TYP=“ROOT”}’ module.addOption(‘EventSelector’,’Input’,opt) step = Step() step.addModule(module) step.addParameter(‘INPUT_FILE’,’INPUTDATA’, ’lfn:/lhcb/production/DST/xxxx.dst’) job = Job() job.addStep(step) job.addParameter(‘DAVINCI_VERSION’,’PARAM’,’v12r3’) job.addParameter(‘MaxCPUTime’,’JDL’,’100000’) job.addHeader(‘Author’,’atsareg’) xml_string = job.toXML() jdl_string = job.toJDL()

7. 7 Steps  Main purpose - container of modules  Execute modules;  Handle module failures;  Reporting job progress;  Steps are indivisible from the Workload management perspective;  Steps can be dependant one on each other  Job object can represent a DAG of arbitrary complexity

8. 8 Module types  The actual work is done in Modules:  Shell Any shell script either preinstalled or provided as a Module parameter;  Scriptlet Python code executed within the job process  Script Python class with an execute() method – Functor; Either shipped with the job or preinstalled.  Application More involved Python script for invoking a (Gaudi) application Preinstalled as part of the DIRAC software

9. 9 Module parameters  Modules have access to the parameters of the containing Step and to the parameters of the Job;  Module parameters can be defined in terms of the Step or Job parameters:  Allows to define and reuse very complex workflows (as templates with parameter placeholders);  Just few parameters on the Job level should be defined in order to instantiate the workflow.

10. 10 Job further enhancements  Possibility to generate complete job steering code from interconnected modules  Advantages  Job object execution does not depend on the local DIRAC software installation  More flexibility when connecting steps: can pass complex objects from step to step  Already used inside the DIRAC Production Console  Generating simpler XML job representation to submit to DIRAC WMS

11. 11 Workflow definition  Workflow is an object of the same Job class;  Possibly not fully defined  Dirac Console provides a graphical interface to assemble workflows of any complexity from standard building blocks (modules) :  Very useful for production workflow definitions;  May be too powerful ( for the moment ) for a user job.

12. 12 Job Submission: UI  First, the DIRAC UI should be installed:  Single script installation;  Requires only local python interpreter;  Will be installed on lxplus for common use.  Job submission CLI:  dirac-job-submit ; Returns jobID (number);  dirac-job-status Returns python dictionary;  dirac-job-get-output Retrieves OutputSandbox to the local directory  The Python API is provided as well

13. 13 Job Submission: API  Job submission API to be used in the user scripts or UI applications ( GANGA ) example with simplified Job definition API from Dirac import * dirac = Dirac() job=Job() job.setApplication('DaVinci','v12r11') job.setOption(‘DV_Pi0Calibr.opts’) job.setInputSandbox([‘DV_Pi0Calibr.opts’,'Application_DaVinci_v12r11/lib']) job.setInputData(['/lhcb/production/DC04/v2/DST/00000743_00008447_9.dst’, ‘/lhcb/production/DC04/v2/DST/00000743_00008479_9.dst’]) job.setOutputSandbox(['DVHistos.root','DVNtuples.root']) jobid = dirac.submit(job) print "Job ID = ",jobid

14. 14 Example JDL file  JDL file is generated behind the scene to be sent to DIRAC WMS  Similar to LCG JDL Requirements = ( member("Lyon_HPSS",other.LocalSE) ) && ( member(“CERN_Castor",other.LocalSE) ); Executable = "$LHCBPRODROOT/DIRAC/scripts/jobexec"; Arguments = "jobDescription.xml"; JobName = "DaVinci_1"; SoftwarePackages = { "DaVinci.v12r11" }; JobType = "user"; Owner = "ibelyaev"; InputSandbox = { "lib.tar.gz", "jobDescription.xml", "DV_Pi0Calibr.opts" }; StdOutput = "std.out"; StdError = "std.err"; OutputSandbox = { "std.out", "std.err", "pi0calibr.hbook", "pi0histos.hbook" }; InputData = { "LFN:/lhcb/production/DC04/v2/DST/00000743_00008447_9.dst", "LFN:/lhcb/production/DC04/v2/DST/00000743_00008479_9.dst”}; OutputData = { "/lhcb/test/DaVinci/v1r0/LOG/DaVinci_v12r11.alog" }; parameters = [ STEPS = "1"; STEP_1_NAME = "0_0_1" ]; ProductionId = "00000000"; JobId = 1347

15. 15 Job execution sequence Batch system WN … Site VO A execution environment VO B execution environment VO C execution environment Agent CE Job wrapper User interface Job JDL DIRAC WMS DIRAC WMS CE handler CE handler Job wrapper CE handler CE handler Job JDL  Remarks  CE handler can be the same both with WMS- like and direct job submission  Job wrapper does not depend on the CE back-end Depends on the VO execution environment  VO software;  VO services (DB’s, monitoring, etc);

16. 16 Job Wrapper  The wrapper script is what is submitted to CE and in fact executed on the Worker Node:  Automatically generated by the WMS ( Agent );  Sets up the environment for the job in place.  Wraps the executable provided by the user:  Brings in input data;  Brings in InputSandbox;  Invokes user analysis executable;  Reports the job status  Uploads resulting output data

17. 17 Job Wrapper  Wrapper uses external ( standard ) services via plug-ins:  Configuration Service  File Catalog Plug-ins for BK, Alien, LFC catalogs  Data moving with Replica Manager Plug-ins for gridftp, xxxtp, file, rfio accessible storages  Job status/parameters reporting Plug-ins for DIRAC monitoring, MonALISA E.g. providing R-GMA plug-in would be easy

18. 18 Job Wrapper  Same wrappers are used on all the DIRAC CE back-ends  If necessary back-end specific wrappers can be generated by the specific CE clients.  Other VO execution environments can be adapted by providing VO specific plug-ins

19. 19 DIRAC CE  CE interface (back-end):  submitJob(wrapper,jdl,batchname=None) Returns localJobID  getJobStatus(localJobID)  killJob(localJobID)  getDynamicInfo() Info on numbers of queuing/running jobs  Other useful functions ( missing ):  getTimeLeft(localJobID)  getLimits: Scratch, memory, spool, etc

20. 20 DIRAC CE  Implementation  Special class used by the agents running closely to the actual CE;  Can be easily transformed into a client-server pair for remote job submission: Example: ComputingElementEDG was done this way. No changes to the Agent code Agent CE Batch System Agent CE client Batch System CE server