1. ADA Job Builder A Graphical Approach to Job Building ATLAS Software and Computing Workshop 23-27 May 2005 Chun Lik Tan clat@hep.ph.bham.ac.uk

2. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Agenda The current situation. A very quick tour of command-line job building What the ADA Job Builder hopes to achieve. One more time... this time graphically! Life after job submission... Current status. Future plans. Questions?

3. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building The current situation Release 1.10 of DIAL is the basis for the current ADA system. Job building is currently a command line affair. ADA/DIAL services are currently accessible from a ROOT client which provides the user with a comprehensive suite of DIAL commands... all within the ROOT environment. More recently, with PyDial ( developed by Karl Harrison at Cambridge ), the user can now work in the Python environment with.

4. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Command-line Job Building Search for a dataset: root [0] print(dsc.query("level = 'TOP' and owner='rome' and name like '%recov10%'", 30)) List has 6 entries: rome.003035.recov10.J2_Pt_35_70.AOD rome.004324.recov10.AcerMC_ttbb_l.AOD rome.004100.recov10.T1_McAtNLO_top.AOD rome.004402.recov10.SU2_Jimmy_focus.AOD rome.004403.recov10.SU3_Jimmy_bulk.AOD rome.004406.recov10.SU4_Jimmy_lowmass.AOD At the ROOT client command line:

5. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Command-line Job Building Check if dataset is suitable: root [1] print(dsc.attributes("rome.004403.recov10.SU3_Jimmy_bulk.AOD")) Row has 10 entries: level = TOP modtime = 20050413164430 name = rome.004403.recov10.SU3_Jimmy_bulk.AOD nevt = 69855 nfile = 1441 nsub = 40 owner = rome type = AOD uid = 10013-41913 virtual = 0 At the ROOT client command line:

6. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Command-line Job Building At the ROOT client command line: Note dataset’s name, retrieve it’s id and extract the dataset from the dataset repository: root [2] did = dsc.id("rome.004403.recov10.SU3_Jimmy_bulk.AOD"); root [3] pdst = dr.extract(did); root [4] pprint(pdst) EventMergeDataset 10013-41913 with no parent is locked and not empty Content includes 1 block: Dataset content block: Dataset type: AtlasPoolEventDataset Content label: AOD Content ID list has 36 entries: type BJetContainer with with key BCandidates type ElectronContainer with with key ElectronCollection type INavigable4MomentumCollection with with key MuonboyTrackParticles type INavigable4MomentumCollection with with key StacoTrackParticles type INavigable4MomentumCollection with with key TrackParticleCandidate type INavigable4MomentumCollection with with key TrackParticleCandidateXK type JetTagContainer with with key BJetCollection...

7. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Selecting a task: root [5] tid = tsc.id("atlasopt_example_zll"); root [6] ptsk = tr.extract(tid); root [7] pprint(ptsk) Task 10301-25 has 2 files: atlas_release jo.py You will probably want to modify the task so extract the embedded files: root [8] ptsk->write_files("mytask", true) (const int)0 At this point, you open a shell and edit the relevant files. Once you are done with the modifications, resume the session. root [9] ptsk = new dial::Task("atlas_release jo.py", "mytask"); root [10] pprint(ptsk) Task 10301-83 has 2 files: atlas_release jo.py Command-line Job Building At the ROOT client command line:

8. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Command-line Job Building Do the same for application and then you are ready to submit! root [11] aid = asc.id("atlasopt"); root [12] papp = ar.extract(aid); root [13] pprint(papp) Application 10201-18 has 2 files: build_task run root [14] submit() Application 10201-18 Task 10301-83 Dataset 10013-41913 *** Submitting job *** Submitted job status: CompoundJob 10501-8046 is initializing Application: 10201-18 Task 10301-83 Dataset 10013-41913 with 69855 events Job preferences ID 0-0 Run host: adial01.usatlas.bnl.gov... At the ROOT client command line: There is an easier way!

9. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Aims The graphical ADA Job Builder aims to: hide away the intricacies of the command line, present the user with a convenient and user friendly environment, provide job monitoring capabilities, manage job results, and be fully integrated with GANGA.

10. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Graphical Job Building

11. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Graphical Job Building SQL Query Builder constructing a dataset query:

12. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Graphical Job Building

13. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Graphical Job Building

14. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Graphical Job Building

15. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Graphical Job Building At this point, with application, task, dataset (and preferences) objects configured, the user can proceed to: save the configuration for future reference, and/or submit the job to the ADA Job scheduler. Submit

16. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Job Monitoring

17. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Job Monitoring Under the bonnet Based on the Twisted Framework (http://twistedmatrix.com). Client and server model. Connection-oriented asynchronous communications. Push architecture using function callbacks. Server polls the ADA Job Scheduler and pushes updated job status information to subscriber clients (e.g. the ADA Job Builder). Based on GANGA v4 prototype monitoring component.

18. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Current Status Application selection & modification Task selection & modification Dataset selection Job submission Loading & saving of job configuration (job template) Job monitoring Sub-job monitoring Job result retrieval ADA Job Scheduler selector GSI certificate authentication Implement native DIAL web service client

19. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Future plans Short term (next few weeks) First beta release Documentation Job result retrieval Sub-job monitoring Job scheduler selector Middle term (next few of months) Integration with GANGA v4 GSI certificate authentication Native DIAL web service client

20. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Questions ?

21. 25th May 2005ADA Job Builder - A Graphical Approach to Job Building Thank you!