1. GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga sw week, 28.02.2008 johannes ebke, tariq mahmoud

2. automating of job submission in Athena/AthenaMC outline I. GangaTasks, motivation and overview II. news … III. conclusion, to do tariq mahmoud atlas sw week, cern, 28022008

3. automating of job submission in Athena/AthenaMC I. motivation started in summer 07: 1) need a tool which automates job (re)submission 2) need a tool which accounts for job chaining (interconnection) first report in october 07 tariq mahmoud atlas sw week, cern, 28022008

4. automating of job submission in Athena/AthenaMC I. overview (motivation, mc) 13597 246108 1 1 9 7 7 7 13597 246 8 1 1 79 3 6 35 63 3 5 56 5 6 6 6 simul:5 recon:10 evgen:50 tariq mahmoud atlas sw week, cern, 28022008

5. 13597 246108 1 1 79 3 6 35 63 13597 246 8 1 1 79 3 5 35 53 13597 246 8 1 1 79 3 6 35 63 7 5 6 13597 246 8 1 1 9 7 7 3 5 5 5 13597 246 8 1 1 79 3 6 35 63 13597 246 8 1 1 79 3 6 35 63 7 5 6 13597 246 8 1 1 9 7 7 3 5 5 5 13597 246 8 1 1 79 3 6 35 63 13597 246 8 1 1 9 7 7 3 5 5 6 7 4 4 44 need automate job (re)submission need job chaining (interconnection) automating of job submission in Athena/AthenaMC I. overview (motivation, mc) tariq mahmoud atlas sw week, cern, 28022008

6. complete dataset A,B,C: f1,f2,f3, …,fn A,B,C  f1,f2,f3 A,B,C  f4,f5,f6 A,B,C  f7,f8,f9 A,B,C  f4,f5,f6 A,B,C  f7,f8,f9 A,B,C  fn-2,fn-1,fn incomplete dataset A: f1,f2,f3 B: f1,f5,f6,f7,f8,f9 C: f1,f2,f7,f8,f9,f10 A,C  f1,f2 B  f5,f6 A  f3 B,C  f7,f8,f9 C  f10 automating of job submission in Athena/AthenaMC I. overview (motivation, analysis) tariq mahmoud atlas sw week, cern, 28022008

7. task few parameters: task parameters: float, name, number of events etc. process parameters: datasets, process name, geometry etc. input/output specifications Athena version AbstractJob generated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, … AbstractJob job Ganga jobs automating of job submission in Athena/AthenaMC I. overview (concept) tariq mahmoud atlas sw week, cern, 28022008

8. MCTask MCJob AnaTask AnaJob Task AnaJob repository/tools.data TaskList(GangaObject ) GPI.tasks automating of job submission in Athena/AthenaMC I. overview (concept) tariq mahmoud atlas sw week, cern, 28022008

9. automating of job submission in Athena/AthenaMC V. conclusion GangaTask package: automates job management (small scale production system) works with Ganga Athena and Ganga AthenaMC. wiki in progress: https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTasks To do: 1. gather the largest number of files possible of an in-complete dataset. 2. test everything on Lxplus “and elsewhere”. 3. soon available in one of the next in Ganga releases automating of job submission in Athena/AthenaMC II. news https://twiki.cern.ch/twiki/bin/view/Atla/ - GangaTasks (explanation, gen. tutorial) - AnaTask (tutorial) 1.site-test-code implemented 2.more Ganga properties: EventView, Output dataset, Excluding sites/files, … 3.act on user suggestions. 1.Ganga-site-index. 2.tested at cern and gridka (KFZ) 3.MCTasks: implemented since Ganga 4.4.3. AnaTasks soon in Ganga 4.4.9. (wget http://cern.ch/mahmoud/Tasks/Tasks _200208.tgz) tariq mahmoud atlas sw week, cern, 28022008

10. automating of job submission in Athena/AthenaMC II. news (control functions) 1.instructions, what to do in case of warnings and errors. 2.most attributes of GangaTasks are immutable.  more control functions: extend/exclude sites/files 3.do not change used files: job option & user area  pause task. 4.many other features … tariq mahmoud atlas sw week, cern, 28022008 safe analysis

11. automating of job submission in Athena/AthenaMC V. conclusion GangaTask package: works with Ganga Athena and Ganga AthenaMC at several places. Two wikis: general and AnaTask more immutabilities to save jobs after submitting. more controlling functions. To do: 1. change site if job fails. 2. making use of GangaRobot 3. acting on users responds tariq mahmoud atlas sw week, cern, 28022008

12. automating of job submission in Athena/AthenaMC III. illustration: chaining task steps mc.total_events = 200; mc.events_per_job={"evgen": 50, "simul": 5, "recon":10} 1 3 5 7 9 1 3 5 7 9 1 3 5 7 9 1 3 5 7 9 13597 246 10 8 13597 246 8 13597 246 8 13597 246 8 2341 evgen:i-0 simul:i-j recon:i-k

13. mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10 AbstractJob.prerequisites() automating of job submission in Athena/AthenaMC III. illustration : interconnections, prerequisites automate job (re)submission job chaining (interconnection)  the user sets few parameters … and gets RESULTS