GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga sw week, johannes ebke, tariq mahmoud

automating of job submission in Athena/AthenaMC outline I. GangaTasks, motivation and overview II. news … III. conclusion, to do tariq mahmoud atlas sw week, cern,

automating of job submission in Athena/AthenaMC I. motivation started in summer 07: 1) need a tool which automates job (re)submission 2) need a tool which accounts for job chaining (interconnection) first report in october 07 tariq mahmoud atlas sw week, cern,

automating of job submission in Athena/AthenaMC I. overview (motivation, mc) simul:5 recon:10 evgen:50 tariq mahmoud atlas sw week, cern,

need automate job (re)submission need job chaining (interconnection) automating of job submission in Athena/AthenaMC I. overview (motivation, mc) tariq mahmoud atlas sw week, cern,

complete dataset A,B,C: f1,f2,f3, …,fn A,B,C  f1,f2,f3 A,B,C  f4,f5,f6 A,B,C  f7,f8,f9 A,B,C  f4,f5,f6 A,B,C  f7,f8,f9 A,B,C  fn-2,fn-1,fn incomplete dataset A: f1,f2,f3 B: f1,f5,f6,f7,f8,f9 C: f1,f2,f7,f8,f9,f10 A,C  f1,f2 B  f5,f6 A  f3 B,C  f7,f8,f9 C  f10 automating of job submission in Athena/AthenaMC I. overview (motivation, analysis) tariq mahmoud atlas sw week, cern,

task few parameters: task parameters: float, name, number of events etc. process parameters: datasets, process name, geometry etc. input/output specifications Athena version AbstractJob generated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, … AbstractJob job Ganga jobs automating of job submission in Athena/AthenaMC I. overview (concept) tariq mahmoud atlas sw week, cern,

MCTask MCJob AnaTask AnaJob Task AnaJob repository/tools.data TaskList(GangaObject ) GPI.tasks automating of job submission in Athena/AthenaMC I. overview (concept) tariq mahmoud atlas sw week, cern,

automating of job submission in Athena/AthenaMC V. conclusion GangaTask package: automates job management (small scale production system) works with Ganga Athena and Ganga AthenaMC. wiki in progress: To do: 1. gather the largest number of files possible of an in-complete dataset. 2. test everything on Lxplus “and elsewhere”. 3. soon available in one of the next in Ganga releases automating of job submission in Athena/AthenaMC II. news - GangaTasks (explanation, gen. tutorial) - AnaTask (tutorial) 1.site-test-code implemented 2.more Ganga properties: EventView, Output dataset, Excluding sites/files, … 3.act on user suggestions. 1.Ganga-site-index. 2.tested at cern and gridka (KFZ) 3.MCTasks: implemented since Ganga AnaTasks soon in Ganga (wget _ tgz) tariq mahmoud atlas sw week, cern,

automating of job submission in Athena/AthenaMC II. news (control functions) 1.instructions, what to do in case of warnings and errors. 2.most attributes of GangaTasks are immutable.  more control functions: extend/exclude sites/files 3.do not change used files: job option & user area  pause task. 4.many other features … tariq mahmoud atlas sw week, cern, safe analysis

automating of job submission in Athena/AthenaMC V. conclusion GangaTask package: works with Ganga Athena and Ganga AthenaMC at several places. Two wikis: general and AnaTask more immutabilities to save jobs after submitting. more controlling functions. To do: 1. change site if job fails. 2. making use of GangaRobot 3. acting on users responds tariq mahmoud atlas sw week, cern,

automating of job submission in Athena/AthenaMC III. illustration: chaining task steps mc.total_events = 200; mc.events_per_job={"evgen": 50, "simul": 5, "recon":10} evgen:i-0 simul:i-j recon:i-k

mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10 AbstractJob.prerequisites() automating of job submission in Athena/AthenaMC III. illustration : interconnections, prerequisites automate job (re)submission job chaining (interconnection)  the user sets few parameters … and gets RESULTS