GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga sw week, johannes ebke, tariq mahmoud

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC outline I. introduction, concept of tasks II. starting and submitting a task III. automating of job submission IV. job interconnection V. conclusion

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC I. motivation aim: automate job submission. job chaining (interconnection) the user gives few parameters … and gets RESULTS

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC I. concept task few parameters: task parameters: float, name, number of events etc. process parameters: datasets, process name, geometry etc. input/output specifications Athena version AbstractJob generated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, … AbstractJob job Ganga jobs

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC I. GangaTasks MCTask MCJob AnaTask AnaJob Task AnaJob repository/tools.data TaskList(GangaObject ) GPI.tasks

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC II: starting and submitting a new task running numbering on all tasks created by the same user if the task is 'new', make some settings permanent and set the status to 'running‘

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC III: automating Task()AnaTask() AnaJob() II: starting and submitting a new task auto submitting info resubmitting

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC III: automating s=AnaTask() s.name=" PythiaZmumu " s.float= 10 s.application_option_file=‘ $HOME/athena/…./run/AnalysisSkeleton_topOptions.py ' s.inputdata_dataset= " trig1_misal1_csc PythiaZmumu.recon.AOD.v _tid " s.requirements_sites=[ 'LRZ', 'FZK', 'LYON‘ ] s.outputdata_outputdata=[“ AnaPythiaZmumu.root "] s.submit()

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC III: info and more

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC III: resubmitting

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC IV: jobs interconnections Task()MCTask() MCJob() II: starting and submitting a new task III: auto submitting info resubmitting chaining task steps interconnection, prerequisites

automating of job submission in Athena/AthenaMC IV: chaining task steps mc.total_events = 200; mc.events_per_job={"evgen": 50, "simul": 5, "recon":10} tariq mahmoud atlas sw week, cern, evgen:i-0 simul:i-j recon:i-k

mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10 AbstractJob.prerequisites() tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC IV: interconnections, prerequisites (1) Task.float: does NOT apply for subjobs

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC AnaTask: optimise sites CE= ‘ lcg-lrz-ce.lrz-muenchen.de:2119/jobmanager-sge-atlas ’ requirements_sites= ['LRZ', 'FZK', 'LYON'] optimize_sites=False inputdata_match_ce_all =True inputdata_min_num_files=100 sites_optimized=[]

tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC V. conclusion GangaTask package: automates job management (small scale production system) works with Ganga Athena and Ganga AthenaMC. wiki in progress: To do: 1. gather the largest number of files possible of an in- complete dataset. 2. test everything on Lxplus “and elsewhere”. 3. soon available in one of the next in Ganga releases the user gives few parameters … and gets RESULTS