GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga hepcg workshop, siegen, 13.12.2007 johannes ebke, johannes elmsheuser, tariq mahmoud.

Slides:

Advertisements

Similar presentations

Max Mustermann Folientitel Veranstaltung Online Steering of HEP Applications Daniel Lorenz University of Siegen Cracow Grid Workshop –

Advertisements

User view Ganga classes and functions can be used interactively at a Python prompt, can be referenced in scripts, or can be used indirectly via a Graphical.

1 iDST/Analysis Status M.Ellis & J.J. Gomez-Cadenas HARP Analysis Meeting Tuesday 24 th June 2003.

6/4/20151 Introduction LHCb experiment. LHCb experiment. Common schema of the LHCb computing organisation. Common schema of the LHCb computing organisation.

Production Planning Eric van Herwijnen Thursday, 20 june 2002.

I.1 ii.2 iii.3 iv.4 1+1=. i.1 ii.2 iii.3 iv.4 1+1=

I.1 ii.2 iii.3 iv.4 1+1=. i.1 ii.2 iii.3 iv.4 1+1=

Israel Cluster Structure. Outline The local cluster Local analysis on the cluster –Program location –Storage –Interactive analysis & batch analysis –PBS.

Athena. Outline Setting up the environment Running an Athena job.

Experiment Support Introduction to HammerCloud for The LHCb Experiment Dan van der Ster CERN IT Experiment Support 3 June 2010.

Asynchronous Web Services Approach Enrique de Andrés Saiz.

AMI S.A. Datasets… Solveig Albrand. AMI S.A. A set is… A number of things grouped together according to a system of classification, or conceived as forming.

Analysis demos from the experiments. Analysis demo session Introduction –General information and overview CMS demo (CRAB) –Georgia Karapostoli (Athens.

Common Experiment Workbook Status Report Anne Heavey, Rob Kutschke May 22, 2013.

CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.

UAB Dynamic Monitoring and Tuning in Multicluster Environment Genaro Costa, Anna Morajko, Paola Caymes Scutari, Tomàs Margalef and Emilio Luque Universitat.

K.Harrison CERN, 21st November 2002 GANGA: GAUDI/ATHENA AND GRID ALLIANCE - Background and scope - Project organisation - Technology survey - Design -

David Adams ATLAS ATLAS Distributed Analysis David Adams BNL March 18, 2004 ATLAS Software Workshop Grid session.

03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.

11/10/2015S.A.1 Searches for data using AMI October 2010 Solveig Albrand.

David Adams ATLAS AJDL: Analysis Job Description Language David Adams BNL December 15, 2003 PPDG Collaboration Meeting LBL.

CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services Job Monitoring for the LHC experiments Irina Sidorova (CERN, JINR) on.

Nick Brook Current status Future Collaboration Plans Future UK plans.

Belle MC Production on Grid 2 nd Open Meeting of the SuperKEKB Collaboration Soft/Comp session 17 March, 2009 Hideyuki Nakazawa National Central University.

David Adams ATLAS DIAL status David Adams BNL July 16, 2003 ATLAS GRID meeting CERN.

David Adams ATLAS ATLAS Distributed Analysis Plans David Adams BNL December 2, 2003 ATLAS software workshop CERN.

Status of the LHCb MC production system Andrei Tsaregorodtsev, CPPM, Marseille DataGRID France workshop, Marseille, 24 September 2002.

Ganga A quick tutorial Asterios Katsifodimos Trainer, University of Cyprus Nicosia, Feb 16, 2009.

David Adams ATLAS DIAL status David Adams BNL November 21, 2002 ATLAS software meeting GRID session.

Andreas Morsch, CERN EP/AIP CHEP 2003 Simulation in ALICE Andreas Morsch For the ALICE Offline Project 2003 Conference for Computing in High Energy and.

Experiment Support ANALYSIS FUNCTIONAL AND STRESS TESTING Dan van der Ster, CERN IT-ES-DAS for the HC team: Johannes Elmsheuser, Federica Legger, Mario.

Zprávy z ATLAS SW Week March 2004 Seminář ATLAS SW CZ Duben 2004 Jiří Chudoba FzÚ AV CR.

David Adams ATLAS DIAL/ADA JDL and catalogs David Adams BNL December 4, 2003 ATLAS software workshop Production session CERN.

Nurcan Ozturk University of Texas at Arlington US ATLAS Transparent Distributed Facility Workshop University of North Carolina - March 4, 2008 A Distributed.

Karsten Köneke October 22 nd 2007 Ganga User Experience 1/9 Outline: Introduction What are we trying to do? Problems What are the problems? Conclusions.

ATLAS Distributed Analysis Dietrich Liko. Thanks to … pathena/PANDA: T. Maneo, T. Wenaus, K. De DQ2 end user tools: T. Maneo GANGA Core: U. Edege, J.

A PanDA Backend for the Ganga Analysis Interface J. Elmsheuser 1, D. Liko 2, T. Maeno 3, P. Nilsson 4, D.C. Vanderster 5, T. Wenaus 3, R. Walker 1 1: Ludwig-Maximilians-Universität.

David Adams ATLAS DIAL: Distributed Interactive Analysis of Large datasets David Adams BNL August 5, 2002 BNL OMEGA talk.

EMI INFSO-RI Guidelines and SQA Process Maria Alandes Pradillo (CERN) SA2.2 Task Leader.

CERN IT Department t LHCb Software Distribution Roberto Santinelli CERN IT/GS.

Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,

HammerCloud Functional tests Valentina Mancinelli IT/SDC 28/2/2014.

Update on Diffractive Dijet Production Search Hardeep Bansil University of Birmingham Birmingham ATLAS Weekly Meeting 24/01/2013.

ALFA GeoModel implementation and simulation status  ATHENA, what is our job?  ALFA GeoModel implementation  MC simulation – first steps  Problems 

K. Harrison CERN, 22nd September 2004 GANGA: ADA USER INTERFACE - Ganga release status - Job-Options Editor - Python support for AJDL - Job Builder - Python.

David Adams ATLAS ATLAS Distributed Analysis: Overview David Adams BNL December 8, 2004 Distributed Analysis working group ATLAS software workshop.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Andrea Sciabà Hammercloud and Nagios Dan Van Der Ster Nicolò Magini.

Using Ganga for physics analysis Karl Harrison (University of Cambridge) ATLAS Distributed Analysis Tutorial Milano, 5-6 February 2007

Pavel Nevski DDM Workshop BNL, September 27, 2006 JOB DEFINITION as a part of Production.

GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga sw week, johannes ebke, tariq mahmoud.

GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga sw week, johannes ebke, tariq mahmoud.

INFSO-RI Enabling Grids for E-sciencE Using of GANGA interface for Athena applications A. Zalite / PNPI.

CERN IT Department CH-1211 Genève 23 Switzerland t COOL Performance Tests ATLAS Conditions Database example Romain Basset, IT-DM October.

1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.

Distributed Physics Analysis Past, Present, and Future Kaushik De University of Texas at Arlington (ATLAS & D0 Collaborations) ICHEP’06, Moscow July 29,

Alex Undrus – GRID Testing – 18 Oct Nightlies Testing on the GRID: Status Oct Purposes: Moderate scale production for quick validation (when.

K. Harrison CERN, 21st February 2005 GANGA: ADA USER INTERFACE - Ganga release Python client for ADA - ADA job builder - Ganga release Conclusions.

David Adams ATLAS ATLAS Distributed Analysis (ADA) David Adams BNL December 5, 2003 ATLAS software workshop CERN.

CERN IT Department CH-1211 Genève 23 Switzerland t Future Needs of User Support (in ATLAS) Dan van der Ster, CERN IT-GS & ATLAS WLCG Workshop.

Future of Distributed Production in US Facilities Kaushik De Univ. of Texas at Arlington US ATLAS Distributed Facility Workshop, Santa Cruz November 13,

Planning Session. ATLAS(-CMS) End-to-End Demo Kaushik De is the Demo Czar Need to put team together Atlfast production jobs –Atlfast may be unstable over.

David Adams ATLAS ADA: ATLAS Distributed Analysis David Adams BNL December 15, 2003 PPDG Collaboration Meeting LBL.

Eduardo RodriguesLHCb Collaboration Week, 26th November Studies of Generator-level Selection Eduardo Rodrigues, CERN I. Proposal II. Procedure III.

ATLAS Physics Analysis Framework James R. Catmore Lancaster University.

Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.

Seven things you should know about Ganga K. Harrison (University of Cambridge) Distributed Analysis Tutorial ATLAS Software & Computing Workshop, CERN,

Duration: How long will a lecture take?

LHCb Computing Model and Data Handling Angelo Carbone 5° workshop italiano sulla fisica p-p ad LHC 31st January 2008.

Status and plans for bookkeeping system and production tools

Presentation transcript:

GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga hepcg workshop, siegen, johannes ebke, johannes elmsheuser, tariq mahmoud

tariq mahmoud hepcg workshop, siegen, automating of job submission in Athena/AthenaMC outline I. introduction, motivation II. concept of tasks III. some illustration IV. conclusion

automating of job submission in Athena/AthenaMC I. motivation, mc simul:5 recon: evgen:50 79 tariq mahmoud hepcg workshop, siegen, need automate job (re)submission

automating of job submission in Athena/AthenaMC I. motivation, mc tariq mahmoud hepcg workshop, siegen,

automating of job submission in Athena/AthenaMC I. motivation, mc need automate job (re)submission need job chaining (interconnection) tariq mahmoud hepcg workshop, siegen,

automating of job submission in Athena/AthenaMC I. motivation, analysis complete dataset tariq mahmoud hepcg workshop, siegen, A,B,C: f1,f2,f3, …,fn A,B,C  f1,f2,f3 A,B,C  f4,f5,f6 A,B,C  f7,f8,f9 A,B,C  f4,f5,f6 A,B,C  f7,f8,f9 A,B,C  fn-2,fn-1,fn incomplete dataset A: f1,f2,f3 B: f1,f5,f6,f7,f8,f9 C: f1,f2,f7,f8,f9,f10 A,C  f1,f2 B  f5,f6 A  f3 B,C  f7,f8,f9 C  f10

automating of job submission in Athena/AthenaMC II. concept of tasks task few parameters: task parameters: float, name, number of events etc. process parameters: datasets, process name, geometry etc. input/output specifications Athena version AbstractJob generated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, … AbstractJob job Ganga jobs tariq mahmoud hepcg workshop, siegen,

automating of job submission in Athena/AthenaMC II. concept of tasks: GangaTasks MCTask MCJob AnaTask AnaJob Task AnaJob repository/tools.data TaskList(GangaObject ) GPI.tasks tariq mahmoud hepcg workshop, siegen,

automating of job submission in Athena/AthenaMC III. illustration, resubmission tariq mahmoud hepcg workshop, siegen, automate job (re)submission need job chaining (interconnection)

automating of job submission in Athena/AthenaMC III. illustration: chaining task steps mc.total_events = 200; mc.events_per_job={"evgen": 50, "simul": 5, "recon":10} evgen:i-0 simul:i-j recon:i-k tariq mahmoud hepcg workshop, siegen,

mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10 AbstractJob.prerequisites() tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC III. illustration : interconnections, prerequisites automate job (re)submission job chaining (interconnection)  the user sets few parameters … and gets RESULTS

automating of job submission in Athena/AthenaMC IV. conclusion GangaTask package: automates job management ( small scale production system ) works with Ganga Athena and Ganga AthenaMC ( available from Ganga443 onwards ) works at cern wiki with tutorial: applicable to any job submission-software on the grid To do: 1. implementing site-test-code 2. making use of GangaRobot 3. act on user suggestions tariq mahmoud hepcg workshop, siegen, Configure once – run anywhere