Computing Lectures Introduction to Ganga 1 Ganga: Introduction Object Orientated Interactive Job Submission System –Written in python –Based on the concept.

Slides:



Advertisements
Similar presentations
User view Ganga classes and functions can be used interactively at a Python prompt, can be referenced in scripts, or can be used indirectly via a Graphical.
Advertisements

ATLAS/LHCb GANGA DEVELOPMENT Introduction Requirements Architecture and design Interfacing to the Grid Ganga prototyping A. Soroko (Oxford), K. Harrison.
13/05/2004Janusz Martyniak Imperial College London 1 Using Ganga to Submit BaBar Jobs Development Status.
GANGA Overview Germán Carrera, Alfredo Solano (CNB/CSIC) EMBRACE COURSE Monday 19th of February to Friday 23th. CNB-CSIC Madrid.
GRID INTEROPERABILITY USING GANGA Soonwook Hwang (KISTI) YoonKee Lee and EunSung Kim (Seoul National Uniersity) KISTI-CCIN2P3 FKPPL Workshop December 1,
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL March 25, 2003 CHEP 2003 Data Analysis Environment and Visualization.
Workload Management Workpackage Massimo Sgaravatto INFN Padova.
GRID Workload Management System Massimo Sgaravatto INFN Padova.
Workload Management Massimo Sgaravatto INFN Padova.
Experiment Support Introduction to HammerCloud for The LHCb Experiment Dan van der Ster CERN IT Experiment Support 3 June 2010.
Operating Systems.
Slide 1 of 9 Presenting 24x7 Scheduler The art of computer automation Press PageDown key or click to advance.
DIRAC API DIRAC Project. Overview  DIRAC API  Why APIs are important?  Why advanced users prefer APIs?  How it is done?  What is local mode what.
Analysis demos from the experiments. Analysis demo session Introduction –General information and overview CMS demo (CRAB) –Georgia Karapostoli (Athens.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.
Track 1: Cluster and Grid Computing NBCR Summer Institute Session 2.2: Cluster and Grid Computing: Case studies Condor introduction August 9, 2006 Nadya.
The Pipeline Processing Framework LSST Applications Meeting IPAC Feb. 19, 2008 Raymond Plante National Center for Supercomputing Applications.
Grid Initiatives for e-Science virtual communities in Europe and Latin America DIRAC TEAM CPPM – CNRS DIRAC Grid Middleware.
Central Reconstruction System on the RHIC Linux Farm in Brookhaven Laboratory HEPIX - BNL October 19, 2004 Tomasz Wlodek - BNL.
INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
1 Overview of the Application Hosting Environment Stefan Zasada University College London.
Belle MC Production on Grid 2 nd Open Meeting of the SuperKEKB Collaboration Soft/Comp session 17 March, 2009 Hideyuki Nakazawa National Central University.
DIRAC Review (13 th December 2005)Stuart K. Paterson1 DIRAC Review Exposing DIRAC Functionality.
Stuart Wakefield Imperial College London Evolution of BOSS, a tool for job submission and tracking W. Bacchi, G. Codispoti, C. Grandi, INFN Bologna D.
Ganga A quick tutorial Asterios Katsifodimos Trainer, University of Cyprus Nicosia, Feb 16, 2009.
Enabling Grids for E-sciencE EGEE-III INFSO-RI Using DIANE for astrophysics applications Ladislav Hluchy, Viet Tran Institute of Informatics Slovak.
Introduction to Alexander Richards Thanks to Mike Williams, ICL for many of the slides content.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Direct gLExec integration with PanDA Fernando H. Barreiro Megino CERN IT-ES-VOS.
July 11-15, 2005Lecture3: Grid Job Management1 Grid Compute Resources and Job Management.
Introduction to Ganga Karl Harrison (University of Cambridge) ATLAS Distributed Analysis Tutorial Milano, 5-6 February 2007
Getting started DIRAC Project. Outline  DIRAC information system  Documentation sources  DIRAC users and groups  Registration with DIRAC  Getting.
Ganga Status Update Will Reece. Will Reece - Imperial College LondonPage 2 Outline User Statistics User Experiences New Features in Upcoming Features.
Successful Distributed Analysis ~ a well-kept secret K. Harrison LHCb Software Week, CERN, 27 April 2006.
Ganga 4 Basics - Tutorial Jakub T. Moscicki ARDA/LHCb Ganga Tutorial, November 2005.
EGEE-III INFSO-RI Enabling Grids for E-sciencE Ricardo Rocha CERN (IT/GS) EGEE’08, September 2008, Istanbul, TURKEY Experiment.
INFSO-RI Enabling Grids for E-sciencE Ganga 4 – The Ganga Evolution Andrew Maier.
Distributed Analysis K. Harrison LHCb Collaboration Week, CERN, 1 June 2006.
Ganga 4 Basics - Tutorial Jakub T. Moscicki ARDA/LHCb Ganga Tutorial, September 2006.
April 27, 2006 The New GANGA GUI 26th LHCb Software Week C L Tan
INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Ganga User Interface EGEE Review Jakub Moscicki.
Tier3 monitoring. Initial issues. Danila Oleynik. Artem Petrosyan. JINR.
K. Harrison CERN, 22nd September 2004 GANGA: ADA USER INTERFACE - Ganga release status - Job-Options Editor - Python support for AJDL - Job Builder - Python.
Using Ganga for physics analysis Karl Harrison (University of Cambridge) ATLAS Distributed Analysis Tutorial Milano, 5-6 February 2007
2 June 20061/17 Getting started with Ganga K.Harrison University of Cambridge Tutorial on Distributed Analysis with Ganga CERN, 2.
Development of test suites for the certification of EGEE-II Grid middleware Task 2: The development of testing procedures focused on special details of.
ATLAS-specific functionality in Ganga - Requirements for distributed analysis - ATLAS considerations - DIAL submission from Ganga - Graphical interfaces.
ADA Job Builder A Graphical Approach to Job Building ATLAS Software and Computing Workshop May 2005 Chun Lik Tan
INFSO-RI Enabling Grids for E-sciencE Using of GANGA interface for Athena applications A. Zalite / PNPI.
K. Harrison CERN, 21st February 2005 GANGA: ADA USER INTERFACE - Ganga release Python client for ADA - ADA job builder - Ganga release Conclusions.
David Adams ATLAS ATLAS Distributed Analysis and proposal for ATLAS-LHCb system David Adams BNL March 22, 2004 ATLAS-LHCb-GANGA Meeting.
INFSO-RI Enabling Grids for E-sciencE Ganga 4 Technical Overview Jakub T. Moscicki, CERN.
A GANGA tutorial Professor Roger W.L. Jones Lancaster University.
+ Auto-Testing Code for Teachers & Beginning Programmers Dr. Ronald K. Smith Graceland University.
Geant4 GRID production Sangwan Kim, Vu Trong Hieu, AD At KISTI.
Claudio Grandi INFN Bologna Virtual Pools for Interactive Analysis and Software Development through an Integrated Cloud Environment Claudio Grandi (INFN.
Development of portlets for special jobs: parametric, collections, workflows Mario Torrisi National Institute of Nuclear Physics.
Seven things you should know about Ganga K. Harrison (University of Cambridge) Distributed Analysis Tutorial ATLAS Software & Computing Workshop, CERN,
Development of portlets for special jobs: parametric, collections, workflows Mario Torrisi Istituto Nazionale di Fisica Nucleare.
User view Ganga classes and functions can be used interactively at a Python prompt, can be referenced in scripts, or can be used indirectly via a Graphical.
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL May 19, 2003 BNL Technology Meeting.
Workload Management Workpackage
Distrubuited Analysis using GANGA
Blueprint of Persistent Infrastructure as a Service
Progress on NA61/NA49 software virtualisation Dag Toppe Larsen Wrocław
IW2D migration to HTCondor
The Ganga User Interface for Physics Analysis on Distributed Resources
X in [Integration, Delivery, Deployment]
Presentation transcript:

Computing Lectures Introduction to Ganga 1 Ganga: Introduction Object Orientated Interactive Job Submission System –Written in python –Based on the concept of a job object Developed by Atlas + LHCb –With dedicated (very helpful) support team –Used widely - including outside HEP (e.g. biomedics) –Ensures that it is an experiment-neutral framework All experiment specific extensions are plug-ins. Very easy automated install –It just takes two commands and < 5 minutes! Multiple backends – local, psub, condor as well as GRID (glite-wms-*) –A very powerful feature: run the same job as another process on the same machine or on the GRID just by changing its backend attribute Both standalone and a framework –Is a complete ready to run system for submitting jobs –Also API on which additional features can be layered Using local backend is great for development before switching to GRID e.g. MINOS have developed a fault tolerant batch system

Computing Lectures Introduction to Ganga 2 Ganga: The central concept: A job A job is the central object –Jobs are created, configured, submitted and when complete, examined. –They are persistant Saved automatically into a Registry Quit Ganga at any time and come back later and resume Creation –On creation each gets unique ID –This is a serial number that can be used to access the job Configuration –At any time can configure all aspects - executable, environment, input and output sandboxes, backend Cloning –Having configured one job it can be cloned to create others Submission –Job gets submitted to the configured backend Polling –A job poller runs in the background and monitors submitted jobs –It reports state changes. –When job is complete it retrieves output and places in the job’s output directory Resubmit –If a job fails it can be resubmitted Removal –Once a job is no longer needed it can be removed along with its input and output directories Global summaries –Get one line per job summary either of all jobs or just a slice (by ID or some other attribute).

Computing Lectures Introduction to Ganga 3 Ganga: Two Interfaces Example –To create job, configure, submit, check status and examine output:- my_job = Job() my_job.application = Executable(exe=File('~/somescript'), args=['1','2','3']) my_job.backend = LCG() my_job.submit() my_job.status my_job.peek(‘stdout’) Ganga comes with ipython –help with objects - they list methods and state e.g.: help(my_job) –tab completion - just type start of data or function and hit tab e.g. my_job.p gives my_job.peek Command line GUI

Computing Lectures Introduction to Ganga 4 Ganga: Other Features Subjobs –Have a master job and a dataset it is to operate on –Job Splitter creates subjobs each taking part of the dataset –Submitting the master job submits all the subjobs –Subjobs can also be accessed individually e.g. fix and resubmit errors. –When all subjobs complete Job Merger recombines the output Trees –Initially job repository is a flat structure –Can create folder hierarchy each with own jobs –Can be used to perform global operations on collections of jobs Templates –Configurable like jobs but cannot be submitted –Used to create preconfigured jobs Authentication management –GRID certificate – will ask to re-authenticate when expired –AFS token Configuration of all aspects –By resource file read on start up –Interactively during execution Verbosity control –By level (CRITICAL, ERROR, WARNING, INFO, and DEBUG.) –By specific parts of GANGA