Computing Lectures Introduction to Ganga 1 Ganga: Introduction Object Orientated Interactive Job Submission System –Written in python –Based on the concept of a job object Developed by Atlas + LHCb –With dedicated (very helpful) support team –Used widely - including outside HEP (e.g. biomedics) –Ensures that it is an experiment-neutral framework All experiment specific extensions are plug-ins. Very easy automated install –It just takes two commands and < 5 minutes! Multiple backends – local, psub, condor as well as GRID (glite-wms-*) –A very powerful feature: run the same job as another process on the same machine or on the GRID just by changing its backend attribute Both standalone and a framework –Is a complete ready to run system for submitting jobs –Also API on which additional features can be layered Using local backend is great for development before switching to GRID e.g. MINOS have developed a fault tolerant batch system

Computing Lectures Introduction to Ganga 2 Ganga: The central concept: A job A job is the central object –Jobs are created, configured, submitted and when complete, examined. –They are persistant Saved automatically into a Registry Quit Ganga at any time and come back later and resume Creation –On creation each gets unique ID –This is a serial number that can be used to access the job Configuration –At any time can configure all aspects - executable, environment, input and output sandboxes, backend Cloning –Having configured one job it can be cloned to create others Submission –Job gets submitted to the configured backend Polling –A job poller runs in the background and monitors submitted jobs –It reports state changes. –When job is complete it retrieves output and places in the job’s output directory Resubmit –If a job fails it can be resubmitted Removal –Once a job is no longer needed it can be removed along with its input and output directories Global summaries –Get one line per job summary either of all jobs or just a slice (by ID or some other attribute).

Computing Lectures Introduction to Ganga 3 Ganga: Two Interfaces Example –To create job, configure, submit, check status and examine output:- my_job = Job() my_job.application = Executable(exe=File('~/somescript'), args=['1','2','3']) my_job.backend = LCG() my_job.submit() my_job.status my_job.peek(‘stdout’) Ganga comes with ipython –help with objects - they list methods and state e.g.: help(my_job) –tab completion - just type start of data or function and hit tab e.g. my_job.p gives my_job.peek Command line GUI

Computing Lectures Introduction to Ganga 4 Ganga: Other Features Subjobs –Have a master job and a dataset it is to operate on –Job Splitter creates subjobs each taking part of the dataset –Submitting the master job submits all the subjobs –Subjobs can also be accessed individually e.g. fix and resubmit errors. –When all subjobs complete Job Merger recombines the output Trees –Initially job repository is a flat structure –Can create folder hierarchy each with own jobs –Can be used to perform global operations on collections of jobs Templates –Configurable like jobs but cannot be submitted –Used to create preconfigured jobs Authentication management –GRID certificate – will ask to re-authenticate when expired –AFS token Configuration of all aspects –By resource file read on start up –Interactively during execution Verbosity control –By level (CRITICAL, ERROR, WARNING, INFO, and DEBUG.) –By specific parts of GANGA