INFSO-RI Enabling Grids for E-sciencE Ganga 4 Technical Overview Jakub T. Moscicki, CERN

Ganga Overview - J.T.Moscicki 2 Enabling Grids for E-sciencE INFSO-RI Ganga Overview AtlasPR OD DIAL DIRAC LCG2 gLite localhos t LSF submit, kill, get output update status store, retrieve job definition executa ble Gaudi Athena DaVi nci Brun el prepare, configure Ganga4

Ganga Overview - J.T.Moscicki 3 Enabling Grids for E-sciencE INFSO-RI User Interface Ganga.C ore GPI X programming API for job submission/monitoring rapidity/easiness of command line End-User-Interface: enhanced python shell (IPython) Qt-based GUI (3.x) scripts embedding into existing framework: use Ganga as a library/package

Ganga Overview - J.T.Moscicki 4 Enabling Grids for E-sciencE INFSO-RI Ganga Public Interface Ganga GPI Hello World >>> job = Job(exe='/bin/echo',parameters=['hello world']) >>> job.submit() submitting job >>> outfile = file(job.directory+'/output/std.out') >>> print outfile.read() Job started at: Fri Feb 18 14:05: Processing input files... /bin/echo Done hello world Application executed with the status code 0 Processing output files... Exiting... Job finished at: Fri Feb 18 14:05: >>> job2 = job.copy() >>> job2.backend = “LSF” >>> job2.submit()

Ganga Overview - J.T.Moscicki 5 Enabling Grids for E-sciencE INFSO-RI Ganga Public Interface Inspecting the jobs >>> print job.id 5 >>> print jobs Statistics: 5 jobs jobs ID status name # 1 completed # 2 new Job # 3 completed Job # 4 submitted Job # 5 completed Job >>> for j in jobs[1:3]:... print j.id 1 2

Ganga Overview - J.T.Moscicki 6 Enabling Grids for E-sciencE INFSO-RI Ganga Public Interface Complex scenarios >>> j = Job() >>> j.application = DaVinci() >>> j.application.options = 'my.opts' >>> j.backend = Glite() >>> j.backend.requirements = 'other.GlueCEUniqueID == "grid- ce.desy.de:2119/jobmanager-lcgpbs-short"' more on configuring applications in Atlas and LHCb specific presentations

Ganga Overview - J.T.Moscicki 7 Enabling Grids for E-sciencE INFSO-RI Additional Requirements Can I ? –add a new backend (e.g. PBS) for my site [plug-ins] –add a new application or modify existing one easily [plug-ins] –inspect at home the jobs which I submitted at work [remote job registry] –use Ganga GPI in my python scripts or integrate into existing python system –configure application from my laptop but I haven't installed full ATLAS/LHCB sofwtare release [remote application manager] –make sure that my job output will not be purged from the grid system before I get it [remote job manager]

Ganga Overview - J.T.Moscicki 8 Enabling Grids for E-sciencE INFSO-RI Extending Ganga Creating a Ganga 4 handler is easy –use the plug-in mechanism –example: application handler for Gaudi Create code: –create a class derived from GangaObject –define a schema of data the handler requires –implement the interface for the handler Update the config file ~/.ganga4 [Configuration] RUNTIME_PATH = ~/MyApplication

Ganga Overview - J.T.Moscicki 9 Enabling Grids for E-sciencE INFSO-RI Handler Plugin Example ## Import the GPI from Ganga.GPIDev.Base import GangaObject from Ganga.GPIDev.Schema import * […] class Gaudi(GangaObject): _schema = Schema(Version(1,0),{'optsfile':FileItem(), 'version':SimpleItem(None), 'platform':SimpleItem(None), 'package': SimpleItem(None), 'appname':SimpleItem(None)}) _category='applications‘ Describe the data needed to configure and run an application Set the type of your plugin Create a class based on GangaObject. Provide a Schema for you data Fulfill the interface for the type of handler

Ganga Overview - J.T.Moscicki 10 Enabling Grids for E-sciencE INFSO-RI Handler Plugin Example def configure(self): […] extra=GaudiExtras() […] extra.flatopts=FileParser.writeString(gaudiopts,"exp and") return (None,extra) def list_choices(self,property): class GaudiExtras(GangaObject): _schema = Schema(Version(1,0),{"flatopts": SimpleItem(None)}) _name="GaudiExtras" _category="extras“ _name='Gaudi' Use a GaudiExtras object to return information added by configure Create a Schema with the data of the configured app Set the category of this class to “extras”

Ganga Overview - J.T.Moscicki 11 Enabling Grids for E-sciencE INFSO-RI Remote Job Repository Remote job repository –access to the your jobs from multiple locations –in the future possibility to share jobs between users Ganga Client Ganga Client ARDA MD Server / PostgreSQL local repository

Ganga Overview - J.T.Moscicki 12 Enabling Grids for E-sciencE INFSO-RI Internal architecture Application Manager Job Manager Remote Registry Client Ganga 4 is decomposed into 4 functional components These components also describe the components in a distributed model. Strategy: Design each component so that it could be a separate service. But allow to combine two or more components into a single service

Ganga Overview - J.T.Moscicki 13 Enabling Grids for E-sciencE INFSO-RI Client Application Manager Job Manager Remote Registry Client Runs the Ganga interface (CLIP, GPI, GUI) The user interacts exclusively through the client With the client, the user creates, modifies, submits and monitors jobs Job configuration is kept in a registry which can be local (within the client) or remote.

Ganga Overview - J.T.Moscicki 14 Enabling Grids for E-sciencE INFSO-RI Client Application Manager Job Manager Remote Registry Client The client is a thin client (pure python) The client can be a command line client or a GUI The client interacts with the application manager to configure applications It submits and monitors jobs via the job manager It keeps state by storing persistent information in the registry

Ganga Overview - J.T.Moscicki 15 Enabling Grids for E-sciencE INFSO-RI Application Manager Application Manager Job Manager Remote Registry Client Prepares and configures the application Compiles user code Sets-up the necessary environment Provides information to the client on available applications, versions, platforms, etc.

Ganga Overview - J.T.Moscicki 16 Enabling Grids for E-sciencE INFSO-RI Job Manager Application Manager Job Manager Remote Registry Client Submits the configured job to the submission backend A submission handler submits a job to a backend –Creates the starter script and the JDL – performs the monitoring The application runtime handler –prepares the application dependent wrapper script, depending on the backend. –E.g., DIRAC knows how to run LHCb applications with a different setup as LSF.

Ganga Overview - J.T.Moscicki 17 Enabling Grids for E-sciencE INFSO-RI Remote Registry Application Manager Job Manager Remote Registry Client Keeps track of jobs Is a “passive” data store, typically using a database backend Keeps a roaming profile of the user jobs Keeps track of the job status May cache the output of a job in a local SE. This component is the most likely to be run on a separate host.

Ganga Overview - J.T.Moscicki 18 Enabling Grids for E-sciencE INFSO-RI Ganga Release Status Old series (2.x,3.x) –released regularly in 2004 until March 2005 –integrated wxPython GUI, all basic use-cases implemented for single-client mode and local registry Ganga4 –first public release June, in alpha testing now (alphas released weekly) –completely re-engineered around GPI –integrated LHCb and Atlas/ADA applications –GUI comes later

Ganga Overview - J.T.Moscicki 19 Enabling Grids for E-sciencE INFSO-RI Summary Factsheet (4-0-0-alpha4) –size:  Ganga base: ~300KB, pure-python (no install)  Atlas extensions: ~50KB  LHCb extension: ~50KB –existing functionality:  basic job manipulation via GPI  easy configuration / extension  local and remote registry –planned functionality:  splitting/merging  remote application and job manager  monitoring component

Ganga Overview - J.T.Moscicki 20 Enabling Grids for E-sciencE INFSO-RI