Ganga development - Theory and practice - Ganga 3 - Ganga 4 design - Ganga 4 components and framework - Conclusions K. Harrison CERN, 25th May 2005

25th May Ganga development: theory and practice (1) -Ganga started (2002) as a joint ATLAS-LHCb project for an integrated system to help with configuring Gaudi/Athena jobs and running them efficiently in a distributed environment  Deal with all phases of a job lifecycle  Hide Grid technicalities from the user -Priority has stayed the same, but aim for a generic solution  Simplify running any application on any backend  Interest shown by other experiments (BaBar and CMS), and by groups outside HEP -Project supported by GridPP (UK) and by ARDA -Development work coordinated by U.Egede

25th May Ganga development: theory and practice (2) -Ganga team has developed Python tools providing functionality relevant to the ATLAS and LHCb computing environments, and a framework for using these tools  Also benefit from other Python-based work in the two experiments -LHCb involvement in Ganga has tended to be more pro-active than that of ATLAS  More input from ATLAS would be welcomed  Joint ATLAS-LHCb Ganga meeting included in agenda for current Software Workshop  Dedicated Ganga Workshop being held at Imperial College London, 13th-15th June

25th May Ganga in ATLAS Ganga  User interface for job-related operations: configuration, submission, splitting, merging, monitoring, output retrieval, etc LSFGrid 3NorduGridPBSBQSOtherLCG DIAL Production System Condor Other catalogues and repositories DIAL catalogues and repositories ATLAS Metadata Interface (AMI)

25th May Ganga 3 -Ganga 3.0 released 2nd March 2005  User guide available on Ganga web site: -A lot of useful functionality included  Job configuration and management through Command Line Interface in Python (CLIP) or through GUI  Support for Athena applications  Job-Options Editor (JOE)  Access to AMI  Submission to backends including PBS, LSF, LCG, gLite -Some issues with extensibility and performance  Problems addressed in redesign for Ganga 4

25th May Ganga 3: graphical interfaces

25th May Ganga 4: Design Command Line in Python (CLIP)Graphical User Interface (GUI) Ganga Public Interface (GPI) Job Manager Catalogue interfaces Toolkit - Job-Options Editor - Data management - ROOT via pyROOT - Other Backend plug-insJob-Description plug-ins External catalogues (AMI, DIAL, other) Job Repository Switchboard plug-ins ADA Athena Other LCG DIAL

25th May Job Repository (A.Soroko) -Job repository allows Ganga job representations to be stored and subsequently retrieved -Repository can be local to the user, or can be on a remote server  API in the two cases is the same -Both local and remote implementations use tools from ARDA Metadata Catalogue Project (B.Koblitz) -Local repository is a file-type database -For remote repository, server side supports various types of database (mysql, postgres, etc) -Local implementation and client for remote repository require only Python modules

25th May Job Repository: API -The API is partly inspired by CVS, providing possibilities to commit, checkout and update jobs -Additional methods allow retrieval/modification of job status without checkout of the full job, and allow job selection based on some criteria -API supports bulk operations, to optimise communication with the underlying database

25th May Job repository: performance usersjobscommitcheckoutdelete Time in seconds per job for requests to remote repository

25th May Interface to AMI (C.L.Tan) -pyAMI written as a Python client for the AMI web service  Uses ZSI v1.5 and PyXML v Allows use of main AMI methods for querying and updating catalogue -Checks made for mandatory parameters before a command is sent to AMI -XML results returned by AMI automatically converted to appropriate Python object -Positive feedback from S.Albrand: suggestions to be taken into account in future development

25th May PyAMI example (1) -Log in to the web service >>> from pyAMI import * >>> AC = pyAMI.AMI_WS_Client( '__myID__', '__myPassword__' ) -Obtain list of recognised commands >>> print AC.getCommands().getResults() [[GetVersion], [GetCommands], [ListProjects], [ListProcesses], [ListEntities], [SearchQuery], [AddElement], [RemoveElement], [UpdateElement], [ListEntityProperties], [ListElement ], [AddDataset], [RemoveDataset], [UpdateDataset], [ListDataset], [ListDatasetProvenance], [AddPartition], [RemovePartition], [UpdatePartition], [ListPartition]]

25th May PyAMI example (2) -List properties for the entity ‘dataset’ >>> xmlResult = AC.listEntityProperties( 'dataset' ) >>> resultWebPage = xmlResult.toHTML( True )

25th May Interface to DIAL (K.Harrison) -PyDial has been developed as a Python client for DIAL -Currently build PyDial on top of a DIAL release, using lcgdict to generate dictionaries for DIAL C++ classes  Dictionaries imported into Python using PyLCGDict2  Manual checking for completeness needed for each new DIAL release, but could try to automate this  After checking, build performed in 5-10 minutes -PyDial additionally provides Python functions that simplify use of the DIAL classes -In longer term, might aim for a pure Python implementation (less essential if DIAL becomes easier to deploy)

25th May PyDial functionality -PyDial allows access to all DIAL services and catalogues -Have been able to run test jobs successfully using services at both CERN and BNL -Histograms included in job result can be displayed from inside Python using pyROOT

25th May Interface to Production System (F.Brochu) -Not strictly developed as a Ganga component, but CreateRecords Python module should readily allow Ganga to submit jobs to the Production System Ganga

25th May Ganga 4 framework (J.T.Moscicki) -Ganga framework takes care of managing components and making their functionality available through the Ganga Public Interface (GPI) -Infrastructure is in place, and have plug-ins for a small number of job-description types and backends  Still some optimisation needed, but already works well  Once final details are sorted out (next month or so), addition of plug-ins for other job-description types and backends will be very rapid -As ATLAS example have ADA/DIAL job description and DIAL backend -As LHCb example (A.Maier) have Gaudi job description and LSF backend

25th May Working with GPI (1) -Functionality in example available in Ganga 4 developers’ release, although current syntax is slightly different >>> j = AdaJob() >>> j.application = “atlasopt” >>> j.task = “atlas_release jo.py” >>> j.dataset = “hma.dc digit.A1.z_ee.aod files” >>> j.backend.schedulerURL = “lxgate21.cern.ch.20014” >>> j.submit() >>> print j.status >>> print j.result >>> j.copyResult( “myDirectory” ) - Have shown commands given at Python prompt  Same commands can be used in a script

25th May Working with GPI (2) -Relevant plug-ins not yet available, but would expect to be able to do something like the following >>> j = AthenaJob() >>> j.release = “10.0.3” >>> j.jobOptions = “myOpts.py” >>> j.cmtRequirements = “myRequirements” >>> j.backend = “LCG” >>> j.submit()

25th May Ganga 4: graphical interfaces -Priority for Ganga 4 has been to have infrastructure in place, and to have functionality available at the command line  Ganga 4 public release planned for June, without GUI -Subsequently expect work on graphical job builder and monitoring for ADA (C.L.Tan) to be directly reused in Ganga

Graphical job builder

Job monitoring

25th May Conclusions -Excellent progress made with Ganga 4 development  Infrastructure in place and first plug-ins written  Public release (without GUI) planned for June -Ganga project has produced Python tools for interfacing to different components of the ATLAS computing system, and a framework for using these (and other) tools -Dedicated Ganga Workshop being held at Imperial College London, 13th-15th June  Significant ATLAS participation encouraged  All input on priorities welcome