1. Ganga development - Theory and practice - Ganga 3 - Ganga 4 design - Ganga 4 components and framework - Conclusions K. Harrison CERN, 25th May 2005

2. 25th May 20052 Ganga development: theory and practice (1) -Ganga started (2002) as a joint ATLAS-LHCb project for an integrated system to help with configuring Gaudi/Athena jobs and running them efficiently in a distributed environment  Deal with all phases of a job lifecycle  Hide Grid technicalities from the user -Priority has stayed the same, but aim for a generic solution  Simplify running any application on any backend  Interest shown by other experiments (BaBar and CMS), and by groups outside HEP -Project supported by GridPP (UK) and by ARDA -Development work coordinated by U.Egede

3. 25th May 20053 Ganga development: theory and practice (2) -Ganga team has developed Python tools providing functionality relevant to the ATLAS and LHCb computing environments, and a framework for using these tools  Also benefit from other Python-based work in the two experiments -LHCb involvement in Ganga has tended to be more pro-active than that of ATLAS  More input from ATLAS would be welcomed  Joint ATLAS-LHCb Ganga meeting included in agenda for current Software Workshop  Dedicated Ganga Workshop being held at Imperial College London, 13th-15th June

4. 25th May 20054 Ganga in ATLAS Ganga  User interface for job-related operations: configuration, submission, splitting, merging, monitoring, output retrieval, etc LSFGrid 3NorduGridPBSBQSOtherLCG DIAL Production System Condor Other catalogues and repositories DIAL catalogues and repositories ATLAS Metadata Interface (AMI)

5. 25th May 20055 Ganga 3 -Ganga 3.0 released 2nd March 2005  User guide available on Ganga web site: http://ganga.web.cern.ch/ganga/ http://ganga.web.cern.ch/ganga/ -A lot of useful functionality included  Job configuration and management through Command Line Interface in Python (CLIP) or through GUI  Support for Athena applications  Job-Options Editor (JOE)  Access to AMI  Submission to backends including PBS, LSF, LCG, gLite -Some issues with extensibility and performance  Problems addressed in redesign for Ganga 4

6. 25th May 20056 Ganga 3: graphical interfaces

7. 25th May 20057 Ganga 4: Design Command Line in Python (CLIP)Graphical User Interface (GUI) Ganga Public Interface (GPI) Job Manager Catalogue interfaces Toolkit - Job-Options Editor - Data management - ROOT via pyROOT - Other Backend plug-insJob-Description plug-ins External catalogues (AMI, DIAL, other) Job Repository Switchboard plug-ins ADA Athena Other LCG DIAL

8. 25th May 20058 Job Repository (A.Soroko) -Job repository allows Ganga job representations to be stored and subsequently retrieved -Repository can be local to the user, or can be on a remote server  API in the two cases is the same -Both local and remote implementations use tools from ARDA Metadata Catalogue Project (B.Koblitz) -Local repository is a file-type database -For remote repository, server side supports various types of database (mysql, postgres, etc) -Local implementation and client for remote repository require only Python modules

9. 25th May 20059 Job Repository: API -The API is partly inspired by CVS, providing possibilities to commit, checkout and update jobs -Additional methods allow retrieval/modification of job status without checkout of the full job, and allow job selection based on some criteria -API supports bulk operations, to optimise communication with the underlying database

10. 25th May 200510 Job repository: performance usersjobscommitcheckoutdelete 1100.4707000.1083000.019000 1500.2998200.0943600.015800 11000.3577200.0629800.010820 5100.2820000.0558800.026040 5500.1902800.0959400.021432 51000.2228800.0872060.018146 10 0.4986200.0862200.048350 10500.3198180.1029280.030642 101000.3503620.1413840.028472 40100.3266000.0936280.028412 40500.6771250.1822970.081015 401001.0869550.2527260.209584 Time in seconds per job for requests to remote repository

11. 25th May 200511 Interface to AMI (C.L.Tan) -pyAMI written as a Python client for the AMI web service  Uses ZSI v1.5 and PyXML v0.8.3 -Allows use of main AMI methods for querying and updating catalogue -Checks made for mandatory parameters before a command is sent to AMI -XML results returned by AMI automatically converted to appropriate Python object -Positive feedback from S.Albrand: suggestions to be taken into account in future development

12. 25th May 200512 PyAMI example (1) -Log in to the web service >>> from pyAMI import * >>> AC = pyAMI.AMI_WS_Client( '__myID__', '__myPassword__' ) -Obtain list of recognised commands >>> print AC.getCommands().getResults() [[GetVersion], [GetCommands], [ListProjects], [ListProcesses], [ListEntities], [SearchQuery], [AddElement], [RemoveElement], [UpdateElement], [ListEntityProperties], [ListElement ], [AddDataset], [RemoveDataset], [UpdateDataset], [ListDataset], [ListDatasetProvenance], [AddPartition], [RemovePartition], [UpdatePartition], [ListPartition]]

13. 25th May 200513 PyAMI example (2) -List properties for the entity ‘dataset’ >>> xmlResult = AC.listEntityProperties( 'dataset' ) >>> resultWebPage = xmlResult.toHTML( True )

14. 25th May 200514 Interface to DIAL (K.Harrison) -PyDial has been developed as a Python client for DIAL -Currently build PyDial on top of a DIAL release, using lcgdict to generate dictionaries for DIAL C++ classes  Dictionaries imported into Python using PyLCGDict2  Manual checking for completeness needed for each new DIAL release, but could try to automate this  After checking, build performed in 5-10 minutes -PyDial additionally provides Python functions that simplify use of the DIAL classes -In longer term, might aim for a pure Python implementation (less essential if DIAL becomes easier to deploy)

15. 25th May 200515 PyDial functionality -PyDial allows access to all DIAL services and catalogues -Have been able to run test jobs successfully using services at both CERN and BNL -Histograms included in job result can be displayed from inside Python using pyROOT

16. 25th May 200516 Interface to Production System (F.Brochu) -Not strictly developed as a Ganga component, but CreateRecords Python module should readily allow Ganga to submit jobs to the Production System Ganga

17. 25th May 200517 Ganga 4 framework (J.T.Moscicki) -Ganga framework takes care of managing components and making their functionality available through the Ganga Public Interface (GPI) -Infrastructure is in place, and have plug-ins for a small number of job-description types and backends  Still some optimisation needed, but already works well  Once final details are sorted out (next month or so), addition of plug-ins for other job-description types and backends will be very rapid -As ATLAS example have ADA/DIAL job description and DIAL backend -As LHCb example (A.Maier) have Gaudi job description and LSF backend

18. 25th May 200518 Working with GPI (1) -Functionality in example available in Ganga 4 developers’ release, although current syntax is slightly different >>> j = AdaJob() >>> j.application = “atlasopt” >>> j.task = “atlas_release jo.py” >>> j.dataset = “hma.dc2.003007.digit.A1.z_ee.aod-904.10files” >>> j.backend.schedulerURL = “lxgate21.cern.ch.20014” >>> j.submit() >>> print j.status >>> print j.result >>> j.copyResult( “myDirectory” ) - Have shown commands given at Python prompt  Same commands can be used in a script

19. 25th May 200519 Working with GPI (2) -Relevant plug-ins not yet available, but would expect to be able to do something like the following >>> j = AthenaJob() >>> j.release = “10.0.3” >>> j.jobOptions = “myOpts.py” >>> j.cmtRequirements = “myRequirements” >>> j.backend = “LCG” >>> j.submit()

20. 25th May 200520 Ganga 4: graphical interfaces -Priority for Ganga 4 has been to have infrastructure in place, and to have functionality available at the command line  Ganga 4 public release planned for June, without GUI -Subsequently expect work on graphical job builder and monitoring for ADA (C.L.Tan) to be directly reused in Ganga

21. Graphical job builder

22. Job monitoring

23. 25th May 200523 Conclusions -Excellent progress made with Ganga 4 development  Infrastructure in place and first plug-ins written  Public release (without GUI) planned for June -Ganga project has produced Python tools for interfacing to different components of the ATLAS computing system, and a framework for using these (and other) tools -Dedicated Ganga Workshop being held at Imperial College London, 13th-15th June  Significant ATLAS participation encouraged  All input on priorities welcome