1. K. Harrison Liverpool, 14th September 2004 GANGA DEVELOPMENT - Objectives and organisation - Ganga in ATLAS and LHCb - Release status - Work in progress - Conclusions

2. 14th September 20042 Main objectives - Aim to help physicists in ATLAS and LHCb carry out studies in the complex computing environment of LHC Help configure applications developed within the Gaudi/Athena framework shared by ATLAS and LHCb Deal with submitting and monitoring jobs to distributed (Grid) and local batch systems Help users to keep track of what theyve done Hide all technicalities: allow user to concentrate on the physics Provide a single user-friendly environment for all tasks Ganga: Gaudi/Athena and Grid Alliance - Ganga is the Hindi name for the Ganges, and is the river goddess who carried lost souls to salvation - Ganga software will carry jobs to the Grid underworld, and hopefully bring them back

3. 14th September 20043 Ganga: first-level decomposition GANGA UI Grid Services MDSvc Metadata Service WorkLoad Manager SE File catalog WLMProSvcMonitor Internal Model Profile Service GAUDI Program Instr. … CE

4. 14th September 20044 People - Ganga is being developed as an ATLAS/LHCb common project, with support in UK from GridPP - Current main contributors are: Development: K.Harrison, J.Moscicki, A.Soroko, C.L.Tan Technical direction: D.L.Adams (ATLAS), U.Egede (LHCb) GridPP coordination: R.W.L.Jones (ATLAS), G.N.Patrick (LHCb) - Additional contributions, past and present, from many others: N.Brook, P.Charpentier, K.Ciba, C.Collins-Tooth, F.Harris, A.Maier, J.Martyniak, P.Mato, W.Lavrijsen, G.Rybkine, C.E.Tull Appologies to anyone left out

5. 14th September 20045 Organisation - Ganga-related information regularly updated on project web site: http://ganga.web.cern.ch/ganga/ - Software status is tracked using Savannah: https://savannah.cern.ch/projects/ganga/ - Have two active mailing lists, with archives accessible via web site: General interest: project-ganga@cern.chproject-ganga@cern.ch Technical issues: project-ganga-developers@cern.chproject-ganga-developers@cern.ch - Usually have a telephone meeting once a week Times, agendas and minutes placed on web site and circulated to project-ganga@cern.ch mailing listproject-ganga@cern.ch - Give status reports at ATLAS Software Workshops, at LHCb Software Weeks, at GridPP meetings, and at various other meetings

6. 14th September 20046 ATLAS and LHCb: similarities and differences (1) - Two experiments use (almost) the same software framework: Gaudi/Athena User chooses alogrithms to run, and sets algorithm parameters, via job options LHCb uses text job options ATLAS starting to use Python job options

7. 14th September 20047 Gaudi architecture Converter Algorithm Event Data Service Persistency Service Data Files Algorithm Transient Event Store Detec. Data Service Persistency Service Data Files Transient Detector Store Message Service JobOptions Service Particle Prop. Service Other Services Histogram Service Persistency Service Data Files Transient Histogram Store Application Manager Converter Event Selector

8. 14th September 20048 ATLAS and LHCb: similarities and differences (2) - Main applications have similar scope: event generation, detector simulation, reconstruction, analysis - Both experiments use Configuration Management Tool (CMT) for code organisation, but with differences in how its used - Experiments have similar views on job building, but have different requirements for implementation: ATLAS favours use of Analysis Job Description Language (AJDL), a key ingredient of ATLAS Distributed Analysis (ADA) system LHCb happy with more-direct mapping to job scripts

9. 14th September 20049 Job building: LHCb DaVinci Workflow Select WorkFlow Prepare Sandbox Prepare AlgFlowOptions and DLLs DaVinci AlgorithmFlow Edit AlgorithmFlow AlgParamOptions Select Datasets Edit AlgParamOptions DatasetOptions AlgFlowOptions Sandbox DLLs JobOptions FileCatalog slice Submit Job Metadata catalog AlgOptions catalog DLLs File catalog

10. 14th September 200410 Job building: ATLAS

11. 14th September 200411 AJDL in outline - AJDL (Analysis Job Description Language) provides for high-level description of job content - Basic elements include: Application: name and version number Task: user configuration of application Dataset: data to be processed Preferences: resource requirements - AJDL aims to define way in which job information is transmitted between software components: Allow multiple clients to communicate with independently implemented services by defining a common interface - AJDL represents job data using XML DIAL provides C++ classes that interpret and manipulate the XML

12. 14th September 200412 Client-service communication via AJDL Initial ADA

13. 14th September 200413 ATLAS and LHCb: similarities and differences (3) - Experiments have different metadata catalogues ATLAS has AMI LHCb has bookkeeping database Different client tools for extracting information - Both experiments see Ganga as front-end to distributed analysis system: ATLAS interested in Ganga as front-end to web services with AJDL interface LHCb interested in Ganga as front-end to DIRAC Workload Management System

14. 14th September 200414 Ganga as front-end for ATLAS system From GANGA

15. 14th September 200415 Ganga as front-end for LHCb system DIRAC Job Management Service DIRAC Job Management Service DIRAC CE LCG Resource Broker Resource Broker CE 1 DIRAC Sites Agent CE 2 CE 3 Production manager Production manager GANGA UI User CLI JobMonitorSvc JobAccountingSvc AccountingDB Job monitor InfomarionSvc FileCatalogSvc MonitoringSvc BookkeepingSvc BK query webpage BK query webpage FileCatalog browser FileCatalog browser User interfaces DIRAC services DIRAC resources DIRAC Storage DiskFile gridftp bbftp rfio

16. 14th September 200416 ATLAS and LHCb: similarities and differences (4) - Similarities between ATLAS and LHCb are motivation for common project - Differences have led to a component-based design, capable of satisfying a wide range of requirements Core components of general applicability supplemented by experiment-specific components Approach works well for ATLAS and LHCb Ganga has also been used successfully in BaBar (J.Martyniak)

17. 14th September 200417 Ganga design: schematic Job Definition Job Registry Job Handling File Transfer Python Native Software Bus CLI GUI Py ROOT Gaudi Python PyCMT PyAMI PyDial BaBar Job Definition and Splitting Gaudi/Athena Job Options Editor Gaudi/Athena Job Definition

18. 14th September 200418 Ganga core classes JobHandler Requirements JobAttributes Credentials 1 1… Job JobsRegistry Application ApplicationHandler Parameter Executable 1 0… 1 1 1 1 1… Job definition component Job registry component Job handling component File FileHandler 0… 1 Configuration Options ExtraOptions 1 1 Application handling component 0, 1 1

19. 14th September 200419 Example Ganga handler classes JobHandler GridJobHandler PBSJobHandler LSFJobHandler Job handling component ApplicationHandler BaBarApplicationHandler GaudiApplicationHandler DaVinchiApplicationHandler AtlfastApplicationHandler Application specific components BatchJobHandler LocalJobHandler WMSJobHandler EDGJobHandler

20. 14th September 200420 Release history - Ganga 1.0 released in July 2003 Basic functionality implemented Tutorial given at US-ATLAS Computing and Physics Meeting, BNL, 27th August 2003, with about 50 participants Tutorial provided useful feedback to developers - Major overhaul and upgrade begun in September 2003 - Ganga 2.0 released May 2004, and followed by several minor releases Improved GUI, revised core classes, better stability, additional functionality Ganga 2.3 released 13th September 2004 - Ganga 3.0 scheduled for October 2004 Interfaces to experiment tools and services, user-friendly command-line interface

21. 14th September 200421 Ganga 2.x - Ganga 2.x is a working system: download and try! - Installation instructions and user guides available from Ganga web site: http://ganga.web.cern.ch/ganga/user/v2/QuickStart.htmlhttp://ganga.web.cern.ch/ganga/user/v2/QuickStart.html - Main features include: Graphical User Interface (GUI) Command echoing Job-options editor (JOE) Job submission to different batch systems (localhost, LSF, PBS) Job submission to Grid via DIRAC, in experimental form Example of aided application configuration, for LHCb analysis Interface to LHCb metadata catalogue Job catalogue Automatic monitoring

22. 14th September 200422 Job creation

23. 14th September 200423 Job definition

24. 14th September 200424 Examination of job output

25. 14th September 200425 Interface to LHCb metadata catalogue

26. 14th September 200426 Job-options editor: main features - Standalone Qt-based graphical editor, available from Ganga - Presents hierarchical view of available options, and helps user with value entry - Multiple job-options files may be edited simultaneously - Support for both text and Python job-options files - Files saved with included files expanded, so that result is portable

27. 14th September 200427 JOE: main window

28. 14th September 200428 JOE: option entry

29. 14th September 200429 Work in progress - Work in progress in several areas: AS: Job submission to DIRAC KH, CLT: Job submission via AJDL interface JM: User-friendly command-line interface CC-T, CLT: Interface to ATLAS metadata catalogue (AMI) KC: Output merger GR: Package-management tool WL: Upgrade of component manager (PyBus) - Last two arent strictly Ganga activities, but will benefit Ganga

30. 14th September 200430 Job submission to DIRAC - Job handler for submission to DIRAC is already available in Ganga 2.x, but testing is still in progress - Handler creates job-description files required by DIRAC - Job submission and monitoring performed making direct use of DIRAC API - Output files retrieved automatically when ready, in separate thread - Log file is checked for error messages, then job state is updated

31. 14th September 200431 Job submission via AJDL interface - Python client for AJDL services developed by producing bindings for DIAL C++ classes Generate LCG dictionaries using lcgdict Compile resulting _dict.cpp files against ReflectionBuilder to create libraries that can be imported into Python using PyLCGDict2 Procedure now largely automated - Have added set of Python functions to simplify the use of some of the classes For example, can submit job to remove DIAL server (GSI authentication) and query status with: import pydial pydial.submit(dataset.xml,task.xml,application.xml) pydial.showStatus()

32. 14th September 200432 Prototype GUI for AJDL job builder Working on appropriate handlers for connection to Ganga

33. 14th September 200433 Interface to ATLAS metadata catalogue - Python client for ATLAS metadata catalogue (AMI) developed - Allows use of main AMI methods for querying and updating catalogue - Checks for mandatory parameters are made before a command is sent to AMI - XML results returned by AMI are automatically converted to appropriate Python object

34. 14th September 200434 Prototype GUI for Ganga interface to AMI - Currently very simple - Allows user to select AMI project / processing step - User builds query - User selects LFNs and Ganga adds them to a job script

35. 14th September 200435 Output merger - Output merger is being developed as a standalone application, Josie (more a companion for JOE than an acronym), which will be available from Ganga - Core classes have been implemented - Currently, can produce macros for merging ROOT ntuples and histograms, concatenate text files, and perform text filtering - Functionality available via command line and GUI

36. 14th September 200436 User-friendly command-line interface - Ganga echoes commands executed when operations are performed through the GUI - Echoed commands can be entered directly at Python prompt, but can be cumbersome - Draft scheme for more-intuitive command-line interface has been written, and a prototype implementation is available

37. 14th September 200437 Conclusions - Differences between ATLAS and LHCb create challenges for Ganga, but these are being met - Significant Ganga-related activity in both experiments GridPP developers have key roles, but also others are making major contributions - Ganga release 2.x is a working system Installation instructions and user guide available from Ganga web site - Ganga 3.0, with submission to DIRAC and via AJDL interface, scheduled for October - LHCb-UK software course to be held at Cambridge in December Ganga will be used as tool for submitting jobs