K. Harrison CERN, 3rd March 2004 GANGA CONTRIBUTIONS TO ADA RELEASE IN MAY - Outline of Ganga project - Python support for AJDL - LCG analysis service - Task Editor and Job-Options Editor - Conclusions

3rd March Ganga project: aims - Project set up to provide a front end for performing physics studies in the computing environment of LHC: distributed datasets in the petabyte range; distributed resources; complex, highly configurable software  Help configure applications developed within the Gaudi/Athena framework shared by ATLAS and LHCb  Deal with submitting and monitoring jobs on distributed (Grid) and local batch systems  Help users to keep track of what they’ve done  Hide all technicalities: allow user to concentrate on the physics  Provide a single user-friendly desktop environment for all tasks  Ganga: Gaudi/Athena and Grid Alliance

3rd March Ganga project: organisation - Ganga is an ATLAS/LHCb common project, with support in UK from GridPP  Interest shown by other experiments, particularly BaBar, which is now contributing to the development - Developers: K.Harrison, J.Martyniak, A.Soroko, C.L.Tan - Experiment contacts: D.Adams (ATLAS), U.Egede (LHCb) - GridPP coordination: R.W.L.Jones (ATLAS), G.N.Patrick (LHCb) - Ganga-related information regularly updated on web site: - A mailing list has been set up: - Have semi-regular telephone meeting  Details of times placed on web site and circulated to mailing list

3rd March Ganga project: software - Ganga prototype, addressing all of the main project aims, available from Gaudi CVS repository since summer 2003  Written in Python - Prototype provides an example GUI, and is successful in a number of areas:  allows transparent access to different batch systems (localhost, LSF, PBS, EDG)  demonstrates job tracking, from job creation to output collection  introduces Job-Options Editor (optimised for Atlfast) - Work since September has dealt with making the system more modular, on improving the reliability, and on extending the functionality - Moving to client-service model, consistent with ADA design

3rd March Job creation with Ganga GUI

3rd March Python support for AJDL - Ganga (Python) client and services will support Analysis Job Definition Language, so that they may be used interchangeably with other clients and services in ADA

3rd March Using AJDL from Python (1) - Two approaches to supporting AJDL from Python: 1) Provide Python bindings for C++ implementation of DIAL  Advantages: - Makes use of work already done - Little maintenance required (on Python side) - Demonstrates possibility to use Python with non-Python components  Disadvantage: - Lose portability: rely on significant number of shared libraries, and must recompile on each platform

3rd March Using AJDL from Python (2) 2) Write Python implementation of AJDL  Advantages: - Portable  Disadvantages: - Work has to start from scratch - Code must be maintained, to reflect any future changes in AJDL - Ideally, probably want both solutions:  Python bindings for C++ implementation to give most complete functionality  Minimal Python implementation to allow portability

3rd March Binding DIAL C++ implementation of AJDL (1) - For May, will provide Python bindings for DIAL C++ implementation of AJDL - From outside Python, use lcgdict command, provided by SEAL, to generate LCG dictionaries from C++ class header files - Compile dictionary files (suffix _dict.cpp) to produce shared-object libraries - From inside Python, use PyLCGDict from SEAL to load libraries and access classes

3rd March Binding DIAL C++ implementation of AJDL (2) - Dictionaries and libraries successfully created for DIAL AJDL classes  Thanks to M.Marino for help with PyLCGDict - Have been able to create AJDL objects from Python/Ganga, and have tested some simple methods  Will undertake more-extensive testing, to ensure everything’s working  Will move soon from PyLCGDict to PyLCGDict2

3rd March AOD analysis on local batch system - User workflow for running an AOD analysis with Athena on a local batch system might typically include the following steps: 1) Write code (Athena algorithm) 2) Specify job options 3) Select dataset 4) Provide job-execution script 5) Submit job to batch system, for example using: bsub < myJobScript 6) Wait for job to complete, then look at output

3rd March AOD analysis on distributed system - User workflow for running AOD analysis with distributed resources and datasets should be no more complicated than for analysis on local batch system - In ADA, steps 1) to 4) of workflow become as follows;  Steps 1) and 2)  Define task  Step 3)  Define dataset  Step 4)  Define application - Client tools should help with these, and should perform actual job submission - Analysis service should take care of getting the job to run

3rd March LCG analysis service - Work on LCG analysis service is still at an early stage - Have experimented with running jobs under LCG1  Mechanics of process understood - Tests have been for Gaudi-based LHCb analysis jobs  Situation when running Athena-based jobs should be similar - Next steps: 1) Formulate job request in terms of AJDL 2) Provide Python component that will translate an AJDL request into a JDL file and job script that may be submitted to LCG 3) Convert Python component into a web service

3rd March Task Editor and Job-Options Editor (1) - For an AOD analysis using Athena, a task will typically be defined by some combination of: 1) user code 2) pointers to user code and/or libraries 3) job options - Task Editor (TEd) must allow user to examine/modify all of these

3rd March Task Editor and Job-Options Editor (2) - Work on part dealing with job options is progressing well  Job-Options Editor (JOE) under development by C.L.Tan  Editor will present hierarchical view of available options, and will help user with value entry  Work currently focused on compiling database of job-options information, ideally to contain option descriptions and valid ranges provided by algorithm authors  Have tools for automated extraction of job options and default values from algorithm libraries: don’t only rely on completeness of job options in text/Python files - For May, expect something fairly basic for dealing with other parts of task, but should at least be able to list any files referenced, and open a text editor for examining user code

3rd March Conclusions - Python bindings for DIAL C++ implementation of AJDL have been successfully generated  Have been able to create AJDL objects from Ganga/Python; more extensive testing to start soon - Have experimented with running jobs under LCG1  Steps towards providing LCG analysis service are well defined - Work on Job-Options Editor is progressing well  Basic implementation of Task Editor will follow  Expect Ganga contributions to ADA to be delivered on time for May release