1. EGEE is a project funded by the European Union under contract IST-2003-508833 “Interfacing to the gLite Prototype” Andrew Maier / CERN LCG-SC2, 13 August 2004 www.eu-egee.org http://cern.ch/arda cern.ch/lcg

2. Interfacing to gLite Prototype Andrew Maier 2 Contents Introduction GLite Handler for Ganga Running DaVinci in gLite Current status Back to Ganga

3. Interfacing to gLite Prototype Andrew Maier 3 Introduction LHCb within ARDA uses GANGA as main component. The LHCb/GANGA plans:  enable physicists (via GANGA) to analyze the data being produced during 2004 for their studies  It naturally matches the ARDA mandate  Deploy the prototype where the LHCb data will be the essential (CERN, RAL, …) At the beginning, the emphasis is to validate the tool focusing on usability, validation of the splitting and merging functionality for users jobs

4. Interfacing to gLite Prototype Andrew Maier 4 Testing the LHCb Bookkeeping DB Another ARDA activity (not covered today) is the stress test of the LHCb bookkeeping DB Tested behaviour of the DB for various parameters  Different number of clients  Different queries  Comparison with direct RPC call  …  See more in the near future

5. Interfacing to gLite Prototype Andrew Maier 5 GLite Handler for Ganga A prototype jobhandler for Ganga exists to submit jobs to gLite Uses an example DaVinci job on gLite Implemented outside most of the GANGA application- handler

6. Interfacing to gLite Prototype Andrew Maier 6 Running DaVinci in gLite Ganga uses an example DaVinci job which it submits to gLite This job consists of:  a small script to start DaVinci  A tar file which contains DaVinci and all its dependencies (Gaudi, LHCb, FieldMap, XmlDDDB)  A JDL which describes the executable (the starter script), the input files (the tar ball) and the output files (the hbook file) The script simply unpacks the tar ball copied into the sandbox by gLite, sets up the environment and runs DaVinci. The example job runs DaVinci on the default job-options file, which reads some data from CASTOR.

7. Interfacing to gLite Prototype Andrew Maier 7 First Steps The tar file with the DaVinci executable is created with the makedist script of the DataManagment software used by Dirac This script is recursively called for all dependencies. Then all is wrapped up in one big tar file The tar file is added to the gLite catalogue. Simple, but pragmatic approach

8. Interfacing to gLite Prototype Andrew Maier 8 This approach has some advantages:  It frees DaVinci from the CMT build environment  DaVinci can run in just about any environment  It is easy to implement But:  Everything is bundled in one big tar file (~250 MB)  Every job has a complete copy of DaVinci and its dependencies in the local sandbox  Updating is complicated

9. Interfacing to gLite Prototype Andrew Maier 9 Next Step Next step:  Use a package manager.  GLite provides a package manager based on packman Strategy:  Pack DaVinci and all its dependencies as separate packages  Use the package manager to install all packages and set up the environment  Start DaVinci from the gLite installed location and pass the appropriate job-options file.  Finally, add a user supplied algorithm and add it to the installed DaVinci environment

10. Interfacing to gLite Prototype Andrew Maier 10 gLite Package Manager Define a package:  A tar.gz file is put into the gLite catalogue as / /  A tag is added to define the package  A package can declare its dependencies as metadata  The package can contain an optional.gLiteEnvironment (bourne shell) script to set up its environment The only argument passed to the script is its location on where it is going to be installed Subpackages do not necessarily get installed in the same directory as the main package Dependent packages are installed and configured first

11. Interfacing to gLite Prototype Andrew Maier 11 Adapt the packing scripts The makedist script installs a setup script in the tar-file.  It assumes that the dependent packages are installed beforehand and that they are installed in the same location  A setup script is only created for applications (like DaVinci) and libraries (like LHCb), but not for databases (like FieldMap)  An application sets up its own environment and that of its dependents For gLite the strategy has to be changed:  Every package has to have its own setup script.  Each package only configures its own environment independent of its dependencies  Once all dependencies are resolved, the environment setup is complete

12. Interfacing to gLite Prototype Andrew Maier 12 Progress so far The makedist script has been changed to include gLite compatible setup scripts for libraries and applications  Discovered some bugs with the new compiler (gcc323) and some “missing” libraries in DaVinci v12r0 Next:  Need to add setup scripts for databases  Test outside gLite, simulating the package manager to test the setup scripts  Integrate into gLite  Run DaVinci from within gLite as an installed package  Finally add a user algorithm

13. Interfacing to gLite Prototype Andrew Maier 13 Coming back to Ganga Once the packaging works, a gLite jobhandler gets easier Ganga only has to supply  The (flattened) job-options file  The JDL file  (Optionally) the user algorithm  A small wrapper script to start DaVinci

14. Interfacing to gLite Prototype Andrew Maier 14 Conclusions Ganga has the ability to submit DaVinci jobs to gLite Simple analysis is possible in gLite  Stuart Paterson will be the first person to try this More advanced user friendly approaches are being developped Lots of work in progress More to come