1. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Introduction to GANGA Hurng-Chun Lee 27 Feb 2007, Innsbruck

2. EGEE-II INFSO-RI-031688 Outlines Ganga overview Using Ganga Developing application plugins within Ganga More about Ganga

3. EGEE-II INFSO-RI-031688 Considerations of scientific application development Computing environment is heterogeneous Computing technology is evolving User requirement is also evolving Application users prefer to learn as few as possible the tools which are light-weight, handy and well-integrated. Ganga tries to answer the questions: –How to minimize developer’s effort in supporting user applications? –How to minimize user’s effort in running applications?

4. EGEE-II INFSO-RI-031688 What’s Ganga? GANGA: Gaudi/Athena and Grid Alliance – jointly developed by ATLAS and LHCb experiments – Nevertheless, it’s application neutral A common framework extensible for not only HEP applications An easy-to-use front-end for job definition and management A light-weight application component fully implemented in Python

5. EGEE-II INFSO-RI-031688 Ganga Job Where the Ganga journey starts … Mandate Optional

6. EGEE-II INFSO-RI-031688 Plug-in based design Common interface Specific implementation ✦ Ease user’s experience in switching between different technologies ✦ Concentrate developer’s effort in specific domain

7. EGEE-II INFSO-RI-031688 Applications & backends

8. EGEE-II INFSO-RI-031688 Ganga development Ganga development team – F.Brochu (Cambridge), U.Egede (Imperial), J.Elmsheuser (München), K.Harrison (Cambridge), H.C.Lee (ASGC), D.Liko (CERN), A.Maier (CERN), J.T.Moscicki (CERN), A.Muraru (Bucharest), V.Romanovsky (IHEP), A.Soroko (Oxford), C.L.Tan (Birmingham) Release status – current stable release branch - 4.2.x – experimental beta release branch - 4.3.x-beta

9. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Using Ganga

10. EGEE-II INFSO-RI-031688 Download, Install, First launch wget http://ganga.web.cern.ch/ganga/download/ganga-install python ganga-install \ --prefix=~/opt/ganga \ --extern=GangaAtlas,GangaGUI,GangaPlotter \ 4.2.8 Download & Install export PATH $HOME/opt/ganga/install/slc3_gcc323/4.2.8/bin:$PATH ganga -g ganga -o’[LCG]ENABLE_EDG=False’ -o’[LCG]ENABLE_GLITE=False’ *** Welcome to Ganga *** Version: Ganga-4-2-8 Documentation and support: http://cern.ch/ganga Type help() or help('index') for online help. In [1]: Do you really want to exit ([y]/n)? First Launch download installer installation prefix Installation of external modules Ganga version generate configuration file start Ganga with inline configurations Ganga CLIP -D to exit Ganga CLIP

11. EGEE-II INFSO-RI-031688 Get your hands dirty... Job().submit() submit and run a test job on local machine Job(backend=LCG()).submit() submit and run a a test job on LCG jobs browse the created jobs (job history) j = jobs[1] get the first job from the job history j print the details of the job and see what you can set for a job j.copy().submit() make a copy of the job and submit the new job j. see what you can do with the job

12. EGEE-II INFSO-RI-031688 Configurations [Configuration] TextShell = IPython... [LCG] EDG_ENABLE = True... Syntax Hardcoded configurations export SITE_CONFIG_PATH = /some/physics/subgroup.ini:GangaLHCb/LHCb.ini ganga --config-path=/some/pyhsycis/subgroup.ini:GangaLHCb/LHCb.ini ~/.gangarc ganga -o How to set configurations user config > site config > release config Sequenc e Python ConfigParser standard release config site config user config

13. EGEE-II INFSO-RI-031688 The ‘gangadir’ created at the first launch [DefaultJobRepository] local_root = /alternative/gangadir Metadata of jobs Data of jobs

14. EGEE-II INFSO-RI-031688 *** Welcome to Ganga *** Version: Ganga-4-2-8 Documentation and support: http://cern.ch/ganga Type help() or help('index') for online help. In [1]: jobs Out[1]: Statistics: 1 jobs -------------- # id status name subjobs application backend backend.actualCE # 1 completed Executable LCG lcg- compute.hpc.unimelb.edu.au:2119/jobmanage CLIP User interfaces GUI #!/usr/bin/env ganga #-*-python-*- import time j = Job() j.backend = LCG(middleware=’GLITE’) j.submit() while not j.status in [‘completed’,’failed’]: print(‘job still running’) time.sleep(30)./myjob.exec ganga./myjob.exec In [1]:execfile(“myjob.exec”) GPI & Scripting

15. EGEE-II INFSO-RI-031688 Some handy functions completion for cmd history system command integration Job template In[1]: plugins() – plugins(‘backends’) In[2]: help() etc. In[1]: j = jobs[1] In[2]: cat $j.outputdir/stdout Hello World In[1]: t = JobTemplate(name=’lcg_simple’) In[2]: t.backend = LCG(middleware=’EDG’) In[3]: templates Out[3]: Statistics: 1 templates -------------- # id status name subjobs application backend backend.actualCE # 3 template lcg_simple Executable LCG In[4]: j = Job(templates[3]) In[5]: j.submit()

16. EGEE-II INFSO-RI-031688 j = Job() j.application = Athena() j.application.option_file = ‘myOpts.py’ j.application.prepare(athena_compile = False) j.inputdata = DQ2Dataset() j.inputdata.dataset = ‘interestingDataset.AOD.v12003104’ j.inputdata.type = ‘DQ2_Local’ j.outputdata = AthenaOutputDataset() j.outputdata.outputdata = ‘myOutput.root’ j.splitter = AthenaSplitterJob(numsubjobs=2) j.merger = AthenaOutputMerger() j.backend = LCG( CE=’ce102.cern.ch:2119/jobmanager-lcglsf-grid_2nh_atlas’ ) j.submit() CLIP mode Real Application: the ATLAS data analysis application application inputdata OutputdataSplitter & Merger Scripting mode

17. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Developing application plugins within Ganga

18. EGEE-II INFSO-RI-031688 Behind the scene...

19. EGEE-II INFSO-RI-031688 Application plugin generic logic specific backend binding Application plugin A good example: python/Ganga/Lib/Executable/Executable.py

20. EGEE-II INFSO-RI-031688 A good example python/Ganga/Lib/Executable/Executable.py snapshot of the “Executable” application – highlight the library import, inheritance of IApplication and IRuntimeHandler, Handler to backend mapping – schema

21. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks More about Ganga

22. EGEE-II INFSO-RI-031688 More features on LCG jobs The Ganga LCG plugin tries the best to improve the LCG usability by – making new LCG features transparent to users (e.g. supporting more efficient gLite bulk submission) – making sure that the user can always get job’s stdout/err if the job failure comes from the user code. – tracing the runtime progress of the grid job. – pre-staging oversized inputsandbox to a temporary cache on the grid storage element.

23. EGEE-II INFSO-RI-031688 More than HEP applications Job statistics on 2580 grid jobs from Ganga

24. EGEE-II INFSO-RI-031688 More than job submission: Monitoring & Accounting submission tool

25. EGEE-II INFSO-RI-031688 More info. Ganga Home: http://cern.ch/gangahttp://cern.ch/ganga Official Ganga User’s Guide: http://ganga.web.cern.ch/ganga/user/html/GangaIntroduction/http://ganga.web.cern.ch/ganga/user/html/GangaIntroduction/ Tutorial for ATLAS data analysis using Ganga: https://twiki.cern.ch/twiki/bin/view/Atlas/DistributedAnalysisUsingGanga https://twiki.cern.ch/twiki/bin/view/Atlas/DistributedAnalysisUsingGanga Looking for helps: – ATLAS user support: hn-atlas-GANGAUserDeveloper@cern.chhn-atlas-GANGAUserDeveloper@cern.ch – direct support from developers: project-ganga-developers@cern.chproject-ganga-developers@cern.ch

26. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Thank you!