EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Introduction to GANGA Hurng-Chun Lee 27 Feb 2007, Innsbruck

EGEE-II INFSO-RI Outlines Ganga overview Using Ganga Developing application plugins within Ganga More about Ganga

EGEE-II INFSO-RI Considerations of scientific application development Computing environment is heterogeneous Computing technology is evolving User requirement is also evolving Application users prefer to learn as few as possible the tools which are light-weight, handy and well-integrated. Ganga tries to answer the questions: –How to minimize developer’s effort in supporting user applications? –How to minimize user’s effort in running applications?

EGEE-II INFSO-RI What’s Ganga? GANGA: Gaudi/Athena and Grid Alliance – jointly developed by ATLAS and LHCb experiments – Nevertheless, it’s application neutral A common framework extensible for not only HEP applications An easy-to-use front-end for job definition and management A light-weight application component fully implemented in Python

EGEE-II INFSO-RI Ganga Job Where the Ganga journey starts … Mandate Optional

EGEE-II INFSO-RI Plug-in based design Common interface Specific implementation ✦ Ease user’s experience in switching between different technologies ✦ Concentrate developer’s effort in specific domain

EGEE-II INFSO-RI Applications & backends

EGEE-II INFSO-RI Ganga development Ganga development team – F.Brochu (Cambridge), U.Egede (Imperial), J.Elmsheuser (München), K.Harrison (Cambridge), H.C.Lee (ASGC), D.Liko (CERN), A.Maier (CERN), J.T.Moscicki (CERN), A.Muraru (Bucharest), V.Romanovsky (IHEP), A.Soroko (Oxford), C.L.Tan (Birmingham) Release status – current stable release branch x – experimental beta release branch x-beta

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Using Ganga

EGEE-II INFSO-RI Download, Install, First launch wget python ganga-install \ --prefix=~/opt/ganga \ --extern=GangaAtlas,GangaGUI,GangaPlotter \ Download & Install export PATH $HOME/opt/ganga/install/slc3_gcc323/4.2.8/bin:$PATH ganga -g ganga -o’[LCG]ENABLE_EDG=False’ -o’[LCG]ENABLE_GLITE=False’ *** Welcome to Ganga *** Version: Ganga Documentation and support: Type help() or help('index') for online help. In [1]: Do you really want to exit ([y]/n)? First Launch download installer installation prefix Installation of external modules Ganga version generate configuration file start Ganga with inline configurations Ganga CLIP -D to exit Ganga CLIP

EGEE-II INFSO-RI Get your hands dirty... Job().submit() submit and run a test job on local machine Job(backend=LCG()).submit() submit and run a a test job on LCG jobs browse the created jobs (job history) j = jobs[1] get the first job from the job history j print the details of the job and see what you can set for a job j.copy().submit() make a copy of the job and submit the new job j. see what you can do with the job

EGEE-II INFSO-RI Configurations [Configuration] TextShell = IPython... [LCG] EDG_ENABLE = True... Syntax Hardcoded configurations export SITE_CONFIG_PATH = /some/physics/subgroup.ini:GangaLHCb/LHCb.ini ganga --config-path=/some/pyhsycis/subgroup.ini:GangaLHCb/LHCb.ini ~/.gangarc ganga -o How to set configurations user config > site config > release config Sequenc e Python ConfigParser standard release config site config user config

EGEE-II INFSO-RI The ‘gangadir’ created at the first launch [DefaultJobRepository] local_root = /alternative/gangadir Metadata of jobs Data of jobs

EGEE-II INFSO-RI *** Welcome to Ganga *** Version: Ganga Documentation and support: Type help() or help('index') for online help. In [1]: jobs Out[1]: Statistics: 1 jobs # id status name subjobs application backend backend.actualCE # 1 completed Executable LCG lcg- compute.hpc.unimelb.edu.au:2119/jobmanage CLIP User interfaces GUI #!/usr/bin/env ganga #-*-python-*- import time j = Job() j.backend = LCG(middleware=’GLITE’) j.submit() while not j.status in [‘completed’,’failed’]: print(‘job still running’) time.sleep(30)./myjob.exec ganga./myjob.exec In [1]:execfile(“myjob.exec”) GPI & Scripting

EGEE-II INFSO-RI Some handy functions completion for cmd history system command integration Job template In[1]: plugins() – plugins(‘backends’) In[2]: help() etc. In[1]: j = jobs[1] In[2]: cat $j.outputdir/stdout Hello World In[1]: t = JobTemplate(name=’lcg_simple’) In[2]: t.backend = LCG(middleware=’EDG’) In[3]: templates Out[3]: Statistics: 1 templates # id status name subjobs application backend backend.actualCE # 3 template lcg_simple Executable LCG In[4]: j = Job(templates[3]) In[5]: j.submit()

EGEE-II INFSO-RI j = Job() j.application = Athena() j.application.option_file = ‘myOpts.py’ j.application.prepare(athena_compile = False) j.inputdata = DQ2Dataset() j.inputdata.dataset = ‘interestingDataset.AOD.v ’ j.inputdata.type = ‘DQ2_Local’ j.outputdata = AthenaOutputDataset() j.outputdata.outputdata = ‘myOutput.root’ j.splitter = AthenaSplitterJob(numsubjobs=2) j.merger = AthenaOutputMerger() j.backend = LCG( CE=’ce102.cern.ch:2119/jobmanager-lcglsf-grid_2nh_atlas’ ) j.submit() CLIP mode Real Application: the ATLAS data analysis application application inputdata OutputdataSplitter & Merger Scripting mode

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Developing application plugins within Ganga

EGEE-II INFSO-RI Behind the scene...

EGEE-II INFSO-RI Application plugin generic logic specific backend binding Application plugin A good example: python/Ganga/Lib/Executable/Executable.py

EGEE-II INFSO-RI A good example python/Ganga/Lib/Executable/Executable.py snapshot of the “Executable” application – highlight the library import, inheritance of IApplication and IRuntimeHandler, Handler to backend mapping – schema

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks More about Ganga

EGEE-II INFSO-RI More features on LCG jobs The Ganga LCG plugin tries the best to improve the LCG usability by – making new LCG features transparent to users (e.g. supporting more efficient gLite bulk submission) – making sure that the user can always get job’s stdout/err if the job failure comes from the user code. – tracing the runtime progress of the grid job. – pre-staging oversized inputsandbox to a temporary cache on the grid storage element.

EGEE-II INFSO-RI More than HEP applications Job statistics on 2580 grid jobs from Ganga

EGEE-II INFSO-RI More than job submission: Monitoring & Accounting submission tool

EGEE-II INFSO-RI More info. Ganga Home: Official Ganga User’s Guide: Tutorial for ATLAS data analysis using Ganga: Looking for helps: – ATLAS user support: – direct support from developers:

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Thank you!