Ganga Developments Karl Harrison (University of Cambridge) 18th GridPP Meeting University of Glasgow, 20th-21st March

20 March 20072/15 Ganga basics Depending on context, Ganga can be any of: (A) a Hindu goddess (B) an hallucinogenic drug (C) a job-management framework (Gaudi/Athena and Grid Alliance) that simplifies running jobs on the Grid Anyone expecting a presentation on (A) or (B) is probably at the wrong meeting

20 March 20073/15 Ganga releases Ganga has well-developed release procedure, with built-in quality checks (1) Code committed to CVS by package maintainer(s) (2) Packages are tagged for release, with changes documented (3) Suite of over 100 tests is run, with results available on web (4) Release made, using purpose-written tool Ganga released October 2006, with subsequent minor releases Ganga at beta testing phase, with public release foreseen for April

20 March 20074/15 Core development Remote repository Job information stored onremote server Certificate-based authentication Easier access to job files Locating job files in Ganga workspace can quickly become tiresome File access simplified by peek() method j.peek( “stdout” ) Validity of Grid proxy and AFS token checked in monitoring loop User prompted to renew credential when validity falls below configurable minimum Monitoring loop Credential monitoring Usage monitoring Information sent to MonALISA repository every time Ganga is started

20 March 20075/15 Developments for LHCb Logical file names for input data can be used for jobs run on local batch system as well as for Grid jobs –Run exactly the same job locally and on the Grid Added possibility of selecting input data using dataset browser New type of job splitting implemented, allowing creation of subjobs that differ from one another in terms of one or more job option Introduced possibility to set CPU limit for DIRAC jobs –Take advantage of short queues not used by production jobs Migrated to new system for setting environment for LHCb applications Example Ganga scripts for running LHCb applications now included in LHCb software releases

20 March 20076/15 Developments for ATLAS Support provided for both legacy and current datasets –Latest (DQ2) datasets can be used both for jobs run on local batch system and for Grid jobs Introduced possibility of fast preselections based on TAG analysis –Also have first version of interface to TAG Navigation Tool User allowed to choose between shipping binaries and compiling code at worker node Added job monitoring based on ARDA dashboard technology Extended range of backends for ATLAS applications: NorduGrid and PANDA for analysis, local resources for user production Introduction to Ganga included in ATLAS Workbook –Workbook is main point of entry for physicists wanting to learn to use ATLAS software

20 March 20077/15 Applications and backends for ATLAS and LHCb PBSOSGNorduGridLocalLSFPANDA US-ATLAS WMS LHCb WMS Executable Athena (Simulation/Digitisation/ Reconstruction/Analysis) AthenaMC (Production) Gauss/Boole/Brunel/DaVinci (Simulation/Digitisation/ Reconstruction/Analysis) LHCbExperiment neutralATLAS Available in Ganga 4.2 Work in progress New in Ganga 4.3

20 March 20078/15 LHCb Ganga Tutorials 3rd LHCb-UK Software Course held in Edinbugh, January 2007 Like previous editions, course provided overview of LHCb applications and running them on the Grid, with focus on physics analysis –Very successful approach Ganga, largely a demo tool in 1st LHCb-UK Software Course, is now a key component –Participants were introduced to Ganga on first day, then used it throughout the course to build code and run jobs Online LHCb analysis tutorial now based on use of Ganga 3rd LHCb-UK Software Course, Edinburgh, January 2007

20 March 20079/15 ATLAS Ganga Tutorials Upsurge of interest in Ganga in ATLAS, following good publicity in presentation (J.Catmore) on use of Ganga by B-physics group Six ATLAS Distributed Analysis Tutorials held since September –Milan (September 2006, February 2007), CERN (October 2006), Mumbai (December 2006), Edinburgh (February 2007), Lyon (March 2007) Ganga is main focus, but tutorials also introduce DQ2 data- management system, metadata catalogue and TAG navigation tool ATLAS Distributed Analysis Tutorial, Edinburgh, February 2007

20 March /15 User survey User survey carried out in December 2006, with responses from 27 physicists: 12 from ATLAS, 15 from LHCb User types ATLAS LHCbATLAS Backends used Mode of use ATLASLHCb Country distribution

20 March /15 Feedback from user survey Favourite features: top 4 Most-wanted features: top 3 Documentation and user support Job splitting Ease of use Data management Monitoring ATLAS General job handling Job splitting Grid/Local switching All functionality LHCb Enhanced/easier data management Faster response Job resubmission ATLAS Output merging Control over output destination Additional applications LHCb Web-based documentation and tutorials good for getting started Useful to have more examples and FAQ High level of support from developers (locally or via ) very much appreciated

20 March /15 Usage monitoring MonALISA monitoring shows more than 300 Ganga users since beginning of year Typically have users per work day (peaks for tutorials)

20 March /15 Ganga beyond ATLAS and LHCb: Geant 4 Regression tests run for Geant 4 major releases - twice per year –Search for differences in calorimeter observables –Need a few years of CPU, concentrated in short period Geant 4 team now runs these tests on the Grid, using Ganga in conjunction with the DIANE job-distribution framework

20 March /15 Ganga beyond ATLAS and LHCb: Cambridge Ontology Cambridge Ontology is developing solutions for image retrieval based on analysis of image content –combines state-of-the-art computer vision with machine learning, natural-language processing, and information-retrieval methods Large amounts of CPU time needed for image processing Mini-PIPSS project set up to enable processing to be done on the Grid Ganga is used for the job submission -Specialised Ganga components for Cambridge Ontology applications

20 March /15 Conclusions Continued excellent progress with core and experiment-specific Ganga developments Number of successful tutorials held over past six months –Demand for tutorials especially high in ATLAS Feedback from user survey has been very positive MonALISA monitoring shows that Ganga has had more than 300 users since the beginning of the year Ganga used to good effect in applications beyond ATLAS and LHCb –Geant 4 regression tests, Cambridge Ontology image processing Clouds on the horizon with anticipated loss of members of Ganga development team at end of GridPP2 –Lose vital expertise just when LHC is about to start taking data and user support is essential