1. www.see-grid.eu SEE-GRID-2 The SEE-GRID-2 initiative is co-funded by the European Commission under the FP6 Research Infrastructures contract no. 031775 SEE-GRID operational tools and Grid services improvements Antun Balaz WP3 Leader Institute of Physics, Belgrade antun@phy.bg.ac.yu EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007

2. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 2 Overview SEE-GRID WP3 Infrastructure Operations Operational tools  HGSM, HGSM+SAM integration  WiatG  BBmSAM, BBmobileSAM WP3 ongoing work

3. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 3 SEE-GRID WP3 Develop the next-generation SEE-GRID infrastructure  Next generation of EGEE middleware (gLite) and services Support in deployment and operations of the Resource Centres  Monitoring, helpdesk, overall upgrade of infrastructure Network resource provision and assurance  in close cooperation with the SEEREN2 project  Bandwidth-on-Demand requirements CA and RA guidelines and deployment  catch-all Certification Authority (CA)  per-country CA deployment and User portal deployment and operations  P-GRADE

4. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 4 Infrastructure

5. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 5 Infrastructure status (1) SEE-GRID Core services  Catch-all Certification Authority  enables regional sites to obtain user and host certificates  Virtual Organisation Management Service (VOMS),  authorization system for the SEE-GRID Virtual Organisation (VO),  supporting groups and roles  Workload management service (lcg-RB and glite-WMSLB) and Information Services (BDII)  deployed several instances for failover  MyProxy is operational  supports certificate renewal  FTS deployed  used in production

6. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 6 Infrastructure status (2) SEE-GRID infrastructure contains currently the following resources:  31 sites in SEE-GRID production  5 sites in certification phase (AL + HR + 2 RO)  CPUs: ~950 total; Storage: 23.94 TB gLite assessment done, results positive, upgrade done on all sites (GLITE-3_0_2)  http://wiki.egee-see.org/index.php/SG_GLITE-3_0_2_Guide glite-CE deployed at several sites, assessment results inconclusive, service probably not stable enough for production glite-WMSLB deployed at several sites, assessment results show that it is not so stable as lcg-RB, but has various new features and is therefore actively used WN deployment closely follows latest developments of gLite:  http://wiki.egee-see.org/index.php/SL4_WN  http://wiki.egee-see.org/index.php/SL4_WN_glite-3.1

7. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 7 Operations

8. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 8 Operational procedures Distributed operations Pilot SLA established Monitoring and Accounting Tools Helpdesk tickets procedures  Generic support group for users  TPM-like (monitoring open tickets created by users, trying to solve the simple ones, route the tickets, etc.).  Country level user support groups  Associate with country level mailboxes  GOOD shifts introduced, initial results positive  Tickets handling: response times need to be improved! SEEGRID Wiki with detailed information for site administrators  http://wiki.egee-see.org/index.php/SEE-GRID_Wiki

9. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 9 SLA Conformance Improvements seen after the first quarter of pilot SLA enforcement

10. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 10 Operational & monitoring tools (1) Operational & monitoring tools deployment status  Hierarchical Grid Site Management (HGSM) – Turkey  Service Availability Monitoring (SAM) (+ porting to MySQL) – Bosnia and Herzegovina with CERN support  Helpdesk - Romania  BBmSAM - Bosnia and Herzegovina  GridICE – FYR of Macedonia  SEE-GRID GoogleEarth – Turkey + Gidoon Moont  SEE-GRID GoogleMaps - Turkey  Global Grid Information Monitoring System (GStat) – Min-Hong Tsai  Relational Grid Monitoring Architecture (R-GMA) – Bulgaria  Nagios - Bulgaria  Real Time Monitor (RTM) – Gidoon Moont and Turkey (HGSM)  MONitoring Agents using a Large Integrated Services Architecture (MonALISA) – Romania  What is at the Grid (WiatG) – CERN with support from Serbia

11. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 11 Operational & monitoring tools map HGSM HELP-DESK BDII R-GMA SAM GSTAT (Taiwan) GSTAT (Taiwan) VOMS RTM (UK) RTM (UK) Google maps Google maps BBmSAM GridICE MonALISA NAGIOS WiatG

12. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 12 Operational & monitoring tools (2) Integration status  HGSM+SAM, HGSM+BBmSAM  Automatic creation of list of sites to be tested  HGSM+BDII  Automatic creation of list of sites in the infrastructure  HGSM+GStat  Automatic creation of list of sites to be monitored  HGSM+RTM, HGSM+R-GMA  Automatic creation of list of sites monitoring and for accounting  VOMS+Helpdesk  Automatically create new user accounts when accessing helpdesk  Certificate based access to Helpdesk HGSM HELP-DESK BDII R-GMA SAM GSTAT VOMS RTM Google maps Google maps BBmSAM

13. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 13 HGSM database SEE-GRID GOCDB  Introduced as a lightweight version of GOCDB  Allows us to easily change its format when necessary and to adapt it to regional needs  Allows us to provide custom exports on demand, depending on operational tools/application developers Contains statical information about all sites Developed and maintained by TUBITAK-ULAKBIM, Turkey  https://hgsm.grid.org.tr/ Used by EUMedGRID, other regional projects expressed interest

14. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 14 HGSM+SAM integration has been done in collaboration between TUBITAK-ULAKBIM and U of Banjaluka Periodical export of HGSM data to XML file  XML if full dump of database and represents all relevant tables  Generated data is universal and can be used for other purposes Periodical import of HGSM data first to local MySQL DB then to Oracle XE SAM DB  Only SAM relevant data is imported into Oracle  Other data resides in local MySQL DB if needed for other use and not to burden Oracle DB HGSM+SAM Integration (1)

15. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 15 HGSM+SAM Integration (2) HGSM (MySQL) XML (PHP) Local copy of HGSM (MySQL) SAM DB (Oracle) BBmSAM (PHP) SAM portal (Python) BBmSAM (PHP) SAM sync (PHP) c01.grid.etfbl.netc16.grid.etfbl.net hgsm.grid.org.tr

16. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 16 HGSM – SAM – planned improvements  Currently SAM retrieves node/service from mix of different sources (the “official” way)  All the data is already present in HGSM  The intention is to communicate directly and only with HGSM as it is considered to be reference copy for data Having HGSM DB copy at the same place enables us to further develop (BBm)SAM portal  Checking whether someone is site administrator and allowing him/her to request out-of-order tests  Soft real-time tracking of test progress  Exporting data in any structured form – moving to XML and/or HGSM+SAM Integration (3)

17. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 17 WiatG: New BDII operations tool Web application for visualization of BDII information  http://bdii.phy.bg.ac.yu/WiatG/pl/WiatG.pl Highly responsive tool because it uses AJAX  Partial refresh (client receives part by part of the page)  Asynchronous (server processing in the background, so one may send several requests) Current version seeks for: CE, gCE, RB, gRB, SE, LFC, FTS and GridICE Used as an operational tool for site monitoring Documentation available:  http://wiki.egee-see.org/index.php/WiatG Supports several regional projects: EUMedGRID, EUChinaGrid, EELA, and BalticGrid, as well as LHC VOs and OPS

18. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 18 WiatG Architecture

19. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 19 WiatG in action

20. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 20 Further development of WiatG Addition of new services (MyProxy, localLFC, VO software tags, …) Correctness check of site-BDII data  Alarms dashboard  Automatic creation of tickets Development of the new tool “What should be at the Grid” (WsbatG)  Based on the site configuration exported from HGSM (SEE-GRID GOCDB)  Visually identical tool, providing the expected status of BDII in WiatG Comparison of WiatG and WsbatG data  Alarms dashboard  Automatic creation of tickets

21. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 21 BBmSAM portal  Created for SLA monitoring  Generating site availability statistics according to several criteria  Overview (HTML) and full dump (CSV) of data possible  Extended into full SAM portal  Availability for last 24h period for all sites/services  Latest results per service  History for nodes/services  Currently being ported to MySQL  Developed by U of Banjaluka  https://c01.grid.etfbl.net/ BBmSAM

22. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 22 BBmSAM as a SAM portal (1)

23. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 23 BBmSAM as a SAM portal (2)

24. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 24 BBmSAM and SLA (1)

25. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 25 BBmSAM and SLA (2)

26. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 26 BBmobileSAM Optimized for small-screen devices and low bandwidth Possible filtering of sites  For a single site (example: BA-01-ETFBL) http://c01.grid.etfbl.net/mobile.php?site=BA-01-ETFBL  For all sites in a country (example: BA) http://c01.grid.etfbl.net/mobile.php?site=BA  For all SEE-GRID sites http://c01.grid.etfbl.net/mobile.php Possible three levels of details  Basic level (critical test status for all nodes and services) http://c01.grid.etfbl.net/mobile.php?site=BA-01-ETFBL&details=0  Single test level (all tests status for all nodes and services) http://c01.grid.etfbl.net/mobile.php?site=BA-01-ETFBL&details=1  Single test level with timestamp http://c01.grid.etfbl.net/mobile.php?site=BA-01-ETFBL&details=2 Detail levels work independently of site filter, which means that http://c01.grid.etfbl.net/mobile.php?details=2 will produce detailed results for all sites in SEE-GRID

27. EGEE/WLCG Operations Workshop 2007, Stockholm, 13-15 June 2007 27 WP3 ongoing work Optimization of site/top-level BDIIs through indexing  http://wiki.egee-see.org/index.php/Fixing_BDII_response_time SAM porting to MySQL WiatG/WsbatG HGSM improvements gLite-WMSLB performance and stability assessment Proxy renewal on RB/WMS with full VOMS capabilities