1. Grid Operations Procedures
AEGIS Training for Site Administrators
Institute of Physics Belgrade, Dec 2008
Antun Balaz
Institute of Physics Belgrade
Serbia
The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no

2. Overview
SEE-GRID operational and monitoring tools (and their relation to EGEE tools)
HGSM/GOCDB
Helpdesk/GGUS
BBmSAM/SAM
GStat
Nagios/CIC portal
Accounting portal
Downtime procedures
Upgrade procedures
Grid-Operator-On-Duty (GOOD)
Service Level Agreement (SLA)

3. Operational & monitoring tools
GSTAT
(Taiwan)
HELPDESK
VOMS
BDII
SAM
Accounting
NAGIOS
BBmSAM
R-GMA
HGSM

4. HGSM/GOCDB (1)

5. HGSM/GOCDB (2)

6. HGSM/GOCDB (3)
Static database containing all relevant data about all SEE-GRID and AEGIS sites
Must be kept synchronized with the real situation
All sheets must be properly updated
Site Info
Contacts
Site Nodes
Downtimes
XML dumps – the easiest way to apply changes is to download XML dump of the data, edit it appropriately, and then upload the new XML file; this also allows keeping of backups

7. HGSM/GOCDB (4) The essential fields in HGSM:
GIIS URL
Monitoring: Yes
Status: certified
Type: seegrid_production, seegrid_certified, egee_production
Site Commitments
Contacts and administrators
All fields have to have correct values!
URL:

8. Helpdesk/GGUS (1)

9. Helpdesk/GGUS (2)

10. Helpdesk/GGUS (3)
Central reference point for tracking of all operational and user problems
Identified problems are reported through the Helpdesk and assigned to the appropriate supported
If problems cannot be solved within the SEE-GRID community, they are propagated to other projects/initiatives/support systems (e.g. GGUS)
URL:

11. BBmSAM/SAM

12. BBmSAM History

13. BBmSAM
Portal that provides access to the database of SAM tests results
Central tools for identification of operational problems
Should be checked by each site admin on a daily basis
Should be used to troubleshoot problems
Also provides SLA figures
URL:

14. GStat (1)

15. GStat (2)
Central tool for monitoring of the information system of SEE-GRID infrastructure
Provides useful data
Identifies problems with sites
Should be checked by each site admin on a daily basis and used for troubleshooting
Useful ldapsearch commands can be found on GStat pages!
URL:

16. Nagios/CIC portal (1)

17. Nagios/CIC Portal (2)

18. Nagios/CIC portal (3) Collection of alarms raised by various tools
The aim is to integrate all the tools and make the life of site admins and infrastructure managers easier
In the future, automatic creation of Helpdesk tickets will be implemented
URL:

19. Accounting portal (1) Accounting by site
Accounting by countries and institutions
Accounting by applications

20. EGEE Accounting portal

21. Accounting portal (2)
Collects the accounting data from all SEE-GRID and AEGIS sites through apel accounting publisher developed by the project
Provides aggregated accounting data by site, country, institution, application
Each site must publish the accounting data properly
URL:

22. Downtime procedures
Downtimes must be announced well in advance (1 week is reasonable time)
There are always downtime due to hardware etc. failures that cannot be anticipated
All downtimes must be entered properly in HGSM
That way they are not be counted against the site’s availability
In addition, all downtimes must be broadcasted by to the GIM, APP and proper VO mailing lists
Downtime should not exceed 10% of the total time (monthly, quarterly)
If yes, explanation must be provided
If the explanation is not accepted by the project management, SA1 claims will be rejected

23. Upgrade procedures
All upgrades/updates are announced over the GIM list
The broadcasts contain links to further instructions for upgrades for each Grid service
Site admins should carefully examine them before performing the update!
In addition, possible SEE-GRID-specific instructions are given in the
For especially important updates/changes, tickets are created for each site
For some upgrades/updates to be performed, downtimes may be required
OS updates must be regularly installed, to minimize security risks

24. Grid-Operator-On-Duty (GOOD)
Rotating shifts on a weekly basis
Each country’s GIM is responsible to monitor sites during his/her shift
Tickets are submitted to sites with problems, according to the status of sites in various monitoring tools (BBmSAM, GStat, Nagios, Accounting portal, etc.)
Older tickets that are not resolved are escalated
Support is given to sites that cannot resolve earlier identified operational problems
User tickets are assigned to the appropriate supporters
Wiki documentation is updated, or new wiki pages created if necessary
URLs:

25. Usual problems and links to (possible) solutions
BDII
siteBDII (GIIS) or top-level BDII is Unreachable
No info published CA
CA version test failed with error message: This CA is an old one and time allowed to upgrade is over
CE (Computing Element)
Job submission failed with error message: Brokerhelper: Cannot plan. No compatible resources:
Job submission failed with error message: Got a job held event, reason: Unspecified gridmanager error
Job submission failed with error message: Cannot read JobWrapper output, both from Condor and from Maradona
Job submission failed with error message: 7 authentication failed
Job submission failed with error message: 10 data transfer to the server failed
4444 Waiting jobs in the GRIS
SE (Storage Element)
File copy and registration failed with error message: FTPD GSSAPI error: GSS Major Status: General failure

26. Service Level Agreement (SLA)
Old URL:
The change to the current one is that the required availability is 80%, and that the availability is calculated on 3h basis, not on a daily basis
BBmSAM portal provides SLA figures
Sites not fully conforming to the SLA will have reduced funding
Sites with the availability <50% will be uncertified
Sites fully conforming to the SLA will be put into seegrid_certified status and become visible to the whole SEE region (i.e. not only SEE-GRID, but also EGEE-SEE etc.)

27. Service Level Agreement (SLA)
Currently:
AEGIS01-PHY-SCL: egee_production
AEGIS02-RCUB and AEGIS04-KG: seegrid_certified - good work!
AEGIS03-ELEF-LEDA: seegrid_production
AEGIS05-ETFBG: experimental (uncertified) – problem!
All AEGIS sites must improve their availability figures, so that we provide better service to our users