1. EGEE Operation Tools and Procedures
M. Jouvin (LAL-Orsay)
Grid Administration Training
LAL, Orsay, September 2008, 15-19

2. Grid Operation Tools and Procedures - M. Jouvin
Agenda
Regional Operation Centers
Site registration
Site monitoring
Grid Support
Accounting
Site downtimes
Activity Visualization
Mailing List
Useful Links
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

3. Regional Operation Centers
EGEE is organized in 12 regions called ROCs
1 country or a group of country
No African ROC so far…
ROCs are part of EGEE SA1 (EGEE operations)
A ROC is in charge of coordinating operations in the region
EGEE contact with the site
In charge of help and validation of new site
Follow-up of site problems in the region
Site coordination in the region
Participate to global support
EGEE-funded people in each ROC
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

4. Grid Operation Tools and Procedures - M. Jouvin
Site Registration
To be published in EGEE production grid, a site must be registered in a central DB : GOCDB
When creating a new site, it must pass validation tests before being seen as a production site
After validation, site will appear in production BDII and be used automatically
A site has registered administrators
Site admins identified by their DN : requires a grid certificate
A site belongs to a Regional Operation Center (ROC)
Senegal : French ROC
South Africa : Italian ROC
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

5. Grid Operation Tools and Procedures - M. Jouvin
Site Monitoring…
SAM : Site Availability Monitoring
Set of tests run every hour at each site checking site health
Run under VO ops
Other VOs can run their own SAM tests under their VO credentials
Several group of tests : CE, SE, BDII… CE is the main one
Several tests per group
Result of tests is published on SAM portal : sam.cern.ch:8443/sam/sam.py
Results organized by ROCs
Site status is OK (green), WARNING (orange), ERROR (red)
Some tests considered critical : site no longer used by WMS until problem is fixed
A VO can publish its own view of site status
Gstat: monitors site perfomances, in particular BDII
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

6. Grid Operation Tools and Procedures - M. Jouvin
… Site Monitoring
CIC Portal
A grid operation portal
Ability to run SAM test on-demand (SAMAP)
Requires to be a site administrator
Requires the site to be in production
Gridmap :
Treemap view of the grid
Display site status
Give access to SAM page for site/resources
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

7. Grid Operation Tools and Procedures - M. Jouvin
Grid Support
GGUS : Global Grid User Support
Unique entry point for grid support, both for users and administrators
Used to report problem both about site behaviour or MW bugs
Ticket processed by dedicated persons (TPM) and assigned to ROC and sites
Problem fixing followed by ROC
Important to report problem there : allow for traceability and global follow-up
Important to answer and close problems
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

8. Grid Operation Tools and Procedures - M. Jouvin
Accounting
Accounting is a critical service in a grid
Users/VO manager need to know what is the real share of the resource they can use
Funding bodies want to know if their « users » have the expected share
Sites need to know who is using their site
Based on a publisher run at each site + 1 portal
Portal : w.php
Publisher runs on a grid machine called a MonBox : publishes into a grid central database
Portal has site-oriented and user-oriented views
Some (detailed) views required appropriate role, others are public
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

9. Grid Operation Tools and Procedures - M. Jouvin
Site Downtimes
Site downtimes must be declared in GOCDB
Users may know the site is unavailable
Prevent “central operators” to raise alarms
Avoid SAM test failures : tests are run but errors are “ignored”
Site status is MAINT(enance) : not used by WMS
Can be both “scheduled” or “unscheduled”
Severity can be “outage” (most often) or “at risk” to declare an important configuration change which should not affect availability
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

10. Grid Operation Tools and Procedures - M. Jouvin
Alarms
CIC Portal raises alarms when there is a SAM error for a resource without a declared downtime
Alarms can be subcribed by site administrators on CIC portal
Subscription per site/resource
Alarm sent as an
Alarm contains reference to the relevant page on SAM portal
If problem remains a few hour after the alarm is sent, ROC should open a GGUS ticket and assign it to the site
Not done automatically but by real persons in charge : 24x7 coverage, 2 persons in charge
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

11. Activity Visualization
LCG RTM: a “propaganda” tool at the beginning but also nice to use at CE/WMS activity
Displays grid activity on top of GoogleEarth
When clicking a site, access to jobs on the CE and WMS
3D application in Java : some problems on VISTA and MacOS
GridIce
An old application to look at queue statistics per site : running, waiting jobs, free CPUs…
Unfortunatly not well maintained and often broken
GridView : grid activity history
Jobs
Data transfers
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

12. Grid Operation Tools and Procedures - M. Jouvin
Mailing Lists
LCG ROLLOUT : list for discussion between site administrators
Don’t use it as an alternative to GGUS
To subscribe, bin/webadmin?REPORT=&t=1&s=0&X=&Y=&z=3&9=O&n=lcg- rollout&a=maybe
Pretty high volume but worth to have at least one person per site
Lot of grid experts on the list
EGEE BROADCAST : list used to broadcast important information related to EGEE operations
All site administrators declared in GOCB are automatically subscribed
Mainly used to broadcast downtimes and availability of new gLite updates
Medium volume (depends on number of downtimes!)
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019

13. Grid Operation Tools and Procedures - M. Jouvin
Useful Links…
EGEE Operational Procedures
edures
Grid Operation Tools and Procedures - M. Jouvin
27/04/2019