1. Enabling Grids for E-sciencE COD 19 meeting, Bologna Nordic ROD experiences Michaela Lechner COD-19, Bologna

2. Enabling Grids for E-sciencE COD 19 meeting, Bologna 2/10 Overview NE ROC split into two regions (Federations)‏ – BeNeLux (Belgium and the Netherlands) and Nordics (Denmark, Finland, Norway, Sweden, and the Baltic states[MoU])‏. Nordic duty shifts – 6 teams, 3 from SNIC and 3 from NDGF. – http://www.egee-ne.org/internal/opdocs/NORDICroster2009 http://www.egee-ne.org/internal/opdocs/NORDICroster2009 –8 hours a day, 5 days a week coverage. Normal working time. –one team (usually one person) on duty for a week. –NDGF-T1 and EGEE operations overlap and are combined: ROD and Operator on Duty (OoD) at the same time!

3. Enabling Grids for E-sciencE COD 19 meeting, Bologna 3/10 ROD on-duty tasks On-duty tasks (for BeNeLux and Nordic ROD)‏ –Perform all 1 st line support and ROD tasks as defined in the 1st line support and ROD operations manual: https://twiki.cern.ch/twiki/bin/view/EGEE/OperationalProceduresforRegionalCODAndInteractionsWit hCCOD https://twiki.cern.ch/twiki/bin/view/EGEE/OperationalProceduresforRegionalCODAndInteractionsWit hCCOD –Watch other monitoring tools for T1 (Nagios, ganglia)‏. (=OoD)‏ –Use ROD dashboard tools –Assist sites where appropriate. Weekly duties (for Nordic OoD)‏ – Handover –Fill out NDGF-T1 production report –Be present at WLCG weekly operation meeting next Mon 16:00 –Prepare notes to weekly meeting on Friday 10:00

4. Enabling Grids for E-sciencE COD 19 meeting, Bologna 4/10 Nordic ROD communication Communication – All on-duty support via chat(s) and mailing list(s). – Mailing list for all NE ROC operators. The ROC mailing list: roc@hpc2n.umu.se is archived under https://www.hpc2n.umu.se/mailman/private/roc/ roc@hpc2n.umu.se https://www.hpc2n.umu.se/mailman/private/roc/ – 1 st line support over support-nordic@egee-ne.orgsupport-nordic@egee-ne.org –Shift handover via support-nordic@egee-ne.org mailing list.support-nordic@egee-ne.org –XMPP (Jabber) chat: ndgf@groupchat.nordu.netndgf@groupchat.nordu.net –Contacts for sites and site admins are available on: https://portal.nordu.net/display/ndgfwiki/Operation-Procedures https://portal.nordu.net/display/ndgfwiki/Operation-Procedures –Communication through C-COD (Vera, Luuk) if necessary. –#cic-on-duty IRC chat @testbed011.cnaf.infn.it –Regular weekly status meetings, one each for EGEE (Thursday) and NDGF (Friday) operations.

5. Enabling Grids for E-sciencE COD 19 meeting, Bologna 5/10 Nordic ROD knowledge sharing Short term –Mailing lists –Handover procedure –NDGF Chat –“None of the duties should be done acting alone” Longer term –Dashboard tools and notes therein. Notes: time and determination –GGUS: solutions are public and available. –NDGF wiki for OoD work –1 st line support over support-nordic@egee-ne.org (archived)‏support-nordic@egee-ne.org

6. Enabling Grids for E-sciencE COD 19 meeting, Bologna 6/10 Daily Duties: Preliminaries Global Grid User Support (GGUS) Overview: https://gus.fzk.de/ws/ticket_search.php?&supportunit=ROC_North&status=open&timeframe=no CIC Portal: https://cic.gridops.org/index.php?section=roc&page=dashboard 1. You have to have a regional role in the GOCDB 2. Don't forget to set your preferences to one of these regional roles, not site Administrator. Log into the COD IRC chat Change your nick in the NDGF chat eg. /nick caela(ROD, OoD)‏

7. Enabling Grids for E-sciencE COD 19 meeting, Bologna 7/10 Duties in detail: The Dashboard Home tab – overview fresh alarms, open tickets Ticket tab – React on colored tickets Dashboard tab – Actual work is done here – Set off alarms, when problems solved – read/write notes – Creating tickets or extract mail adresses used for 1 st line support – Write informal mails Experience: – Nice with glide over and links, but slow in many ways – Not all tickets shown in ticket tab? – Creating tickets easier, administration fine

8. Enabling Grids for E-sciencE COD 19 meeting, Bologna 8/10 Bugs

9. Enabling Grids for E-sciencE COD 19 meeting, Bologna 9/10 When it burns (ASGC-DC)‏

10. Enabling Grids for E-sciencE COD 19 meeting, Bologna 10/10 Going for a cup of coffee

11. Enabling Grids for E-sciencE COD 19 meeting, Bologna 11/10 Nordic helpdesk solutions Regional helpdesk – BeNeLux have own ticketing system, but nothing similar implemented for our Nordic sites, besides support-nordic@egee- ne.org Avoiding to talk with yourself – Being 1 st line support, ROD and helpdesk all at the same time? – Too much responsibility on one person. – Furthermore one persons knowledge is not infinite: waiting one week for new ideas? – Solution: Drawing more knowledge in (NDGF chat, mailinglists)‏

12. Enabling Grids for E-sciencE COD 19 meeting, Bologna 12/10 Nordic ROD workload Takes 2-3 days to get friendly with system – Hard to get an overview of new tools, new sites (Baltics)‏ – Doing many things simultaneously – Too little time dedicated to real problem solving – Dashboard hideously slow (a feature, not a bug!)‏ Should definitely not be done at the same time as TPM!

13. Enabling Grids for E-sciencE COD 19 meeting, Bologna 13/10 Conclusion Found our own good practice procedures :-)‏ Dashboard – On the way to become a great administrative tool, but beware not to forget real support – Preferred sites option is of great help