EGI Network Support task force: Proposal for the identified use cases Mario Reale IGI / GARR mario.reale@garr.it January 24, 2011 EGI OMB f2f meeting Amsterdam EGI.eu 1

Overview Description of what we propose for each one of the identified use cases GGUS EGI PERT Network-related Scheduled Maintenances TroubleShooting on-demand End-to-end Multi Domain monitoring DownCollector Policy and Cooperation 2

1. GGUS based Network Support workflow

GGUS Reference Tools: GGUS / Network Support Unit Additional network monitoring & troubleshooting tools will be involved by the parties involved (NRENs, NOCs, PERTs..) Proposal: Implement a workflow to handle network tickets based on using GGUS Which does not foresee the establishment of a permanent EGI Network Support Team for this GGUS unit the great majority of NGIs being against it

Proposed GGUS Workflow 1/2 A user belonging to a given Virtual Organization (VO) experiences poor performances, or repeated failures while transferring data from site A to site B: A  B Of course first simple debuging is assumed to be carried out at the user level (possibly involving some VO support). Aim is to exclude “trivial” issues ( Software, SE down, ..) Basic troubleshooting can be provided by troubleshooting on-demand tool Also check if monitoring data are available A network ticket is then opened in GGUS describing the problem

Proposed GGUS workflow 2/2 The network ticket is assigned automatically to the Site administrators of site A and Site B They are both responsible for handling the ticket. However, only person should be accountable for the ticket: Site-A (A  B: originator of the data transfer) In case of User Interface node to Site-X transfer, ticket is assigned to Site-X Site Administrator, after first basic debugging by user Site administrators handle the contacting the NRENs contacts (APM, NOC) : They inform their NGI Operation Centers They should contact first local Campus Network Admins, local NREN APM NRENs will handle it, using their PERT team, APM, NOCs, experts, and possibly involving DANTE/GEANT NOC and Federated EduPERT NOCS further involve TELCO operators if/when required according to their workflows At each step the ticket originator, the VO responsible persons and the site admins are kept posted, informed, until the issue is solved

Network Support workflow Ticket posted (Already after initial debugging by user failed) Responible for the ticket processing NGI Operations Center informed / COD informed Site A & Site B Grid site Administrators try to fix it (for an A to B transfer) VO/VRC application expert & Campus Net Admin involved YES Solved ? NO NREN A & NREN B NOCs try to fix it NREN A & NREN B PERTs and local APMs involved YES Solved ? INFORM Site A&B NO GEANT NOC tries to fix it Federated PERT & GEANT APMs involved YES Solved ? NO Other Actors (TELCOs NOCs & Operations..) Problem fixed. Ticket closed

Observations Site A  Site B coordination assumed at least until clear domain of competence for the problem/bottleneck is identified A digging into the issue much more than B should be avoided Grid-to-Network domain crossing is responsibility of the Grid Site Administrators They contact Local Campus Admins, local NREN APMs and/or NOC Many parties involved (subsequently holding the token) but from the point of view of the workflow to be implemented, GGUS just assigns the network ticket to Site-A and Site-B site administrators (A->B) or Site-X admin ( UI->X) They will have to deal with following up the ticket Ultimately, site-A Grid Admin is responsible (start point for the data transfer) Whether NRENs prefer Grid Site Admins to contact their NOC or the Local APM first is something to be clarified ( NREN questionnaire)

Involved Actors A B LAN A Campus A NREN A BACKBONE LAN B Campus B NREN B RESPONSIBLE FOR TICKET user user NREN B APM NOC NREN A APM NOC Campus A Net Sup Grid Site A Admin Campus B Net Sup Grid Site B Admin GEANT/DANTE TEIN3 ORIENT SEEREN2 ALICE2/RedCLARA EUMEDCONNECT2 TELCOs ACCOUNTABLE FOR TICKET VO experts VO experts

Problem Solving: Actors Stack TELCOs operators Stack / Domain Federated PERT GEANT NOC (DANTE) GEANT APM NETWORK NREN PERT NREN-TELCOs operators NREN NOC NREN Local APM Campus Network Administrator GRID Grid Site Administrator NGI Operation Center COD Applications Hands over / Requests support VO/VRC applications support/experts Informs User Functional/Geographical distance from user

2. EGI PERT

EGI PERT At this stage we feel there wasn’t enough consensus by NGIs to establish a permanent EGI PERT team providing both Grid middleware/Applications expertise PERT Networking expertise PERTs will contribute to the general Network Support workflow We propose to leave involvement of PERT Teams to NRENs and GEANT NOC Our proposal is to provide a web contact point (web page) for EGI Users and Site Administrators, to fetch information from, about General PERT Issues and basic procedures and how to reach PERT Teams of NRENs and Federated PERT if required Gather relevant PERT contact information in one location Provide a basic web guide for common PERT-related issues (Example: how do I set the TCP window size on my machine ?, how do I check my machine is not closing a fundamental port for Grid middleware ?) Pointing to the EduPERT knowledge database General procedure for direct involvement of PERT Teams should however fall in the scope of the General GGUS-based workflow

3. Network-related Scheduled Maintenances

Network Related Scheduled Maintenances Reference tool: no specific tool for Network-related maintenances currently GOC DB for GRID-related maintenances. No tools warning users about Network-related scheduled maintenances Locally, Grid Site administrators, warned by the NREN APM of possible availability, can post the unavailability of their sites/services relies on ATM-Grid Site Admin local coordination No automation

Network Related Scheduled Maintenances What is envisageable around Network-related scheduled maintenances is NRENs coordinate with corresponding NGIs in order to have a Mapping between network devices/PoPs and directly impacted Grid resource center NGIs set up a tool implementing a mapping between Grid resource centers/services and involved user to be informed NGIs and NRENs to coordinate so that when a network device/PoP is object of a Scheduled Maintenance impacting on a Grid resource center/service, NREN informs NGI. NGI informs EGI.eu Operations and Users.

Network Related scheduled maintenances Today things are demanded to the good will of local APMs and Grid Site Managers, and their co-ordination A higher level workflow should be put in place, systematically addressing this issue A relevant deepening of this proposal is still required, in close coordination with the OTAG, JRA1 and EGI operations

Network Troubleshooting on Demand

Troubleshooting on demand Based on the experience gained in EGEE SA2, the French NGI has started and developed a new tool called HINTS HINTS has been developed on a volunteering basis by UREC CNRS Flexible, based on PerfSONAR web services and protocols The Task Force proposes this tool for Network Troubleshooting Presentation later on today

End-to-end multi domain monitoring

E2E multi domain monitoring The reference tool for e2e multi domain monitoring we propose is PerfSONAR Many NRENs are familiar with it Long term development by many key organizations and projects both in Europe and America GEANT project The Spanish NREN RedIRIS developed a customized version of PerfSONAR on a live CD for e2e measurements The PerfSONAR Team is presenting today its tools and Use Cases The NetJobs tools has been developed by CNRS and GARR to perform basic network monitoring measurements using Grid Jobs No need for local deployment Presentations today

Down Collector

DownCollector The DownCollector is currently in use EGI Inspire TSA1.4 Our proposal is to improve packaging, installation and configuration of the tool to ease the creation of new instances for the NGIs willing to deploy an instance – could be achieved with reasonable effort On the longer term an integrated system could be built, gathering information from the various distributed instances Building a mesh ( service at site X reached by site Y ) It has to be further discussed also with TSA1.4, JRA1 and OTAG However this cannot be endorsed immediately given manpower, responses to questionnaire and the pending discussions with relevant tasks/teams

Policy and Cooperation

Policy and Cooperation Majority of NGIs is against the establishment of permanent EGI Network Support body for policy and cooperation Some of them are very much in favor though We did not elaborate enough a structured proposal We have only identified fields for cooperation At this stage, our proposal is to invite volunteering NGIs to join the activity of the Network Support coordination (within EGI-Inspire TSA1.7) to further discuss this issue and elaborate a plan for ensuring issues are tackled within TSA1.7

General Issues The proposed GGUS workflow for Network support assumes some basic Network checks are done at the user level  We should ensure users are familiar with basic network debugging operations and tools we are providing Users should somehow try to refer to their site administrators first Site administrators have to deal with network-related issues; they’re likely to have at least basic network know-how  Should we foresee Training on the tools we want to provide ? A guide for users and site administrators around network support and related debugging procedures/tools ?

General Issues For some use cases we should find out more about the NREN-NGI interaction A Questionnaire for NRENs about Grids and NGIs has to be organized, aimed at clarifying Which are the current NREN-NGI communication channels (especially for NGI operations..) ? How feasible is to set up a global system to automatically advertise network-related scheduled maintenances and accidents to users and Site Admins ? What are the used/preferred Multi Domain tools NRENs are currently using and are familiar with ?

General Issues Network-related Scheduled Maintenances requires further analysis to refine our proposal and design/identify the corresponding tools We need to acquire more information from NRENs We also need more internal discussion Within the GGUS-based workflow, we should find out / decide whether NRENs prefer Grid Site admins to contact their local APMs first or their NOC Also to be asked to NRENs

References EGI Network Support coordination: https://wiki.egi.eu/wiki/NST PERT: http://edupert.geant.net/index.html Federated PERT Knowledge DataBase http://kb.pert.geant.net/PERTKB/WebHome