1. perfSONAR in ATLAS/WLCG Shawn McKee, Marian Babik ATLAS Jamboree / Network Section 3 rd December 2014

2. ● perfSONAR ● perfSONAR has been deployed to monitor the network perfSONAR ALL WLCG Tier-2 ● The WLCG Networks and Transfer Metrics working group is in the middle of a campaign to get perfSONAR upgraded and properly operating at ALL WLCG Tier-2 (and above) sites – Info on the working group is at: https://twiki.cern.ch/twiki/bin/view/LCG/NetworkTransferM etrics https://twiki.cern.ch/twiki/bin/view/LCG/NetworkTransferM etrics – perfSONAR install details at: https://twiki.opensciencegrid.org/bin/view/Documentation/ DeployperfSONAR https://twiki.opensciencegrid.org/bin/view/Documentation/ DeployperfSONAR – Primary challenges: newest version 3.4.1 installed, properly configured, firewalls not blocking operation Introduction Network Monitoring and Metrics WG Meeting

3. perfSONAR 3.4 released Oct 14 th Restructuring support and operations – Introduced site-level support via GGUS Rewritten documentatio n – https://twiki.opensciencegrid.org/bin/view/Documentation/DeployperfSONAR https://twiki.opensciencegrid.org/bin/view/Documentation/DeployperfSONAR Responded to ShellShock and Poodle – Sites advised to terminated their instances – Performed security audit and established security procedures Testing and validation of the new perfSONAR central configuration is in progress Ongoing perfSONAR 3.4 update campaign – includes migration to the new configuration system – security considerations documented – progressing well (119 sonars updated out of 214) – See http://grid-monitoring.cern.ch/perfsonar_coverage.txthttp://grid-monitoring.cern.ch/perfsonar_coverage.txt – deadline 8 th January (we start ticketing sites after…) 3 perfSONAR ops Network Monitoring and Metrics WG Meeting

4. Mesh-configuration tool deployed in OSG production – Extra slides at end cover this tool Provides central interface to reconfigure the entire network – All aspects – tests parameters, mesh participation – List of available sonars taken from GOCDB and OIM – Supports hierarchical support model (per mesh admins) – Web interface – Connected to perfSONAR infrastructure monitoring Site reconfiguration needed to adopt – Run as part of 3.4 campaign perfSONAR data store status and plans – Deployed in OSG ITB – several major issues fixed – Scale tests on-going this week; operationally ready for production – Will be the “source” of network metrics for OSG/WLCG SSBAGISSchedConfigDB – Plan is to feed ->SSB->AGIS->SchedConfigDB continuously Network Monitoring and Metrics WG Meeting 4 perfSONAR config and store

5. Via perfSONAR we gather a number of metrics: – Topology/path-information via traceroute – One-way delay via OWAMP – Packet-loss via OWAMP – Usable bandwidth via BWCTL ESnet has some nice pages on using perfSONAR to identify problems – http://fasterdata.es.net/performance- testing/evaluating-network-performance/ http://fasterdata.es.net/performance- testing/evaluating-network-performance/ Some specific examples are discussed in the presentation from last week’s WG meeting: https://indico.cern.ch/event/354593/ https://indico.cern.ch/event/354593/ Network Monitoring and Metrics WG Meeting 5 What?: Metrics and Their Use

6. Saul Youssef has made a study of FTS transfers (see http://egg.bu.edu/atlas/studies%7btype:egg.Hatch%7d/fts- wan-study-2-for-adc/plot_latest/ ) http://egg.bu.edu/atlas/studies%7btype:egg.Hatch%7d/fts- wan-study-2-for-adc/plot_latest/ – Uses FTS transfer data + traceroute; assumes avg rate/file – Identifies problematic FTS channels – Can identify problematic hops in the network – Update at http://egg.bu.edu/atlas/studies%7btype:egg.Hatch%7d/FTS_Nov ember_2014_bonus/ http://egg.bu.edu/atlas/studies%7btype:egg.Hatch%7d/FTS_Nov ember_2014_bonus/ This technique can be extended to other network metrics (perfSONAR, FAX, etc) Is very similar to PuNDITs network tomography algorithmPuNDITs We should plan to incorporate this technique in ops: – Automate this analysis on specific datasets (perfSONAR,FTS,FAX) – Plan to use it to identify problematic paths/links and FIX them! Network Monitoring and Metrics WG Meeting 6 Saul’s Network Study

7. We have a number of tools to track, monitor and manage the perfSONAR deployment – OMD (Nagios “bundle”) to track service status, versions, configuration https://maddash.aglt2.org/WLCGperfSONAR/check_mk (prototype) https://maddash.aglt2.org/WLCGperfSONAR/check_mk Credentials WLCGps/WLCG to “read” Now have a new version respecting x509 certs to be put into prod. – MaDDash to visualize metrics http://maddash.aglt2.org/maddash-webui/ (prototype) http://maddash.aglt2.org/maddash-webui/ – Summary coverage http://grid- monitoring.cern.ch/perfsonar_coverage.txthttp://grid- monitoring.cern.ch/perfsonar_coverage.txt – Mesh config/management (see slides at end) Network Monitoring and Metrics WG Meeting 7 Monitoring/Management Tools

8. perfSONAR instances must be upgraded and properly configured. WG waiting on input from ATLAS (and others) on use-cases/requirements for network metrics – Strawman document ready early next year Discussion topics – How best to correlate perfSONAR instances with storage? – Tuning perfSONAR parameters and coverage – Requirements for “user” API for datastore – Using perfSONAR data (network tomography; problem location; problem identification) Network Monitoring and Metrics WG Meeting 8 perfSONAR Related Items

9. We need to get perfSONAR data consistently available from all our sites, covering all our paths. Get sites upgraded/configured! Questions? Discussion, Comments? Network Monitoring and Metrics WG Meeting 9 Conclusion

10. Network Monitoring and Metrics WG Meeting 10 Mesh-Config GUI Host Groups OSG (Soichi) has developed a nice web interface for mesh creation and configuration Currently implements access based upon x509 credential. No fined-grain authorization: either ‘admin’ or ‘no access’ Instances found from perfSONAR registration information from OIM (OSG) or GOCDB (WLCG)

11. Network Monitoring and Metrics WG Meeting 11 Mesh-Config Parameters Parameters for perfSONAR tests are controlled centrally. Easy to modify as required

12. Network Monitoring and Metrics WG Meeting 12 Mesh-Config Meshes Meshes can be created using this tab. This is the “metadata” needed to organize sets of perfSONAR instances.

13. Network Monitoring and Metrics WG Meeting 13 Mesh-Config Test Definitions What tests get run for a mesh? That is controlled by this section.

14. Once meshes are defined they are exposed via a URL like: http://myosg.grid.iu.edu/pfmesh/json/name/<mesh- name>?new Example for us-atlas: http://myosg.grid.iu.edu/pfmesh/json/name/u s-atlas?new http://myosg.grid.iu.edu/pfmesh/json/name/u s-atlas?new Status: In production but without the ?new will return the old “static” values hosted on CERN AFS Network Monitoring and Metrics WG Meeting 14 Mesh Config URL