1. INTERNATIONAL NETWORKS At Indiana University Jennifer Schopf International Networks, Indiana University Sean Peisert, Brian Tierney Esnet and UC Davis Jason Leigh Laboratory for Advanced Visualization & Applications, University of Hawaii Mānoa Supported by the National Science Foundation Open privacy-aware network measurement, analysis, and visualization service

2. INTERNATIONAL NETWORKS At Indiana University Acknowledgements NetSage is funded by US NSF award #1540933 Joint project with Indiana University University of California at Davis ESnet University of Hawai’i Manoa

3. INTERNATIONAL NETWORKS At Indiana University International Research Network Connections (IRNC) Monitoring: NetSage Open, privacy-aware, network measurement, analysis, and visualization service Designed to address the needs of today’s international networks Focus on useful work with production networks, not on novel research Tasked by NSF to coordinate monitoring work across the IRNC projects 3

4. INTERNATIONAL NETWORKS At Indiana University

5. INTERNATIONAL NETWORKS At Indiana University IRNC Backbones and Exchange Points Backbones: TransPAC4 (Schopf, IU) AmLIght ExP (Ibarra, FIU) Pacific Islands Research and Education Networks (PIREN) (Lassner, U Hawaii) America Connects to Europe (ACE) (Schopf, IU) Exchange Points AtlanticWave (Ibarra, FIU) StarLight (Mambretti, NU) Pacific Wave(Fox, CENIC)

6. INTERNATIONAL NETWORKS At Indiana University 3 NetSage Use Cases 1.Current traffic patterns across IRNC links, and the ability to anticipate growth trends for capacity-planning purposes; 2.The main sources and sinks of large, elephant flows to know where to focus outreach and training opportunities; and 3.Where packet loss is occurring, whether the cause is congestion or other issues, and what impact it has on end-to-end performance. 6

7. INTERNATIONAL NETWORKS At Indiana University NetSage End users Project oversight (NSF, etc) Congressional mandate that NSF projects show societal relevance IRNC NOC and other operators Project planning by backbone and exchange point operators Application engagement staff

8. INTERNATIONAL NETWORKS At Indiana University NetSage Passive Measurements ToolMetricQuestion Answered SNMPUtilizationHow busy is a given hop? Flow data (e.g. NetFlow) src/dst IP/port and transfer rate Who are the Top Talkers? What applications are used to transfer data? tstat Via both port forwarding and on select DTNs TCP / UDP statistics, in addition to Flow data Are the hosts properly tuned? Is the network dropping or reordering packets? Note: All IPs are de-identified by stripping low-order bits

9. INTERNATIONAL NETWORKS At Indiana University NetSage Active Measurements ToolMetricQuestion Answered perfSONAR/iperf3Available bandwidth What is the achievable end-to- end throughput? perfSONAR/owam p Loss and latency What is the end-to- end loss and latency? perfSONAR/trace path Topology infoIs the path asymmetric? MTU?

10. INTERNATIONAL NETWORKS At Indiana University Year 1: Archiving Time Series data System (TSDS) Common archive shared with IRNC NOC Open Source software on commodity hardware Provides well structured and high performance storage and retrieval of timeseries data Capable of tracking and reporting based on metadata eg. viewing interface throughput from the viewpoint of a VLAN or BGP peer sessions from a particular ASN https://github.com/GlobalNOC/tsds-services

11. INTERNATIONAL NETWORKS At Indiana University Year 1-2: Active Measurements on Backbones PerfSONAR Open source, community based tool for latency and throughput data Hook into existing testpoints deployed on IRNC framework Pull data into common archive Enable common queries across IRNC sites Stage 2 Extend tests with added perfSONAR nodes

12. INTERNATIONAL NETWORKS At Indiana University perfSONAR Dashboard

13. INTERNATIONAL NETWORKS At Indiana University Year 1 Passive Measurements on Backbones: SNMP All of the IRNC backbones and exchange points already collect this Requirement of sharing with the NOC was already in place Access via Indiana Univ SNAPP tool Common queries across all IRNC sites

14. INTERNATIONAL NETWORKS At Indiana University Year 1 Passive Measurements on Backbones: Limited Flow Data TransPAC is collecting de-identified flow data Adapting archive and queries to include this format Extending TransPAC’s analysis scripts for broader use Working with privacy committee to come to agreements to collect this data on other backbones in Year 2

15. INTERNATIONAL NETWORKS At Indiana University Year 1: Evaluation of Packet Header Inspection Tools - tstat Selected ‘tstat’ tool, part of the EU ‘mplane’ (Measurement Plane) project http://tstat.polito.it/ Based on libpcap capture of headers requires 30-50% of 1 core (3.5Ghz Xeon) to do full line rate 10Gbps Includes ability to de-identify IP using Crypto-Pan prefix-preserving IP anonymizer Using tstat for TCP retransmission analysis and basic flow data

16. INTERNATIONAL NETWORKS At Indiana University tstat data collection tstat generates 1 line per TCP socket SRC/DST IP and Port TCP stats: (each direction) tcp_rexmit_bytes, tcp_rexmit_pkts, tcp_rtt_avg, tcp_rtt_min, tcp_rtt_max, tcp_rtt_std, tcp_pkts_rto, tcp_pkts_fs, tcp_pkts_reor, tcp_pkts_dup, tcp_pkts_unk, tcp_pkts_fc, tcp_pkts_unrto, tcp_pkts_unfs, tcp_cwin_min, tcp_cwin_max, tcp_out_seq_pkts, tcp_window_scale, tcp_mss, tcp_max_seg_size, tcp_min_seg_size, tcp_win_max, tcp_win_min, tcp_initial_cwin Think of this as Netflow ++

17. INTERNATIONAL NETWORKS At Indiana University Year 2: Work with tstat Test deployment on TransPAC LA De-identification pipeline for data Log cleanup on host site Additional deployments required data policy documents Data Privacy Practices Memorandum of Cooperation with data sites Next deployments TransPAC Seattle ACE WIX AmLight Miami

18. INTERNATIONAL NETWORKS At Indiana University Year 2: Active and Passive Measurements on Exchange Points How do we adequately represent the IRNC investment? What does it mean to monitor and exchange point in this context? SNMP data perfSONAR data Still being actively discussed

19. INTERNATIONAL NETWORKS At Indiana University Year 1: Visualization Analyzed prior IRNC & related visualization approaches (Gloriad/InSight, my.es.net, IU SNAPP) Articulate specific network questions from which data gathering efforts and visualizations can then be prioritized Developed concept sketches for critique Met with Jon Dugan (my.es.net) to discuss leveraging existing framework where applicable.

20. INTERNATIONAL NETWORKS At Indiana University Visualization – Year 1 & 2 What is the max, min, average bandwidth used between links? (Year 1, 2) Which exchange points or networks are congested? (Year 2) When and how often do they remain congested? (Year 2)

21. INTERNATIONAL NETWORKS At Indiana University Visualization - Year 2+ What is the duration and are there any periodic patterns or peak periods? (Year 2+) What are the top sites that use the IRNC links (average and peak)? (Year 2+) What are the top science projects that use the IRNC links? (Year 2+) What is the nature of elephant flows that use the links? (Year 2+) How many elephant flows tend to occur at the same time? How often, how long do this overlaps occur? (Year 2+) Which exchange points or networks service the most elephant flows? (Year 2+) what is the max, min, average duration of elephant flows? (Year 2+) Are the flows bursty or experiencing sustained traffic? (Year 2+) When do elephant flows tend to occur and how regularly? (Year 2+) What protocols are used for elephant flows?(TCP, UDP, RDMA)? (Year 2+) How can we best identify a list of top talkers for each link? (Year 2+)

22. INTERNATIONAL NETWORKS At Indiana University Visualization - Year 3+ Is the loss due to the client, the link or the exchange point? (Year 3) Do losses tend to appear when performing active tests or when applications use the links? (Year 3) How much does packet loss diminish the overall throughput of a flow? (how much of the researchers time is lost due to packet loss) Do losses tend to be due to flows exceeding capacity or network infrastructure problems? How long do packet loss problems persist before detection, source identification and resolution?

23. INTERNATIONAL NETWORKS At Indiana University Now Yesterday Last week Last month Last Year This year Specific (year, day…) Tagged Event: SC Conference Duration: time to time TIME Location Worldwide Country States(Regions) Building Specific Link Institutions Application Database/Mail… UDP/TCP… Audio/Video/Text… Disciplines Physics/ Geoscience … NSF Project QUALIFIERS What is the max, min, average bandwidth What is the duration and are there any periodic patterns or peak periods Which exchange points or networks are congested When and how often do the links / exchange points remain congested What are the top sites that use the IRNC links (average and peak) What are the top science projects that use the IRNC links (average and peak) OPERATIONS

24. INTERNATIONAL NETWORKS At Indiana University Questions from a pull-down menu serve as Query Templates to get you started Query can then be customized/refined

25. INTERNATIONAL NETWORKS At Indiana University Query produces topology map color coded by link and exchange point average throughput. Darker is more.

26. INTERNATIONAL NETWORKS At Indiana University Presentation of link data throughput on axes aligned histograms that enable visual comparison of individual vs all the other links. Comparison of total data transmitted per link in relation to aggregated whole of all the links.

27. INTERNATIONAL NETWORKS At Indiana University Similarly charting for Exchange Points

28. INTERNATIONAL NETWORKS At Indiana University Subsequent queries are appended for custom report generation. All queries can be forwarded (e.g. to NOC) to assist in troubleshooting.

29. INTERNATIONAL NETWORKS At Indiana University More Detailed Charting Max/Min/Avg bandwidth & total data transmitted of links... … broken into institutions … broken int domains Sortable columns based on criteria- see next slide [time frame (Year 2-3)]

30. INTERNATIONAL NETWORKS At Indiana University “What is the duration and are there any periodic patterns or peak periods between IRNC links?” For Periodic Patterns

31. INTERNATIONAL NETWORKS At Indiana University Sample Flow Visualization (from ESnet)

32. INTERNATIONAL NETWORKS At Indiana University Questions/Comments? NetSage Website: http://www.netsage.science Questions? Contact Jennifer Schopf – jmschopf@indiana.edujmschopf@indiana.edu 32