Internet2 E2E piPEs Joining the Federation of Network Measurement Infrastructures Eric L. Boyd 14 November 2018

Agenda Problem Statement A Call to Collaborative Action Performance Measurement Basic Architecture Available Tools and Latest Developments The Larger Goal International Collaboration Achievements to Date Get Involved! Contacts Existing Groups Documentation and Downloads 11/14/2018

Problem Statement: “The Network is Broken” How the can user self-diagnosis first mile problems without being a network expert? How can the user do partial path decomposition across multiple administrative domains? 11/14/2018

Agenda Problem Statement A Call to Collaborative Action Performance Measurement Basic Architecture Available Tools and Latest Developments The Larger Goal International Collaboration Achievements to Date Get Involved! Contacts Existing Groups Documentation and Downloads 11/14/2018

Self-Diagnosis Find a measurement server “near me”. Detect common tests in first mile. Don’t need to be a network engineer. Instead of: “The network is broken.” Hoped for result: “I don’t know what I’m talking about, but I think I have a duplex mismatch problem.” 11/14/2018

Partial Path Decomposition (1) Identify end-to-end path. Discover measurement nodes “near / representative of” hops along the route. Authenticate to multiple measurement domains (local-defined policies). Initiate tests between remote hosts. See test data for already run tests. 11/14/2018

Partial Path Decomposition (2) Instead of: “Can you give me an account on your machine?” “Can you set up and leave up and Iperf server?” “Can you get up at 2 AM to start up Iperf?” “Can you make up a policy on the fly for just me?” Hoped for result: Regular means of authentication Measurement peering agreements No chance of polluted test results Regular and consistent policy for access and limits 11/14/2018

The Importance of Measurement Bandwidth alone does not resolve all performance problems but technical tools and best practices for high performance are recognized and proven Through regular performance monitoring problems can be quickly identified and located thus facilitating resolution. Testing and measuring performance increases value of network to all participants. 11/14/2018

E2E piPEs Goals Recognizing these needs E2E piPEs strives to: Enable end-users & network operators to: determine E2E performance capabilities locate E2E problems contact the right person to get an E2E problem resolved. Enable remote initiation of partial path performance tests Make partial path performance data publicly available Be interoperable with other performance measurement frameworks 11/14/2018

Agenda Problem Statement A Call to Collaborative Action Performance Measurement Basic Architecture Available Tools and Latest Developments The Larger Goal International Collaboration Achievements to Date Get Involved! Contacts Existing Groups Documentation and Downloads 11/14/2018

Performance Measurement: Project Phases Phase 1: Tool Beacons BWCTL (Complete), http://e2epi.internet2.edu/bwctl OWAMP (Complete), http://e2epi.internet2.edu/owamp NDT (Complete), http://e2epi.internet2.edu/ndt Phase 2: Measurement Domain Support General Measurement Infrastructure (Prototype) Abilene Measurement Infrastructure Deployment (Complete), http://abilene.internet2.edu/observatory Phase 3: Federation Support AA (Prototype – optional AES key, policy file, limits file) Discovery (Measurement Nodes, Databases) (Prototype – nearest NDT server, web page) Test Request/Response Schema Support (Prototype – GGF NMWG Schema) 11/14/2018

BWCTL What is it? Typical Solution Typical road blocks A resource allocation and scheduling daemon for arbitration of iperf tests Typical Solution Run “iperf” or similar tool on two endpoints and hosts on intermediate paths Typical road blocks Need permissions on all systems involved Need to coordinate testing with others Need to run software on both sides with specified test parameters 11/14/2018

BWCTL: What’s new 3-Party communication is now supported. Authentication options allow a client to authenticate differently to each side of the test A local bwctld is no longer required bwctl will still detect a local bwctld Port ranges for bwctld peer connections can be specified Support for setting the TOS bits in the test packets. 11/14/2018

OWAMP What is it? Specification Measures one-way latency Control connection used to broker test request based upon policy restrictions and available resources. (Bandwidth/disk limits) Specification http://ietfreport.isoc.org/ids/draft-ietf-ippm-owdp-10.txt 11/14/2018

OWAMP: What’s new Hop count data available Port ranges for test connections can be specified Support for DSCP (RFC 2474) 11/14/2018

NDT “Single shot” diagnostic tool that doesn’t use historical data Measures performance to users desktop Combines numerous Web100 variables to analyze connection Develops network signatures for ‘typical’ network problems End-user based view of network Doesn’t require user to load any new software Can identify performance bottlenecks (could be host problem) Provides some ‘hard evidence’ to users and network administrators to reduce finger pointing 11/14/2018

NDT: What’s new Bug fixes Sourceforge-based Project New link detection algorithm 11/14/2018

Performance Measurement: Current piPEs Deployment 11/14/2018

Global PMP Directory In the absence of a look-up service … … how do you find other measurement beacons? http://e2epi.internet2.edu/pipes/pmp/pmp-dir.html 11/14/2018

Performance Measurement: Project Phases Phase 1: Tool Beacons BWCTL (Complete), http://e2epi.internet2.edu/bwctl OWAMP (Complete), http://e2epi.internet2.edu/owamp NDT (Complete), http://e2epi.internet2.edu/ndt Phase 2: Measurement Domain Support General Measurement Infrastructure (Prototype) Abilene Measurement Infrastructure Deployment (Complete), http://abilene.internet2.edu/observatory Phase 3: Federation Support AA (Prototype – optional AES key, policy file, limits file) Discovery (Measurement Nodes, Databases) (Prototype – nearest NDT server, web page) Test Request/Response Schema Support (Prototype – GGF NMWG Schema) 11/14/2018

piPEs Measurement Infrastructure Version 0.1 alpha prototype released Limited testing Limited configuration support http://e2epi.internet2.edu/pipes Serves as the Abilene Measurement Infrastructure Sample deployment in Europe Allows the configuration of a mesh of regularly scheduled tests 11/14/2018

piPEs Measurement Infrastructure Features Regularly scheduled and on-demand tests co-exist Regularly scheduled tests are implemented as regularly requested on-demand tests Distributed control enables inter-domain testing 11/14/2018

Functional Representation 11/14/2018

Tool Beacon Allows on-demand tests of a specific type to be negotiated. Configured with a set of policy definitions that tell the beacon how extensive a test may be for particular users Used to facilitate any on-demand test If these tests would be destructive to each other, then it must maintain scheduling information. 11/14/2018

Test Controller Runs tests from a single end-point to a tool beacon Reports the results back to a database. Configured with set of tests to run: Test parameters Test frequency Test partners (implemented as Test Beacons) Configured with a single Data Collector it should send the results to. 11/14/2018

Data Collector Listens for connections from test controllers. Receives new test data. Responsible data aggregation. Configured with the full set of Test Controllers that will contact it Configured with aggregation strategy for new data Data aggregation based upon how much data has been received, not current time. 11/14/2018

Data Viewer Collection of CGI scripts that run on the database host that analyzes and makes plots of the data. Web services front end Understands GGF NMWG request schema Returns results in GGF NMWG response schema 11/14/2018

Performance Measurement: Project Phases Phase 1: Tool Beacons BWCTL (Complete), http://e2epi.internet2.edu/bwctl OWAMP (Complete), http://e2epi.internet2.edu/owamp NDT (Complete), http://e2epi.internet2.edu/ndt Phase 2: Measurement Domain Support General Measurement Infrastructure (Prototype) Abilene Measurement Infrastructure Deployment (Complete), http://abilene.internet2.edu/observatory Phase 3: Federation Support AA (Prototype – optional AES key, policy file, limits file) Discovery (Measurement Nodes, Databases) (Prototype – nearest NDT server, web page) Test Request/Response Schema Support (Prototype – GGF NMWG Schema) 11/14/2018

Creating a Federation of Measurement Frameworks Why a Federation? Multiple measurement frameworks currently exist or are under development (piPES, NLANR Advisor, NLANR AMP, etc.) Open cooperation and the development of standards will promote the development of best practice measurement frameworks and interoperability. Future measurement frameworks can be built on the shoulders of current efforts. 11/14/2018

Federation Achievements Broad Achievements to Date We can speak the same (albeit evolving) measurement language. Numbers are growing though we are not yet at critical mass. Important Next Steps We can find each other We can verify each other’s identity 11/14/2018

End-to-End Path 11/14/2018

Solving the First Mile Suggestion #1: Individual campuses set up NDT servers “next to” gateway router. Suggestion #2: Campuses set up federation of NDT servers http://e2epi.internet2.edu/ndt 11/14/2018

Quality Control of the Network Backbone 11/14/2018

Quality Control of the Network Backbone Regularly Scheduled Tests: http://abilene.internet2.edu/observatory Web Service Access: http://abilene.internet2.edu/ami/webservices.html Analysis Tools (e.g MonALISA, NLANR / DAST Advisor 11/14/2018

Quality Control of Network Backbone Suggestion #3: Quality Control of other networks 11/14/2018

Quality Control of Peering Connections Suggestion #4: Measurement Peering Agreements 11/14/2018

Whole Kit-and-Caboodle Suggestion #5: Quality control of gigaPoPs Suggestion #6: Quality Control along Entire E2E Path 11/14/2018

Agenda Problem Statement A Call to Collaborative Action Performance Measurement Basic Architecture Available Tools and Latest Developments The Larger Goal International Collaboration Achievements to Date Get Involved! Contacts Existing Groups Documentation and Downloads 11/14/2018

American / European Collaboration Goals Awareness of ongoing Measurement Framework Efforts / Sharing of Ideas (Good / Not Sufficient) Interoperable Measurement Frameworks (Minimum) Common means of data extraction Partial path analysis possible along transatlantic paths Open Source Shared Development (Possibility, In Whole or In Part) End-to-end partial path analysis for transatlantic research communities VLBI: Haystack, Mass.  Onsala, Sweden HENP: Caltech, Calif.  CERN, Switzerland 11/14/2018

American / European Collaboration Achievements UCL E2E Monitoring Workshop 2003 http://people.internet2.edu/~eboyd/ucl_workshop.html Transatlantic Performance Monitoring Workshop 2004 http://people.internet2.edu/~eboyd/transatlantic_workshop.html Caltech <-> CERN Demo Haystack, USA <-> Onsala, Sweden piPEs Software Evaluation (In Progress) Architecture Reconciliation (In Progress) 11/14/2018

Example Application Community: VLBI (1) Very-Long-Baseline Interferometry (VLBI) is a high-resolution imaging technique used in radio astronomy. VLBI techniques involve using multiple radio telescopes simultaneously in an array to record data, which is then stored on magnetic tape and shipped to a central processing site for analysis. Goal: Using high-bandwidth networks, electronic transmission of VLBI data (known as “e-VLBI”). 11/14/2018

Example Application Community: VLBI (2) Haystack <-> Onsala Abilene, Eurolink, GEANT, NorduNet, SUNET User: David Lapsley, Alan Whitney Constraints Lack of administrative access (needed for Iperf) Heavily scheduled, limited windows for testing Problem Insufficient performance Partial Path Analysis with BWCTL/Iperf Isolated packet loss to local congestion in Haystack area Upgraded bottleneck link 11/14/2018

Example Application Community: VLBI (3) Result First demonstration of real-time, simultaneous correlation of data from two antennas (32 Mbps, work continues) Future Optimize time-of-day for non-real-time data transfers Deploy BWCTL at 3 more sites beyond Haystack, Onsala, and Kashima 11/14/2018

TSEV8 Experiment Intensive experiment Network: Data Antennas: 18 scans, 13.9 GB of data Antennas: Westford, MA and Kashima, Japan Network: Haystack, MA to Kashima, Japan Initially, 100 Mbps commodity Internet at each end, Kashima link upgraded to 1 Gbps just prior to experiment 11/14/2018

TSEV8 e-VLBI Network 11/14/2018

Network Issues In week leading up to experiment, network showed extremely poor throughput ~ 1 Mbps! Network analysis/troubleshooting required: Traditionally: pair-wise iperf testing between hosts along transfer path, step-by-step tracing of link utilization via Internet2/Transpac-APAN network monitoring websites: Time consuming, error prone, not conclusive New approach: automated iperf-testing using Internet2’s bwctl tool (allows partial path analysis), one single website to integrate link utilization statistics into one single website No maintenance required once setup, for the first time an overall view of the network and bandwidth on segment-by-segment basis 11/14/2018

E-VLBI Network Monitoring http://web.haystack.mit.edu/staff/dlapsley/tsev7.html 11/14/2018

E-VLBI Network Monitoring http://web.haystack.mit.edu/staff/dlapsley/tsev7.html 11/14/2018

E-VLBI Network Monitoring Use of centralized/integrated network monitoring helped to enable identification of bottleneck (hardware fault) Automated monitoring allows view of network throughput variation over time Highlights route changes, network outages Automated monitoring also helps to highlight any throughput issues at end points: E.g. Network Inteface Card failures, Untuned TCP Stacks Integrated monitoring provides overall view of network behavior at a glance 11/14/2018

Result Successful UT1 experiment completed June 30 2004. New record time for transfer and calculation of UT1 offset: 4.5 hours (down from 21 hours) 11/14/2018

Acknowledgements Yasuhiro Koyama, Masaki Hirabaru and colleagues at National Institute for Information and Communications Technology Brian Corey, Mike Poirier and colleagues from MIT Haysack Observatory Internet2, TransPAC/APAN, JGN2 networks Staff at APAN Tokyo XP Tom Lehman - University of Southern California - Information Sciences Institute East 11/14/2018

Agenda Problem Statement A Call to Collaborative Action Performance Measurement Basic Architecture Available Tools and Latest Developments The Larger Goal International Collaboration Achievements to Date Get Involved! Contacts Existing Groups Documentation and Downloads 11/14/2018

Getting Started Contact Internet2 for more information General Information: pipes-interest@internet2.edu Technical Contact: pipes-users@internet2.edu Receive Announcements BWCTL https://mail.internet2.edu/wws/info/bwctl-announce OWAMP https://mail.internet2.edu/wws/info/owamp-announce NDT https://mail.internet2.edu/wws/info/ndt-announce 11/14/2018

Participation (1) Set up BWCTL, OWAMP, NDT Beacons Place tool beacons “strategically” Consider locations Determine policy Establish limits “Register” beacons Let others know you are doing testing Use the Global PMP Directory http://e2epi.internet2.edu/pipes/pmp/pmp-dir.html 11/14/2018

Participation (2) Set up a measurement domain Install piPEs software Store performance data Make performance data available via web service Make visualization CGIs available Set up a local regularly scheduled mesh 11/14/2018

Participation (3) Establish Peering Agreements Support regular testing Examples: GiGaPoP, application community 11/14/2018

Resources Available Case Studies Discussion Forum Global PMP Directory http://e2epi.internet2.edu/case-studies/ Please alert us to new Case Studies Discussion Forum http://e2edev.internet2.edu/cgi-bin/forum/discus.cgi Global PMP Directory Make sure your beacons can be found by others http://e2epi.internet2.edu/pipes/pmp/pmp-dir.html 11/14/2018

The Community Application Communities Measurement Domains HENP, VLBI Measurement Domains ITECs, CERN, GEANT Collaboration Partners DANTE, ESnet, APAN Standards Integration GGS NMWG, IETF NLANR/DAST Advisor, MonALISA 11/14/2018

Your Participation Matters We can do partial path analysis, although making sense of the results is till a big issue. We can speak the same measurement language, although it’s still evolving. We are working together in growing numbers, but we need critical mass (become de facto standard). We need to be able to find each other. We need to be able to verify each other’s identity. 11/14/2018