1. Internet2 E2E piPEs Joining the Federation of Network Measurement Infrastructures
Eric L. Boyd
14 November 2018

2. Agenda Problem Statement A Call to Collaborative Action
Performance Measurement
Basic Architecture
Available Tools and Latest Developments
The Larger Goal
International Collaboration
Achievements to Date
Get Involved!
Contacts
Existing Groups
Documentation and Downloads
11/14/2018

3. Problem Statement: “The Network is Broken”
How the can user self-diagnosis first mile problems without being a network expert?
How can the user do partial path decomposition across multiple administrative domains?
11/14/2018

4. Agenda Problem Statement A Call to Collaborative Action
Performance Measurement
Basic Architecture
Available Tools and Latest Developments
The Larger Goal
International Collaboration
Achievements to Date
Get Involved!
Contacts
Existing Groups
Documentation and Downloads
11/14/2018

5. Self-Diagnosis Find a measurement server “near me”.
Detect common tests in first mile.
Don’t need to be a network engineer.
Instead of:
“The network is broken.”
Hoped for result:
“I don’t know what I’m talking about, but I think I have a duplex mismatch problem.”
11/14/2018

6. Partial Path Decomposition (1)
Identify end-to-end path.
Discover measurement nodes “near / representative of” hops along the route.
Authenticate to multiple measurement domains (local-defined policies).
Initiate tests between remote hosts.
See test data for already run tests.
11/14/2018

7. Partial Path Decomposition (2)
Instead of:
“Can you give me an account on your machine?”
“Can you set up and leave up and Iperf server?”
“Can you get up at 2 AM to start up Iperf?”
“Can you make up a policy on the fly for just me?”
Hoped for result:
Regular means of authentication
Measurement peering agreements
No chance of polluted test results
Regular and consistent policy for access and limits
11/14/2018

8. The Importance of Measurement
Bandwidth alone does not resolve all performance problems but technical tools and best practices for high performance are recognized and proven
Through regular performance monitoring problems can be quickly identified and located thus facilitating resolution.
Testing and measuring performance increases value of network to all participants.
11/14/2018

9. E2E piPEs Goals Recognizing these needs E2E piPEs strives to:
Enable end-users & network operators to:
determine E2E performance capabilities
locate E2E problems
contact the right person to get an E2E problem resolved.
Enable remote initiation of partial path performance tests
Make partial path performance data publicly available
Be interoperable with other performance measurement frameworks
11/14/2018

10. Agenda Problem Statement A Call to Collaborative Action
Performance Measurement
Basic Architecture
Available Tools and Latest Developments
The Larger Goal
International Collaboration
Achievements to Date
Get Involved!
Contacts
Existing Groups
Documentation and Downloads
11/14/2018

11. Performance Measurement: Project Phases
Phase 1: Tool Beacons
BWCTL (Complete),
OWAMP (Complete),
NDT (Complete),
Phase 2: Measurement Domain Support
General Measurement Infrastructure (Prototype)
Abilene Measurement Infrastructure Deployment (Complete),
Phase 3: Federation Support
AA (Prototype – optional AES key, policy file, limits file)
Discovery (Measurement Nodes, Databases) (Prototype – nearest NDT server, web page)
Test Request/Response Schema Support (Prototype – GGF NMWG Schema)
11/14/2018

12. BWCTL What is it? Typical Solution Typical road blocks
A resource allocation and scheduling daemon for arbitration of iperf tests
Typical Solution
Run “iperf” or similar tool on two endpoints and hosts on intermediate paths
Typical road blocks
Need permissions on all systems involved
Need to coordinate testing with others
Need to run software on both sides with specified test parameters
11/14/2018

13. BWCTL: What’s new 3-Party communication is now supported.
Authentication options allow a client to authenticate differently to each side of the test
A local bwctld is no longer required
bwctl will still detect a local bwctld
Port ranges for bwctld peer connections can be specified
Support for setting the TOS bits in the test packets.
11/14/2018

14. OWAMP What is it? Specification Measures one-way latency
Control connection used to broker test request based upon policy restrictions and available resources. (Bandwidth/disk limits)
Specification
11/14/2018

15. OWAMP: What’s new Hop count data available
Port ranges for test connections can be specified
Support for DSCP (RFC 2474)
11/14/2018

16. NDT “Single shot” diagnostic tool that doesn’t use historical data
Measures performance to users desktop
Combines numerous Web100 variables to analyze connection
Develops network signatures for ‘typical’ network problems
End-user based view of network
Doesn’t require user to load any new software
Can identify performance bottlenecks (could be host problem)
Provides some ‘hard evidence’ to users and network administrators to reduce finger pointing
11/14/2018

17. NDT: What’s new Bug fixes Sourceforge-based Project
New link detection algorithm
11/14/2018

18. Performance Measurement: Current piPEs Deployment
11/14/2018

19. Global PMP Directory In the absence of a look-up service …
… how do you find other measurement beacons?
11/14/2018

20. Performance Measurement: Project Phases
Phase 1: Tool Beacons
BWCTL (Complete),
OWAMP (Complete),
NDT (Complete),
Phase 2: Measurement Domain Support
General Measurement Infrastructure (Prototype)
Abilene Measurement Infrastructure Deployment (Complete),
Phase 3: Federation Support
AA (Prototype – optional AES key, policy file, limits file)
Discovery (Measurement Nodes, Databases) (Prototype – nearest NDT server, web page)
Test Request/Response Schema Support (Prototype – GGF NMWG Schema)
11/14/2018

21. piPEs Measurement Infrastructure
Version 0.1 alpha prototype released
Limited testing
Limited configuration support
Serves as the Abilene Measurement Infrastructure
Sample deployment in Europe
Allows the configuration of a mesh of regularly scheduled tests
11/14/2018

22. piPEs Measurement Infrastructure Features
Regularly scheduled and on-demand tests co-exist
Regularly scheduled tests are implemented as regularly requested on-demand tests
Distributed control enables inter-domain testing
11/14/2018

23. Functional Representation
11/14/2018

24. Tool Beacon
Allows on-demand tests of a specific type to be negotiated.
Configured with a set of policy definitions that tell the beacon how extensive a test may be for particular users
Used to facilitate any on-demand test
If these tests would be destructive to each other, then it must maintain scheduling information.
11/14/2018

25. Test Controller Runs tests from a single end-point to a tool beacon
Reports the results back to a database.
Configured with set of tests to run:
Test parameters
Test frequency
Test partners (implemented as Test Beacons)
Configured with a single Data Collector it should send the results to.
11/14/2018

26. Data Collector Listens for connections from test controllers.
Receives new test data.
Responsible data aggregation.
Configured with the full set of Test Controllers that will contact it
Configured with aggregation strategy for new data
Data aggregation based upon how much data has been received, not current time.
11/14/2018

27. Data Viewer
Collection of CGI scripts that run on the database host that analyzes and makes plots of the data.
Web services front end
Understands GGF NMWG request schema
Returns results in GGF NMWG response schema
11/14/2018

28. Performance Measurement: Project Phases
Phase 1: Tool Beacons
BWCTL (Complete),
OWAMP (Complete),
NDT (Complete),
Phase 2: Measurement Domain Support
General Measurement Infrastructure (Prototype)
Abilene Measurement Infrastructure Deployment (Complete),
Phase 3: Federation Support
AA (Prototype – optional AES key, policy file, limits file)
Discovery (Measurement Nodes, Databases) (Prototype – nearest NDT server, web page)
Test Request/Response Schema Support (Prototype – GGF NMWG Schema)
11/14/2018

29. Creating a Federation of Measurement Frameworks
Why a Federation?
Multiple measurement frameworks currently exist or are under development (piPES, NLANR Advisor, NLANR AMP, etc.)
Open cooperation and the development of standards will promote the development of best practice measurement frameworks and interoperability.
Future measurement frameworks can be built on the shoulders of current efforts.
11/14/2018

30. Federation Achievements
Broad Achievements to Date
We can speak the same (albeit evolving) measurement language.
Numbers are growing though we are not yet at critical mass.
Important Next Steps
We can find each other
We can verify each other’s identity
11/14/2018

31. End-to-End Path
11/14/2018

32. Solving the First Mile
Suggestion #1: Individual campuses set up NDT servers “next to”
gateway router.
Suggestion #2: Campuses set up federation of NDT servers
11/14/2018

33. Quality Control of the Network Backbone
11/14/2018

34. Quality Control of the Network Backbone
Regularly Scheduled Tests:
Web Service Access:
Analysis Tools (e.g MonALISA, NLANR / DAST Advisor
11/14/2018

35. Quality Control of Network Backbone
Suggestion #3: Quality Control of other networks
11/14/2018

36. Quality Control of Peering Connections
Suggestion #4: Measurement Peering Agreements
11/14/2018

37. Whole Kit-and-Caboodle
Suggestion #5: Quality control of gigaPoPs
Suggestion #6: Quality Control along Entire E2E Path
11/14/2018

38. Agenda Problem Statement A Call to Collaborative Action
Performance Measurement
Basic Architecture
Available Tools and Latest Developments
The Larger Goal
International Collaboration
Achievements to Date
Get Involved!
Contacts
Existing Groups
Documentation and Downloads
11/14/2018

39. American / European Collaboration Goals
Awareness of ongoing Measurement Framework Efforts / Sharing of Ideas (Good / Not Sufficient)
Interoperable Measurement Frameworks (Minimum)
Common means of data extraction
Partial path analysis possible along transatlantic paths
Open Source Shared Development (Possibility, In Whole or In Part)
End-to-end partial path analysis for transatlantic research communities
VLBI: Haystack, Mass.  Onsala, Sweden
HENP: Caltech, Calif.  CERN, Switzerland
11/14/2018

40. American / European Collaboration Achievements
UCL E2E Monitoring Workshop 2003
Transatlantic Performance Monitoring Workshop 2004
Caltech <-> CERN Demo
Haystack, USA <-> Onsala, Sweden
piPEs Software Evaluation (In Progress)
Architecture Reconciliation (In Progress)
11/14/2018

41. Example Application Community: VLBI (1)
Very-Long-Baseline Interferometry (VLBI) is a high-resolution imaging technique used in radio astronomy.
VLBI techniques involve using multiple radio telescopes simultaneously in an array to record data, which is then stored on magnetic tape and shipped to a central processing site for analysis.
Goal: Using high-bandwidth networks, electronic transmission of VLBI data (known as “e-VLBI”).
11/14/2018

42. Example Application Community: VLBI (2)
Haystack <-> Onsala
Abilene, Eurolink, GEANT, NorduNet, SUNET
User: David Lapsley, Alan Whitney
Constraints
Lack of administrative access (needed for Iperf)
Heavily scheduled, limited windows for testing
Problem
Insufficient performance
Partial Path Analysis with BWCTL/Iperf
Isolated packet loss to local congestion in Haystack area
Upgraded bottleneck link
11/14/2018

43. Example Application Community: VLBI (3)
Result
First demonstration of real-time, simultaneous correlation of data from two antennas (32 Mbps, work continues)
Future
Optimize time-of-day for non-real-time data transfers
Deploy BWCTL at 3 more sites beyond Haystack, Onsala, and Kashima
11/14/2018

44. TSEV8 Experiment Intensive experiment Network: Data Antennas:
18 scans, 13.9 GB of data
Antennas:
Westford, MA and Kashima, Japan
Network:
Haystack, MA to Kashima, Japan
Initially, 100 Mbps commodity Internet at each end, Kashima link upgraded to 1 Gbps just prior to experiment
11/14/2018

45. TSEV8 e-VLBI Network
11/14/2018

46. Network Issues
In week leading up to experiment, network showed extremely poor throughput ~ 1 Mbps!
Network analysis/troubleshooting required:
Traditionally: pair-wise iperf testing between hosts along transfer path, step-by-step tracing of link utilization via Internet2/Transpac-APAN network monitoring websites:
Time consuming, error prone, not conclusive
New approach: automated iperf-testing using Internet2’s bwctl tool (allows partial path analysis), one single website to integrate link utilization statistics into one single website
No maintenance required once setup, for the first time an overall view of the network and bandwidth on segment-by-segment basis
11/14/2018

47. E-VLBI Network Monitoring
11/14/2018

48. E-VLBI Network Monitoring
11/14/2018

49. E-VLBI Network Monitoring
Use of centralized/integrated network monitoring helped to enable identification of bottleneck (hardware fault)
Automated monitoring allows view of network throughput variation over time
Highlights route changes, network outages
Automated monitoring also helps to highlight any throughput issues at end points:
E.g. Network Inteface Card failures, Untuned TCP Stacks
Integrated monitoring provides overall view of network behavior at a glance
11/14/2018

50. Result Successful UT1 experiment completed June 30 2004.
New record time for transfer and calculation of UT1 offset:
4.5 hours (down from 21 hours)
11/14/2018

51. Acknowledgements
Yasuhiro Koyama, Masaki Hirabaru and colleagues at National Institute for Information and Communications Technology
Brian Corey, Mike Poirier and colleagues from MIT Haysack Observatory
Internet2, TransPAC/APAN, JGN2 networks
Staff at APAN Tokyo XP
Tom Lehman - University of Southern California - Information Sciences Institute East
11/14/2018

52. Agenda Problem Statement A Call to Collaborative Action
Performance Measurement
Basic Architecture
Available Tools and Latest Developments
The Larger Goal
International Collaboration
Achievements to Date
Get Involved!
Contacts
Existing Groups
Documentation and Downloads
11/14/2018

53. Getting Started Contact Internet2 for more information
General Information:
Technical Contact:
Receive Announcements
BWCTL
OWAMP
NDT
11/14/2018

54. Participation (1) Set up BWCTL, OWAMP, NDT Beacons
Place tool beacons “strategically”
Consider locations
Determine policy
Establish limits
“Register” beacons
Let others know you are doing testing
Use the Global PMP Directory
11/14/2018

55. Participation (2) Set up a measurement domain
Install piPEs software
Store performance data
Make performance data available via web service
Make visualization CGIs available
Set up a local regularly scheduled mesh
11/14/2018

56. Participation (3) Establish Peering Agreements Support regular testing
Examples: GiGaPoP, application community
11/14/2018

57. Resources Available Case Studies Discussion Forum Global PMP Directory
Please alert us to new Case Studies
Discussion Forum
Global PMP Directory
Make sure your beacons can be found by others
11/14/2018

58. The Community Application Communities Measurement Domains
HENP, VLBI
Measurement Domains
ITECs, CERN, GEANT
Collaboration Partners
DANTE, ESnet, APAN
Standards Integration
GGS NMWG, IETF
NLANR/DAST Advisor, MonALISA
11/14/2018

59. Your Participation Matters
We can do partial path analysis, although making sense of the results is till a big issue.
We can speak the same measurement language, although it’s still evolving.
We are working together in growing numbers, but we need critical mass (become de facto standard).
We need to be able to find each other.
We need to be able to verify each other’s identity.
11/14/2018