Warren Matthews and Les Cottrell (SLAC) IEPM/PingER Internet End-to-end Performance Monitoring and the PingER project Warren Matthews and Les Cottrell (SLAC) National Collaboratory Middleware and Network Research Project Review, ANL, August 18-20, 2003.

Overview A little History Evaluate the progress Assess the Value Interactions with other projects Elements that should be added Summary 11/9/2018 6:41 PM

History Ping End-to-end Reporting Began early 1995 Monitor network performance to sites collaborating with SLAC ESnet Network Monitoring Task Force (NMTF) Extended to several DoE labs, strong support from FNAL 1997 ICFA created Network Task Force PingER spreads worldwide Funded by DOE/MICS

Recently In 2001, extended PingER to include bandwidth testing IEPM-BW End-to-end user perception for high performance bulk-transfer Iperf, bbftp, GridFTP … Heavy network impact compared to lightweight PingER

Current Status PingER funding is under Thomas Ndousse DoE/MICS funding runs out at end of year Continues to be extremely useful Most recently began working with ICTP/eJDS to quantify the Digital Divide MAGGIE proposal to develop/extend high performance monitoring (with PSC, ICIR, LBNL)

Overview Evaluate the progress A little History Assess the Value Interactions with other projects Elements that should be added Summary 11/9/2018 6:41 PM

PingER Mature, Successful Widely used in HENP Utilization has been extended beyond HENP EDG, IAEA, XIWT ICTP/eJDS Many others Continues to be extended to meet new needs Better visualization, web services access to data …

PingER Methodology Simple ping monitoring 1 ping to prime caches Send, size Default is 10x100 Byte pkts, 10x1000 Byte pkts Record ping packet loss and RTT Derive unreachability, quiescence, unpredictability, jitter, TCP throughput Also out-of-order packets, duplicate packets

PingER Software Monitoring Analysis Visualization Available from SLAC/FNAL websites Package 11/9/2018 6:41 PM

Using PingER Since 1995 Trouble shooting Identify Sites to Upgrade Choosing a provider Setting expectations for VoIP Routing Choices for multihomed sites Compare with http, ftp Strong correlation 11/9/2018 6:41 PM

PingER Deployment Currently 36 monitoring sites in 14 countries 473 target sites in 79 countries 99% of the worlds on-line population Most extensive end-to-end active R&E network monitoring worldwide Special BaBar, PPDG, Digital Divide. etc groups and pages

End-to-end Monitoring In reality, most projects monitor end-to-end performance End host effects are unavoidable Internet2 end-to-end Performance Initiative (e2epi) has recognized this Most useful to users. 11/9/2018 6:41 PM

IEPM-BW Throughput Monitoring Analysis Traceroute Iperf (+quick iperf), BBftp, BBCP (mem and disk) ABWE (available bandwidth) GridFTP, UDPMON Web100 Netflow Analysis

IEPM-BW Deployment Currently 10 monitoring sites SLAC, FNAL, GATech (SOX) INFN (Milan), NIKHEF, APAN (Japan) UMich, Internet2 (Ann Arbor) UManchester, UCL (UK) 50 unique target sites 11/9/2018 6:41 PM

Using IEPM-BW Usual Also on both testbeds and production nets Baselines Troubleshooting Setting expectations Also on both testbeds and production nets Compare measurement tools (ping vs ABwE vs. iperf/quick iperf vs bbcp vs GridFTP vs tsunami) Compare advanced TCP stacks Eliminate need for multiple streams Look at non TCP bulk transfer

Overview Assess the Value A little History Evaluate the progress Interactions with other projects Elements that should be added Summary 11/9/2018 6:41 PM

Examples Long term trends Short term glitches Troubleshooting Upgrades Vacations Peering 11/9/2018 6:41 PM

2Mbps vacation Multiple OC12s Traffic on ESnet has doubled every year

Ten-155 became operational on December 11. Smurf Filters installed on To North America Smurf Filters installed on NORDUnet’s US connection. To Western Europe

Traffic Typically, Internet traffic is 70% http

Conclusions Establish layer 3 connectivity exists Iperf vs Quick iperf BBftp vs BBCP => implementation IPERF vs BBftp => CPU, Disk Packet Loss < 0.1% TCP/IP must be tuned on high-speed long delay paths Web100/Net100

eJDS PingER continues to be useful Recently joined with electronic journal distribution service (eJDS) Distribute physics journals to member around the world Particularly concerned with quantifying the Digital Divide 11/9/2018 6:41 PM

Limitations ICMP Rate limiting Scheduling with cron Do not monitor routers Rate limiting Blocking is common, especially in developing countries However, study indicates low impact from rate limiting Scheduling with cron

Overview Interactions with other projects A little History Evaluate the progress Assess the Value Interactions with other projects Elements that should be added Summary 11/9/2018 6:41 PM

Comparison to Other Projects Surveyor RIPE AMP NIMI SCNM XIWT NetPhysics

Comparisons Typically results were closely correlated. Often tools complement each other and combined provide insight into network behaviour. Derived throughput from equation of Mathis et al (BW~MSS/(RTT*sqrt(loss)) ) shows good agreement 11/9/2018 6:41 PM

Publishing Network Performance information is critical to the Grid vision Application steering Working with GGF/NMWG & PPDG Monitoring data is available as prototype Web Service OGSI Grid service under development 11/9/2018 6:41 PM

Internet2 PIPES E2e pi PIPES infrastructure IEPM-BW Job manager MAGGIE Analysis Engine 11/9/2018 6:41 PM

Available Bandwidth Estimator (ABwE) Tool under development by SLAC/Rice Part of the DoE/SCIDAC INCITE project Light weight 60 packets in 1 second Iperf 35,000 packets/s for 10-20 seconds No need to tune windows/streams Replace iperf in test engine FreeBSD version created for Abilene Backbone Measurement Infrastructure

Quick Iperf Iperf is the tool of choice for many admins. Considered accurate but intrusive. Errors due to long slow start Use web100 to detect end of slow start. Modify iperf client. Web100 required on client only. Measurement within 10% Save 94% time, 92% traffic

PingER-6 SLAC has native IPv6 service from ESnet PingER ported to IPv6 Monitoring started in November 1999 41 Sites in 10 countries edu/ac.*, net/net.*, com/co.* 11/9/2018 6:41 PM

PingER -vs- PingER6 RTT between SLAC and Purdue in Nov and Dec 1999. IPv6 IPv4 11/9/2018 6:41 PM

Overview Elements that should be added A little History Evaluate the progress Asses the Value Interactions with other projects Elements that should be added Summary 11/9/2018 6:41 PM

MAGGIE Need to further develop IEPM-BW On-demand measurements, visualization, automated trouble shooting Measurement and Analysis for the Global Grid and Internet End-to-end performance A secure, scalable measurement infrastructure providing measurement, analysis and access to data.

Security and scheduling MAGGIE ICIR PSC Akenti IEPM-BW Measurement Engine SLAC NWS FNAL NIMI Security and scheduling Other tools MAGGIE ANL SCIDAC Publishing AMP Fault Finding Analysis Engine NMWG LBNL UCL SLAC RIPE SLAC

Overview Summary A little History Evaluate the progress Assess the Value Interactions with other projects Elements that should be added Summary 11/9/2018 6:41 PM

Meeting the Objectives (1/4) Evaluate the Progress The problem: The user cannot assume the network will be there. Even if it is, the user cannot assume it will perform to their expectation. The vision (realized): PingER has set expectation, provides data for troubleshooting, provides data for research. Continues to be useful. A Unique contribution: Probably the largest monitoring project in the world. IEPM-BW comparing tools, leveraging other efforts.

Meeting the Objectives (2/4) Assess the value PingER is widely used and continues to be useful. Goals get more ambitious Challenges remain 11/9/2018 6:41 PM

Meeting the Objectives (3/4) Interactions Across Projects Long history of involvement in other projects HENP, ESnet, Grid, High Performance, ICFA-SCIC Friends, colleagues and contacts throughout the world (Other worlds coming soon) Bright future for MAGGIE.

Meeting the Objectives (4/4) Assess the Integrated Impact The contribution to the big picture by IEPM-PingER, IEPM-BW and especially the need for MAGGIE have been summarized by Mary-Anne and Thomas But they may not have known it 11/9/2018 6:41 PM

The Big Picture All of the National Collaboratory and Network Research projects have specific goals and objectives, but all of you involved in those projects are also part of a much larger, longer term effort, namely creating an infrastructure that will enable geographically separated scientists to effectively work together as a team and that will facilitate remote access to both facilities and data. -Mary-Anne and Thomas 11/9/2018 6:41 PM

Toward a Monitoring Infrastructure Certainly the need DOE Science Community, SCIDAC Testbed Grid, Large Scale Networking Troubleshooting / E2Epi Many of the ingredients Many monitoring projects Many tools PIPES, MAGGIE (Cross domain) 11/9/2018 6:41 PM

Summary “Unfortunately, network management research has historically been very under-funded, because it is difficult to get funding bodies to recognize this as legitimate networking research.” Sally Floyd IAB Concerns & Recommendations Regarding Internet Research & Evolution. http://www.ietf.org/internet-drafts/draft-iab-research-funding-00.txt

Links Accompanying paper IEPM-BW Home 7 papers and 35 talks in the last 12 months ABwE RIPE-TT E2E PI GGF NMWG AMP TroubleShooting Quick Iperf 11/9/2018 6:41 PM

Credits Connie Logg, Jerrod Williams (SLAC), Jiri Navratil (CESnet/SLAC), David Martin, Frank Nagy, Al Thomas, Maxim Grigoriev (FNAL), Fabrizio Coccetti (INFN/SLAC). Brian Tierney, Eric Boyd, Jeff Boote, Matt Zekauskas, Matt Mathis, Russ Hobby, Vern Paxson, Andy Adams, kc Claffy, Iosif Legrand, Ajay Tirumala, Tom Dunigan. Local admins and other volunteers DoE/MICS