1. 1 Correlating Internet Performance & Route Changes to Assist in Trouble- shooting from an End-user Perspective Les Cottrell, Connie Logg, Jiri Navratil SLAC Passive and Active Monitoring Workshop Antibes, Juan-les-Pins, France April 19-20, 2004 www.slac.stanford.edu/grp/scs/net/talk03/pam04.ppt Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2. 2 Outline Set of integrated measurement tools to aid in troubleshooting for end “user” Traceroute measurements/analysis Topology visualization Lightweight bandwidth estimation Overall visualization Level change anomaly automated detection Correlation of performance & route changes

3. 3 Traceroute measurement Every 10 minutes for each host –Run standard traceroute 2 sec timeout, 1 query/hop, <= 30hops For some hosts use ICMP traceroute –End host responds (7/40) –Intermediate host responds (1/40) Two cases UDP probes better than ICMP One case neither ICMP or UDP probes help –Both forward & reverse (use ssh for reverse route) Need ssh access to remote host for rev trace –Else no reverse route (not a disaster)

4. 4 Significant changes Compare current and previous traceroutes: –If traceroute reports “unknown host” => unknown (!) –Else for each hop/node If both current & previous hops have valid IP addresses (i.e. router does not respond & traceroute reports “*”) –If different i.e. some kind of Route Change has occurred »If IPs same for 1 st 3 octets then => same subnet/colo ( : ) »Else if IPs in same AS then => same AS ( a ) »Else significant change => assign unique route number If only one hop different => color route # orange ( ) Else color route => color route # red ( ) –Elseif 30 hops => no route change but last hop unreachable (|) »If last hop not pingable => color red (|) –Else => no route change (●) Elseif one or both IPs are “*” => route change unclear (*) If “Icmp checksum is wrong” color character orange If significant bandwidth change color cell

5. 5 Route table Compact so can see many routes at once History navigation Multiple route changes (due to GEANT), later restored to original route Available bandwidth Raw traceroute logs for debugging Textual summary of traceroutes for email to ISP Description of route numbers with date last seen User readable (web table) routes for this host for this day Route # at start of day, gives idea of root stability Mouseover for hops & RTT

6. 6 Another example TCP probe type Host not pingable Intermediate router does not respond ICMP checksum error Level change Get AS information for routes

7. 7 Topology Choose times and hosts and submit request DL CLRCCLRC CLRC IN2P3 CESnet ESnet JAnet GEANT Nodes colored by ISP Mouseover shows node names Click on node to see subroutes Click on end node to see its path back Also can get raw traceroutes with AS’ Alternate rt SLAC Alternate route Hour of day

8. 8 Available bandwidth Uses ABwE/Abing (packet pair dispersion) –Needs server at remote end or ssh to launch server –Fast (< 1 sec) –Lightweight < 40 packets for both forward & reverse estimates (5800 Bytes) –Uses min delay for capacity –Inter packet dispersion for cross-traffic –Available BW = Capacity (min RTT) – Cross-traffic (var) Good agreement with other methods Even if poor absolute agreement (25% cases) can spot changes –Also provides RTT Make measurements to about 60 hosts at 5 minute intervals (deployed in IEPM, MonALISA, PlanetLab)

9. 9 Available Bandwidth From SLAC to Caltech Mar 19, 2004 Dynamic bandwidth capacity (DBC) Available bandwidth = DBC – X-traffic Cross-traffic Iperf

10. 10 Achievable throughput & file transfer IEPM-BW –High impact (iperf, bbftp, GridFTP …) measurements 90+-15 min intervals Select focal area Fwd route change Rev route change Min RTT Iperf bbftp iperf1 abing Min RTT

11. 11 Put it all together Two examples –Agreement of iperf & abing –Route changes and available bandwidth

12. 12 AbWE Iperf 28 days bandwidth history. During this time we can see several different situations caused by different routing from SLAC to CALTECH Drop to 100 Mbits/s by Routing (BGP) errors Drop to 622 Mbits/s path back to new CENIC path New CENIC path 1000 Mbits/s Reverse Routing changes Forward Routing changes Scatter plot graphs of Iperf versus ABw on different paths (range 20– 800 Mbits/s) showing agreement of two methods (28 days history) RTT Bbftp Iperf 1 stream

13. 13 Changes in network topology (BGP) can result in dramatic changes in performance Snapshot of traceroute summary table Samples of traceroute trees generated from the table ABwE measurement one/minute for 24 hours Thurs Oct 9 9:00am to Fri Oct 10 9:01am Drop in performance (From original path: SLAC-CENIC-Caltech to SLAC-Esnet-LosNettos (100Mbps) -Caltech ) Back to original path Changes detected by IEPM-Iperf and AbWE Esnet-LosNettos segment in the path (100 Mbits/s) Hour Remote host Dynamic BW capacity (DBC) Cross-traffic (XT) Available BW = (DBC-XT) Mbits/s Notes: 1. Caltech misrouted via Los-Nettos 100Mbps commercial net 14:00-17:00 2. ESnet/GEANT working on routes from 2:00 to 14:00 3. A previous occurrence went un-noticed for 2 months 4. Next step is to auto detect and notify Los-Nettos (100Mbps)

14. 14 Automatic Step change Detection Too many graphs to review each morning! Motivated by drop in bandwidth between SLAC &Caltech –Started late August 2003 –Reduced achievable throughput by factor of 5 –Not noticed until October 2003 –Caused by faulty routing over commercial network –After notifying ISP, it was fixed in 4 hours! –See http://www.slac.stanford.edu/grp/scs/net/case/caltech/ for detailshttp://www.slac.stanford.edu/grp/scs/net/case/caltech/ SLAC Caltech achievable throughput April – November 2003Started

15. 15 Automatic available bandwidth step change detection Still developing, evolving from earlier work: –Arithmetic weighted moving averages –NLANR work, see http://byerley.cs.waikato.ac.nz/~tonym/papers/event.pdf http://byerley.cs.waikato.ac.nz/~tonym/papers/event.pdf Roughly speaking: –Has a history buffer to describe past behavior History buffer duration currently 600 mins –Plus a trigger buffer of data suggesting a change Trigger buffer duration (evaluating typically 10-60 mins) indicates how long the change has to occur for –History mean (  ) and std. dev. (  ) use by trigger selector If new_value outside  +- sensitivity  add to trigger buffer If new_value outside  +- 2*sensitivity  then also an outlier (don’t add to stats) Else goes in history buffer

16. 16 Algorithm If this is a trigger value compare with  and save direction of change If this is a trigger and the direction has changed, reset trigger buffer –Move trigger data to history buffer, recalculate stats, clear trigger buffer If trigger buffer full calculate trigger mean  t and  t –If (  -  t )/  threshold then a & reset trigger buffer –Else remove oldest value from trigger buffer

17. 17 Examples SLAC to Caltech available bandwidth April 6-8, 2004 Alerts History duration: 600 mins, trigger duration: 30 mins, threshold: 40%, sensitivity: 2 With trigger duration: 60 only see one alert, with trigger duration: 10 catch alerts Route change SLAC to NIKHEF (Amsterdam) Mbit/s Avail BW Route changesSLAC - NIKHEF Unreachable

18. 18 BW vs Route changes Route & throughput changes from 11/28/03 thru 2/2/04 –Most (80%) route changes do not result in throughput change –About half throughput changes are due to route changes Location (# nodes) # route chgs # with thru inc. # with thru decr. # thru chgs # thru with rte # thru chg w/o rte Europe (8)370241064 Canada & US (21) 12062425714922 1 Japan (13)14222945

19. 19 More Information ABwE: –http://moat.nlanr.net/PAM2003/PAM2003papers/3781.pdf IEPM –http://www-iepm.slac.stanford.edu/ –http://moat.nlanr.net/PAM2003/PAM2003papers/3768.pdfhttp://moat.nlanr.net/PAM2003/PAM2003papers/3768.pdf Traceroute examples: –www.slac.stanford.edu/comp/net/iepmlite/tracesummaries/to day.htmlwww.slac.stanford.edu/comp/net/iepmlite/tracesummaries/to day.html Step change analysis –http://byerley.cs.waikato.ac.nz/~tonym/papers/event.pdfhttp://byerley.cs.waikato.ac.nz/~tonym/papers/event.pdf