1. Network Monitoring today: why, how, challenges, infrastructures, federations and the Grid
Prepared by Les Cottrell, SLAC, for the
Grid Performance Workshop
UCL, May 12-13, 2004
Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2. Why (Can’t manage what you can’t measure)
Need measurements for both production networks & tesbeds:
Planning, setting expectations, policy/funding
Trouble-shooting: reliability & performance
Problems may not be logical, e.g. most Internet problems caused by operator error (Sci Am Jun’03), most LAN problems are Ethernet duplex, host config, bugs
Made hard by transparency, size & rate of change of network
A distributed system is one in which I can’t get my work done because a computer I never heard of has failed. Butler Lampson
Application steering (e.g. Grid data replication)
E2E performance problem is THE critical user metric

3. E.g. Policy - trends
C. Asia, Russia, S.E. Europe, L. America, M. East, China: 4-5 yrs behind
India, Africa: 7 yrs behind
S.E. Europe, Russia: catching up
Latin Am., Mid East, China: keeping up
India, Africa: falling behind
Important for policy makers
Spreadsheet \cottrell\iepm\esnet-to-all-longterm.xls
CERN data only goes back to Aug-01. It confirms S.E. Europe & Russia are catching up, and India & Africa are falling behind
Note for Africa only one host in Uganda. Actually have been adding hosts 5 countries), but there is considerable disparity in performance so as add hosts from less developed countries the aggregate performance measured to Africa is dropping!
Ghana, Nigeria and Uganda are all satellite links with ms RTTs. The losses to Ghana & Nigeria are 8-12% while to Uganda they are 1-3%. The routes are different. The route from SLAC to Ghana uses ESnet-Worldcom-UUNET, Nigeria goes CalREN-Qwest-Teiianet-New Skies satellite, Uganda goes Esnet-Level3-Intelsat. For both Ghana and Nigeria there are no losses (for 100 pings) until the last hop when over 40 of 100 packets were lost. For Uganda the losses (3 in 100 packets) also occur at the last hop.
Worksheet: for trends: \\Zwinsan2\c\cottrell\iepm\esnet-to-all-longterm.xls
for Africa: \\Zwinsan2\c\cottrell\iepm\africa.xls
traceroute to ( ): 1-30 hops, 38 byte packets
1 rtr-core1-nethub.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
2 rtr-dmz1-ger.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
( ) ms (ttl=252!) ms (ttl=252!) ms (ttl=252!)
4 snv-pos-slac.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms (ttl=251!) ms (ttl=251!) ms (ttl=251!)
5 snvrt1-ge0-snvcr1.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms (ttl=250!) ms (ttl=250!) ms (ttl=250!)
ATM1-0.BR2.SJC1.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms ms ms
ATM3-0.XR1.SJC1.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms ms ms
8 0.so XL1.SJC1.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms (ttl=247!) ms (ttl=247!) ms (ttl=247!)
9 0.so TL1.SAC1.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms (ttl=246!) ms (ttl=246!) ms (ttl=246!)
10 0.so IL1.NYC9.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms (ttl=242!) ms (ttl=242!) 74.4ms (ttl=242!)
11 0.so IR1.NYC12.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms (ttl=241!) ms (ttl=241!) ms (ttl=241!)
12 so TR1.CPH3.ALTER.NET ( ) [AS702 - UUNET, An MCI Worldcom Company] 171 ms (ttl=241!) 171 ms (ttl=241!) 171 ms (ttl=241!)
13 POS5-0.XR1.CPH3.ALTER.NET ( ) [AS702 - UUNET, An MCI Worldcom Company] 172 ms (ttl=241!) 173 ms (ttl=241!) 171 ms (ttl=241!)
14 POS4-0-0.CR1.CPH2.ALTER.NET ( ) [AS702 - UUNET, An MCI Worldcom Company] 171 ms (ttl=240!) 172 ms (ttl=240!) 172 ms (ttl=240!)
15 FastEthernet GW1.CPH2.ALTER.NET ( ) [AS702 - UUNET, An MCI Worldcom Company] 212 ms (ttl=239!) 347 ms (ttl=239!) 408 ms (ttl=239!)
16 satworks.gw.dk.uu.net ( ) [AS702 - UUNET DK Block 4] 226 ms (ttl=238!) 163 ms (ttl=238!) 163 ms (ttl=238!)
( ) [AS702 - UUNET DK Block 4] * 907 ms 865 ms# 43% loss on 100 pings (0 losses until this hop)
asoju.oauife.edu.ng
traceroute to asoju.oauife.edu.ng ( ): 1-30 hops, 38 byte packets
1 rtr-core1-nethub.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
2 rtr-dmz1-ger.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
3 i2-gateway.stanford.edu ( ) ms ms ms
4 STAN.POS.calren2.NET ( ) [AS32 - BN-CIDR ] ms ms ms
5 SUNV--STAN.POS.calren2.net ( ) [AS NET-C2-NORTH] ms ms ms
6 QSV-M10-2-C2.GE.calren2.net ( ) [AS CENIC-DCP] ms (ttl=249!) ms (ttl=249!) ms (ttl=249!)
( ) [AS209 - Qwest Communications] ms (ttl=247!) ms (ttl=247!) ms (ttl=247!)
( ) [AS209 - Qwest Communications] ms ms ms
( ) [AS209 - Qwest Communications] ms (ttl=248!) ms (ttl=248!) ms (ttl=248!)
( ) [AS209 - Qwest Communications] ms (ttl=245!) ms (ttl=245!) ms (ttl=245!)
11 sca-bb1-pos0-0-0.telia.net ( ) [AS TELIANET-BLK] ms (ttl=243!) ms (ttl=243!) ms (ttl=243!)
12 chi-bb1-pos1-0-0.telia.net ( ) [AS TELIANET-BLK] ms (ttl=242!) ms (ttl=242!) ms (ttl=242!)
13 nyk-bb1-pos0-1-0.telia.net ( ) [AS TELIANET-BLK] ms (ttl=238!) 76.1 ms (ttl=238!) ms (ttl=238!)
14 nyk-bb2-pos1-0-0.telia.net ( ) [AS TELIANET-BLK] ms (ttl=239!) ms (ttl=239!) ms (ttl=239!)
15 ldn-bb2-pos1-3-0.telia.net ( ) [AS TELIANET-BLK] 148 ms (ttl=237!) 148 ms (ttl=237!) 147 ms (ttl=237!)
16 ldn-b1-pos11-0.telia.net ( ) [AS TELIANET-BLK] 147 ms (ttl=236!) 148 ms (ttl=237!) 147 ms (ttl=236!)
17 ldn-th-i1-srp1-0.telia.net ( ) [AS TELIANET-BLK] 147 ms (ttl=234!) 147 ms (ttl=234!) 148 ms (ttl=234!)
18 new-skies ldn-th-i1.c.telia.net ( ) [AS TELIANET-BLK] 141 ms (ttl=242!) 141 ms (ttl=242!) 141 ms (ttl=242!)
19 rtr-cor01-pos6-0-0.cha.newskies.net ( ) [AS New Skies Satellites]142 ms (ttl=241!) 142 ms (ttl=241!) 142 ms (ttl=241!)
20 rtr-dvb01-gi cha.newskies.net ( ) [AS New Skies Satellites] 142 ms (ttl=240!) 142 ms (ttl=240!) 143 ms (ttl=240!)
21 * * *
reverse.newskies.net ( ) [AS701 - UUNET - AS 701] ms (ttl=238!) 970 ms (ttl=238!) 988 ms (ttl=238!)# 44% loss on 100 pings for this hop, 0 for others
mail2.starcom.co.ug
traceroute to mail2.starcom.co.ug ( ): 1-30 hops, 38 byte packets
1 rtr-core1-nethub.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
2 rtr-dmz1-ger.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
( ) ms (ttl=252!) ms (ttl=252!) ms (ttl=252!)
4 snv-pos-slac.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms (ttl=251!) ms (ttl=251!) ms (ttl=251!)
5 snvrt1-ge0-snvcr1.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms (ttl=250!) ms (ttl=250!) ms (ttl=250!)
6 paix-pa-snv.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms ms ms
7 gigabitethernet edge1.paix-sjo1.Level3.net ( ) [AS no more prtraceroute whiners ! Just kidding - we love you Nik.] ms ms ms
8 GigabitEthernet3-1.core1.SanJose1.Level3.net ( ) [AS no more prtraceroute whiners ! Just kidding - we love you Nik.] ms ms ms
9 ae0-55.mp1.SanJose1.Level3.net ( ) [AS no more prtraceroute whiners ! Just kidding - we love you Nik.] ms (ttl=246!) ms (ttl=246!) ms (ttl=246!)
( ) [AS no more prtraceroute whiners ! Just kidding - we love you Nik.] ms (ttl=245!) ms (ttl=245!) ms (ttl=245!)
11 so mp1.London2.Level3.net ( ) [AS Level 3 RIPE block] 152 ms (ttl=244!) 152 ms (ttl=244!) 153 ms (ttl=244!)
12 so mp1.London1.Level3.net ( ) [AS Level 3 RIPE block] 152 ms (ttl=243!) 152 ms (ttl=243!) 152 ms (ttl=243!)
13 so gar1.London1.Level3.net ( ) [AS Level 3 RIPE block] 158 ms (ttl=242!) 158 ms (ttl=242!) 158 ms (ttl=242!)
14 pos2-0.metro1-londencyh00.London1.Level3.net ( ) [AS Level 3 RIPE block] 158 ms 158 ms 160 ms
( ) [AS Level 3 (ex Businessnet)] 154 ms (ttl=240!) 153 ms (ttl=240!) 153 ms (ttl=240!)
16 fus-rt001-stm core.globalconnex.net ( ) [AS Intelsat Specific route within RIPE LIR allocation] 178 ms (ttl=239!) 177 ms (ttl=239!) 176 ms (ttl=239!)
17 fus-rt004-fe-0-0-v2.its-dvb.globalconnex.net ( ) [AS Intelsat Specific route within RIPE LIR allocation] 172 ms 171 ms 171 ms
18 * * *
19 * * *
20 * * *
21 mail2.starcom.co.ug ( ) ms ms ms # Loss of 3% for both 100 and 1400 byte packets

4. Esnet-LosNettos segment in the path
E.g. Changes in network topology (BGP) result
in dramatic change in performance
Hour
Samples of traceroute trees generated from the table
Los-Nettos (100Mbps)
Remote host
Snapshot of traceroute summary table
Notes:
1. Caltech misrouted via Los-Nettos 100Mbps commercial net 14:00-17:00
2. ESnet/GEANT working on routes from 2:00 to 14:00
3. A previous occurrence went un-noticed for 2 months
4. Next step is to auto detect and notify
Drop in performance
(From original path: SLAC-CENIC-Caltech
to SLAC-Esnet-LosNettos (100Mbps) -Caltech )
Back to original path
Dynamic BW capacity (DBC)
Changes detected by
IEPM-Iperf and AbWE
Mbits/s
Available BW = (DBC-XT)
Cross-traffic (XT)
Esnet-LosNettos segment in the path
(100 Mbits/s)
ABwE measurement one/minute for 24 hours Thurs Oct 9 9:00am to Fri Oct 10 9:01am

5. Methods Active Measurement probes: Passive tools:
Include: Ping, traceroute, owamp, pathload/abwe, major apps (e.g. bbftp, bbcp, GridFTP…)
Typically used for end-to-end testing
Injects data into network, can be non-negligible
Passive tools:
Include: SNMP, NetFlow, OCxMon, NetraMet, cflowd, SCNM
Typically used at border or inside backbones
SNMP heavily used for utilization, errors on LAN & backbones
Flows for traffic characterization and intrusion detection
Need access to network devices (e.g. routers, taps)
Can generate a lot of data
Need to put together data from multiple sources
Different probes, different source & destinations, network-centric & end-to-end
Iperf/ICMP traffic on Abilene was ~25% by volume for week 2004/03/08.

6. Some Challenges for Active monitoring
Bandwidth used, e.g. iperf etc. & apps
Sampling rate (Nyquist’s theorem),
Relevance to application needs
Measure loss to 10% on a path with 1 in 10K loss requires a million pings
For TCP tools: configuring windows at clients/servers and optimizing windows, streams
Some lightweight tools (e.g. packet pairs) not effective at >> 1Gbits/s
Many tools tuned for shared TCP/IP nets not for dedicated circuits
Simplifying use and understanding for end-user
Automating problem detection & resolution,
Alternatives to iperf include ttcp, nuttcp, UDP tools: UDT, SOBAS, RBUDP
Nyquist’s theorem: if sampling rate < 2 * max frequency in spectrum of sampled signal, then we cannot reconstruct the signal
> Gbps problems for packet pairs/trains include: lack of OS clock granularity, interrupt coalescing, TOE …

7. Network Impact
Heavyweight: iperf, bbcp, bbftp, GridFTP (IEPM-BW, PiPES …)…
Noticeable impact, run infrequently (e.g. hourly) , and for short time (e.g. tens of seconds), only small number of sites
Need scheduling
Close to what applications see
Lightweight: Ping, traceroute, ABwE etc.
E.g. PingER, AMP
Can do on demand, no need to set things up in advance (no server to install), no scheduling needed, can involve thousands of sites
Medium weight (ABwE, pathload etc.)
E.g. IEPM-LITE, Scriptroute
Needs server/mirror install, low traffic (ABwE 1kbps avg), no scheduling

8. Infrastructures
Many measurement projects with different emphases, different communities
Passive (usually requires network control, used at borders and on backbones, e.g. MICSmon/Netflow, ISP/SNMP, SCNM)
Active: amount of network “pollution”
Lightweight (PingER, AMP, Surveyor, RIPE …)
Medium weight (PiPES, NWS, IEPM-Lite …)
Heavy weight/hi-perf (IEPM-BW, NTAF
End-to-end vs net centric (skitter, macroscopic views)
Repetitive (PingER, AMP, IEPM, PiPES, NWS, NTAF, …)
On demand, or non-production (NDT, NIMI, PiPES …)
Dedicated hardware (AMP, RIPE, NDT, PlanetLab …)
Hierarchical (e.g. AMP) vs Full mesh (e.g. PingER)
For a table comparing 13 public domain infrastructures, see:

9. NMI challenges
Sustaining deployment/operation in multi-agency / international world
Scaling beyond hundreds of hosts very hard over the long term:
Hosts change, upgrade, new OS
No control over shared hosts
Depend on friendly admin contacts who may be busy, uninterested, have moved etc.
Policy/fears at remote site can make dedicated changes painful
web100 upgrades not coordinated with Linux upgrades
New TCP kernel upgrades not coordinated with OS upgrades
Hosts age, become measurement bottleneck
Need constant upgrades for dedicated hosts
Probes (iperf etc.) change: new features, patches
Scheduling to prevent interference for heavyweight tests
Appropriate security: keeping track of credentials, upgrade/patches, multiple-policies, port blocking

10. So Recognize
Unrealistic to think multiple admin domains will all deploy one and the same infrastructure
Scaling and interests make unrealistic
Multiple-domain, multi-infrastructures will be deployed
Need to tie together heterogeneous collection of monitoring systems
Create a federation of existing NMIs
Infrastructures work together
Share data with peer infrastructures and others using a common set of protocols for describing, exchanging & locating monitoring data (e.g. GGF NMWG)
Enables much improved overall view of network using multiple measurement types from multiple sources

11. MAGGIE Proposal
Measurement and Analysis for the Global Grid and Internet End-to-end performance
Contribute to, utilize the GGF NMWG naming hierarchy and the schema definitions for network measurements
Develop tools to allow sharing
Web services based
Integrate information from multiple sources
Brings together several major infrastructure participants: LBNL (NTAP, SCNM), SLAC (IEPM-PingER/BW), Internet2 (PiPES, NDT), NCSC (NIMI), U Delaware, ESnet
Will work with others, e.g. MonALISA, AMP, UltraLight, PPDG, StarLIght, UltraScienceNet

12. Federation goals Appropriate security Interoperable
Useful for applications, network engineers, scientists & end users
Easy to deploy & configure
As un-intrusive as possible
As accurate & timely as possible
Identify most useful features of each NMI to improve each NMI faster than working alone

13. From measurements to the Grid
Given measurements or the ability to make them, how is that useful to the Grid?
Grid application needs to place or retrieve data with high performance and robustness
Maybe use multiple sites in parallel
Some similarities with P2P such as BitTorrent, eDonkey, Kazaa, Gnutella etc. chunking of files
But different goals
Grid few well-known sites known in advance, high-perf links, does not face legal troubles, free-riding etc. of P2P
Need to find optimal site(s) to get data from based on expected achievable throughput
Can use existing measurements & predictions
Can make measurements on demand

14. Use Existing Measurements
Need a way to discover “relevant” measurements
Between possible pairs or “closely” related pairs
Need a request protocol/schema
Need a response schema for results
GGF NMWG are working on these issues.

15. On-demand Application somehow knows where chunks of data may be found
Makes measurements of bandwidth from application site to chunk locations
Assumes have appropriate servers at chunk locations (e.g. ABwE reflector), or use ubiquitous server (e.g. ping)
Uses this to choose initial locations to get a complete set of chunks from

16. Challenges
Optimal chunk locations may change during transfer (chunk location may become inaccessible, or its performance may drop)
So need measurements during transfer
This may make it attractive to instrument the application so it can make its own measurements on the data being transferred
Need library to simplify modifying each application
Throughput advantages of multiple parallel site transfers may be no better than multiple parallel streams between a well connected source & destination (may share the same bottleneck)
Do network measurements relate to file transfer rates?

17. NMI Challenges: Reduce “Wizard gap”
Applications cross agency AND international funding boundaries (includes Digital Divide)
Incent multi-disciplinary teams, including people close to scientists, operational teams
Make sure what is produced is used, tested in real environment, include deployment in proposals
Network management research historically underfunded, because it is difficult to get funding bodies to recognize as legitimate networking research, IAB
Without excellent trouble-shooting capabilities, the Grid vision will fail
The other main reason the grid could fail is security.

18. More Information Some Measurement Infrastructures:
CAIDA list:
AMP: amp.nlanr.net/, PMA
IEPM/PingER home site: www-iepm.slac.stanford.edu/
IEPM-BW site: www-iepm.slac.stanford.edu/bw
NIMI: ncne.nlanr.net/nimi/
RIPE:
NWS: nws.cs.ucsb.edu/
Internet2 PiPES: e2epi.internet2.edu/
Tools
CAIDA measurement taxonomy:
SLAC Network Tools:
Internet research needs:

19. Automatic Step change Detection
Too many graphs to review each morning!
Motivated by drop in bandwidth between SLAC &Caltech
Started late August 2003
Reduced achievable throughput by factor of 5
Not noticed until October 2003
Caused by faulty routing over commercial network
After notifying ISP, it was fixed in 4 hours!
See for details
SLAC Caltech achievable throughput April – November 2003
Started

20. Automatic available bandwidth step change detection
Still developing, evolving from earlier work:
Arithmetic weighted moving averages
NLANR “Plateau” algorithm work, see
Goals catches important changes, with few false alerts
Current parameter settings:
History buffer duration: 600 mins
trigger buffer duration: 30 mins
threshold: 40%
sensitivity: 2 (NLANR typically set to 1)
Differences from NLANR algorithm:
Use standard deviations instead of variances
Use threshold instead of count of triggers
Low variation prohibition (flatlining avoidance) use 10% instead of 20%
Track both increases an decreases
In progress & futures:
Evaluate optimum parameters
Generate alarms with filters
Possibly look at medians and percentile to replace mean and standard deviations
See if can fold in periodic (e.g. diurnal) changes
Look at increases AND decreases (e.g. restoration)

21. Plateau algorithm Roughly speaking:
Has a history buffer to describe past behavior
History buffer duration currently 600 mins
Plus a trigger buffer of data suggesting a change
Trigger buffer duration (evaluating typically mins) indicates how long the change has to occur for
History mean (m) and std. dev. (s) use by trigger selector
If new_value outside m +- sensitivity*s add to trigger buffer
If new_value outside m +- 2*sensitivity*s then also an outlier (don’t add to stats)
Else goes in history buffer
Look for big difference in trigger and history buffer means
Also looking at Principal Component Analysis (Crovella et al) of multi variables (e.g. capacity, cross-traffic, RTT …) which may also help with diurnal changes.

22. Examples SLAC to Caltech available bandwidth April 6-8, 2004 Alerts
Route change
History duration: 600 mins, trigger duration: 30 mins, threshold: 40%, sensitivity: 2
With trigger duration: 60 only see one alert, with trigger duration: 10 catch alerts
Unreachable
SLAC - NIKHEF
Route changes
SLAC to NIKHEF (Amsterdam)
Mbit/s
Avail BW