End to end Internet Performance today Prepared by Les Cottrell, SLAC, for the Extending the Reach of Advanced Networking: Special International Workshop Arlington, VA., April 22, 2004 www.slac.stanford.edu/grp/scs/net/talk03/i2-overview-apr04.ppt PingER is arguably the most extensive set of measurements of the end-to-end performance of the Internet going back almost ten years. Measurements are available from over 30 sites in 13 countries to sites in over 100 countries. We will use the PingER results to: demonstrate how the Internet performance to the regions of the world has evolved over the last 9 years; identify regions that have poor connectivity, how far they are behind the developed world and whether they are catching up or falling further behind; and illustrate the correlation between the UN Technology Achievement Index and Internet performance. Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

Countries Monitored Monitor site Remote site Monitoring hosts 35 hosts 13 Countries Countries monitored contain over 90% of the world’s Internet connected population Need >1 host/country to reduce anomalies. Remote hosts >105 countries 560 sites 880 hosts 3650pairs

Loss to world from US Loss Rate < 0.1 to 1 % 1 to 2.5 % 2.5 to 5 % > 12 % 2001 Dec-2003 In 2001 <20% of the world’s population had Good or Acceptable Loss performance BUT by December 2003 It had improved to 77% Spreadsheet: v:Groups\scs\networking\netdev\tai-dec03.xls

Trends C. Asia, Russia, S.E. Europe, L. America, M. East, China: 4-5 yrs behind India, Africa: 7 yrs behind S.E. Europe, Russia: catching up Latin Am., Mid East, China: keeping up India, Africa: falling behind Spreadsheet \cottrell\iepm\esnet-to-all-longterm.xls CERN data only goes back to Aug-01. It confirms S.E. Europe & Russia are catching up, and India & Africa are falling behind Note for Africa only one host in Uganda. Actually have been adding hosts 5 countries), but there is considerable disparity in performance so as add hosts from less developed countries the aggregate performance measured to Africa is dropping! Ghana, Nigeria and Uganda are all satellite links with 800-1100ms RTTs. The losses to Ghana & Nigeria are 8-12% while to Uganda they are 1-3%. The routes are different. The route from SLAC to Ghana uses ESnet-Worldcom-UUNET, Nigeria goes CalREN-Qwest-Teiianet-New Skies satellite, Uganda goes Esnet-Level3-Intelsat. For both Ghana and Nigeria there are no losses (for 100 pings) until the last hop when over 40 of 100 packets were lost. For Uganda the losses (3 in 100 packets) also occur at the last hop. Worksheet: for trends: \\Zwinsan2\c\cottrell\iepm\esnet-to-all-longterm.xls for Africa: \\Zwinsan2\c\cottrell\iepm\africa.xls 10cottrell@flora02:~>ntrace www.ug.edu.gh traceroute to www.ug.edu.gh (213.237.174.3): 1-30 hops, 38 byte packets 1 rtr-core1-nethub.slac.stanford.edu (134.79.19.2) [AS3671 - SU-SLAC] 0.563 ms 0.344 ms 0.312 ms 2 rtr-dmz1-ger.slac.stanford.edu (134.79.135.15) [AS3671 - SU-SLAC] 0.365 ms 0.364 ms 0.344 ms 3 192.68.191.146 (192.68.191.146) 0.405 ms (ttl=252!) 0.406 ms (ttl=252!) 0.386 ms (ttl=252!) 4 snv-pos-slac.es.net (134.55.209.1) [AS293 - Energy Sciences Network (ESnet)] 0.774 ms (ttl=251!) 0.863 ms (ttl=251!) 0.751 ms (ttl=251!) 5 snvrt1-ge0-snvcr1.es.net (134.55.209.90) [AS293 - Energy Sciences Network (ESnet)] 0.823 ms (ttl=250!) 0.798 ms (ttl=250!) 0.820ms (ttl=250!) 6 188.ATM1-0.BR2.SJC1.ALTER.NET (204.255.174.49) [AS701 - UUNET, An MCI Worldcom Company] 2.37 ms 1.47 ms 1.39 ms 7 154.ATM3-0.XR1.SJC1.ALTER.NET (152.63.51.174) [AS701 - UUNET, An MCI Worldcom Company] 1.72 ms 2.23 ms 1.59 ms 8 0.so-0-0-0.XL1.SJC1.ALTER.NET (152.63.55.114) [AS701 - UUNET, An MCI Worldcom Company] 1.81 ms (ttl=247!) 1.77 ms (ttl=247!) 1.75 ms (ttl=247!) 9 0.so-3-0-0.TL1.SAC1.ALTER.NET (152.63.53.250) [AS701 - UUNET, An MCI Worldcom Company] 5.12 ms (ttl=246!) 5.98 ms (ttl=246!) 5.15 ms (ttl=246!) 10 0.so-7-0-0.IL1.NYC9.ALTER.NET (152.63.9.245) [AS701 - UUNET, An MCI Worldcom Company] 74.2 ms (ttl=242!) 74.8 ms (ttl=242!) 74.4ms (ttl=242!) 11 0.so-1-0-0.IR1.NYC12.ALTER.NET (152.63.23.62) [AS701 - UUNET, An MCI Worldcom Company] 75.3 ms (ttl=241!) 75.2 ms (ttl=241!) 73.7 ms (ttl=241!) 12 so-6-0-0.TR1.CPH3.ALTER.NET (146.188.7.193) [AS702 - UUNET, An MCI Worldcom Company] 171 ms (ttl=241!) 171 ms (ttl=241!) 171 ms (ttl=241!) 13 POS5-0.XR1.CPH3.ALTER.NET (146.188.2.214) [AS702 - UUNET, An MCI Worldcom Company] 172 ms (ttl=241!) 173 ms (ttl=241!) 171 ms (ttl=241!) 14 POS4-0-0.CR1.CPH2.ALTER.NET (146.188.3.141) [AS702 - UUNET, An MCI Worldcom Company] 171 ms (ttl=240!) 172 ms (ttl=240!) 172 ms (ttl=240!) 15 FastEthernet10-0-0.GW1.CPH2.ALTER.NET (146.188.30.115) [AS702 - UUNET, An MCI Worldcom Company] 212 ms (ttl=239!) 347 ms (ttl=239!) 408 ms (ttl=239!) 16 satworks.gw.dk.uu.net (213.237.172.50) [AS702 - UUNET DK Block 4] 226 ms (ttl=238!) 163 ms (ttl=238!) 163 ms (ttl=238!) 17 213.237.174.3 (213.237.174.3) [AS702 - UUNET DK Block 4] * 907 ms 865 ms# 43% loss on 100 pings (0 losses until this hop) 9cottrell@flora02:~>ntrace asoju.oauife.edu.ng traceroute to asoju.oauife.edu.ng (63.100.199.60): 1-30 hops, 38 byte packets 1 rtr-core1-nethub.slac.stanford.edu (134.79.19.2) [AS3671 - SU-SLAC] 0.541 ms 0.347 ms 0.319 ms 2 rtr-dmz1-ger.slac.stanford.edu (134.79.135.15) [AS3671 - SU-SLAC] 0.363 ms 0.357 ms 0.343 ms 3 i2-gateway.stanford.edu (192.68.191.83) 0.337 ms 0.337 ms 0.313 ms 4 STAN.POS.calren2.NET (171.64.1.213) [AS32 - BN-CIDR-171.64] 0.431 ms 0.403 ms 0.363 ms 5 SUNV--STAN.POS.calren2.net (198.32.249.73) [AS11423 - NET-C2-NORTH] 0.804 ms 0.820 ms 0.826 ms 6 QSV-M10-2-C2.GE.calren2.net (137.164.12.167) [AS2150 - CENIC-DCP] 0.907 ms (ttl=249!) 1.07 ms (ttl=249!) 0.923 ms (ttl=249!) 7 65.113.32.209 (65.113.32.209) [AS209 - Qwest Communications] 0.893 ms (ttl=247!) 0.907 ms (ttl=247!) 0.845 ms (ttl=247!) 8 205.171.14.97 (205.171.14.97) [AS209 - Qwest Communications] 0.870 ms 0.892 ms 0.932 ms 9 205.171.205.30 (205.171.205.30) [AS209 - Qwest Communications] 1.11 ms (ttl=248!) 1.26 ms (ttl=248!) 1.19 ms (ttl=248!) 10 205.171.1.66 (205.171.1.66) [AS209 - Qwest Communications] 2.31 ms (ttl=245!) 2.31 ms (ttl=245!) 1.94 ms (ttl=245!) 11 sca-bb1-pos0-0-0.telia.net (213.248.86.57) [AS1299 - TELIANET-BLK] 2.93 ms (ttl=243!) 2.65 ms (ttl=243!) 3.02 ms (ttl=243!) 12 chi-bb1-pos1-0-0.telia.net (213.248.80.33) [AS1299 - TELIANET-BLK] 47.8 ms (ttl=242!) 47.8 ms (ttl=242!) 48.1 ms (ttl=242!) 13 nyk-bb1-pos0-1-0.telia.net (213.248.80.5) [AS1299 - TELIANET-BLK] 74.6 ms (ttl=238!) 76.1 ms (ttl=238!) 75.0 ms (ttl=238!) 14 nyk-bb2-pos1-0-0.telia.net (213.248.80.14) [AS1299 - TELIANET-BLK] 93.3 ms (ttl=239!) 75.1 ms (ttl=239!) 75.3 ms (ttl=239!) 15 ldn-bb2-pos1-3-0.telia.net (213.248.65.37) [AS1299 - TELIANET-BLK] 148 ms (ttl=237!) 148 ms (ttl=237!) 147 ms (ttl=237!) 16 ldn-b1-pos11-0.telia.net (213.248.74.14) [AS1299 - TELIANET-BLK] 147 ms (ttl=236!) 148 ms (ttl=237!) 147 ms (ttl=236!) 17 ldn-th-i1-srp1-0.telia.net (193.45.0.132) [AS3301 - TELIANET-BLK] 147 ms (ttl=234!) 147 ms (ttl=234!) 148 ms (ttl=234!) 18 new-skies-01427-ldn-th-i1.c.telia.net (213.248.75.150) [AS1299 - TELIANET-BLK] 141 ms (ttl=242!) 141 ms (ttl=242!) 141 ms (ttl=242!) 19 rtr-cor01-pos6-0-0.cha.newskies.net (80.247.128.58) [AS17175 - New Skies Satellites]142 ms (ttl=241!) 142 ms (ttl=241!) 142 ms (ttl=241!) 20 rtr-dvb01-gi0-0-60.cha.newskies.net (80.247.128.162) [AS17175 - New Skies Satellites] 142 ms (ttl=240!) 142 ms (ttl=240!) 143 ms (ttl=240!) 21 * * * 22 63-100-199-60.reverse.newskies.net (63.100.199.60) [AS701 - UUNET - AS 701] 1391 ms (ttl=238!) 970 ms (ttl=238!) 988 ms (ttl=238!)# 44% loss on 100 pings for this hop, 0 for others 8cottrell@flora02:~>ntrace mail2.starcom.co.ug traceroute to mail2.starcom.co.ug (217.113.72.21): 1-30 hops, 38 byte packets 1 rtr-core1-nethub.slac.stanford.edu (134.79.19.2) [AS3671 - SU-SLAC] 0.510 ms 0.353 ms 0.313 ms 2 rtr-dmz1-ger.slac.stanford.edu (134.79.135.15) [AS3671 - SU-SLAC] 0.369 ms 0.391 ms 0.350 ms 3 192.68.191.146 (192.68.191.146) 0.476 ms (ttl=252!) 0.415 ms (ttl=252!) 0.359 ms (ttl=252!) 4 snv-pos-slac.es.net (134.55.209.1) [AS293 - Energy Sciences Network (ESnet)] 0.762 ms (ttl=251!) 0.798 ms (ttl=251!) 0.728 ms (ttl=251!) 5 snvrt1-ge0-snvcr1.es.net (134.55.209.90) [AS293 - Energy Sciences Network (ESnet)] 0.771 ms (ttl=250!) 0.821 ms (ttl=250!) 0.788 ms (ttl=250!) 6 paix-pa-snv.es.net (134.55.208.205) [AS293 - Energy Sciences Network (ESnet)] 1.89 ms 1.93 ms 1.94 ms 7 gigabitethernet1-0-112.edge1.paix-sjo1.Level3.net (209.245.146.145) [AS3356 - no more prtraceroute whiners ! Just kidding - we love you Nik.] 2.16 ms 1.60 ms 1.90 ms 8 GigabitEthernet3-1.core1.SanJose1.Level3.net (209.244.3.249) [AS3356 - no more prtraceroute whiners ! Just kidding - we love you Nik.] 3.01 ms 1.99 ms 1.95 ms 9 ae0-55.mp1.SanJose1.Level3.net (64.159.2.129) [AS3356 - no more prtraceroute whiners ! Just kidding - we love you Nik.] 2.67 ms (ttl=246!) 2.78 ms (ttl=246!) 2.70 ms (ttl=246!) 10 64.159.3.254 (64.159.3.254) [AS3356 - no more prtraceroute whiners ! Just kidding - we love you Nik.] 79.2 ms (ttl=245!) 79.3 ms (ttl=245!) 79.2 ms (ttl=245!) 11 so-2-0-0.mp1.London2.Level3.net (212.187.128.137) [AS9057 - Level 3 RIPE block] 152 ms (ttl=244!) 152 ms (ttl=244!) 153 ms (ttl=244!) 12 so-2-0-0.mp1.London1.Level3.net (212.187.128.50) [AS9057 - Level 3 RIPE block] 152 ms (ttl=243!) 152 ms (ttl=243!) 152 ms (ttl=243!) 13 so-7-0-0.gar1.London1.Level3.net (212.113.3.2) [AS9057 - Level 3 RIPE block] 158 ms (ttl=242!) 158 ms (ttl=242!) 158 ms (ttl=242!) 14 pos2-0.metro1-londencyh00.London1.Level3.net (212.113.0.113) [AS9057 - Level 3 RIPE block] 158 ms 158 ms 160 ms 15 195.50.116.30 (195.50.116.30) [AS9057 - Level 3 (ex Businessnet)] 154 ms (ttl=240!) 153 ms (ttl=240!) 153 ms (ttl=240!) 16 fus-rt001-stm1-0-1-0.core.globalconnex.net (80.255.34.17) [AS22351 - Intelsat Specific route within RIPE LIR allocation] 178 ms (ttl=239!) 177 ms (ttl=239!) 176 ms (ttl=239!) 17 fus-rt004-fe-0-0-v2.its-dvb.globalconnex.net (80.255.39.68) [AS22351 - Intelsat Specific route within RIPE LIR allocation] 172 ms 171 ms 171 ms 18 * * * 19 * * * 20 * * * 21 mail2.starcom.co.ug (217.113.72.21) 705.604 ms 699.883 ms 731.260 ms # Loss of 3% for both 100 and 1400 byte packets Many institutes in developing world have less performance than a household in N. America or Europe

Within Developing Regions Not unusual for communications within developing regions to go via developed region, e.g. Rio to Sao Paola goes directly within Brazil But Rio to Buenos Aires goes via Florida And… NIIT – NSC (Rawalpindi – Islamabad) few miles apart, No peering in Pakistan, can this be changed? Route goes via England!!!! Takes longer to go few miles than to SLAC! [cottrell@monitor cottrell]$ traceroute 210.56.13.124 -bash: traceroute: command not found [cottrell@monitor cottrell]$ /usr/sbin/traceroute 210.56.13.124 (pcncp29.ncp.edu.pk) traceroute to 210.56.13.124 (210.56.13.124), 30 hops max, 38 byte packets 1 202.125.153.129 (202.125.153.129) 10.779 ms 10.630 ms 10.797 ms 2 s9-1-3.rwp44d2.pie.net.pk (202.125.149.117) 19.952 ms 17.031 ms 17.627 ms 3 f5-0-0.rwp44c1.pie.net.pk (202.125.148.152) 17.304 ms 17.207 ms 17.179 ms 4 p0-0.khi77c1.pie.net.pk (202.125.159.5) 34.811 ms 26.362 ms 26.414 ms 5 g3-0.khi77gw1.pie.net.pk (202.125.128.162) 26.563 ms 26.463 ms 26.606 ms 6 t2a4-p2-3.uk-lon2.concert.net (166.49.209.5) 160.846 ms 160.800 ms 160.72 6 ms 7 t2c1-ge6-0.uk-lon2.concert.net (166.49.176.11) 161.269 ms 174.196 ms 160. 948 ms 8 t2c2-ge4-0.uk-lon1.concert.net (166.49.208.242) 219.868 ms 243.729 ms 258 .386 ms 9 t2a1-ge6-0.uk-lon1.concert.net (166.49.135.17) 250.670 msIcmp checksum is w rong Icmp checksum is wrong 269.430 msIcmp checksum is wrong 201.403 ms 10 peer1.ldn1.flagtel.com (195.66.224.146) 192.730 ms 161.773 ms 161.311 ms 11 62.216.128.197 (62.216.128.197) 161.681 ms 161.883 ms 161.780 ms 12 ge-1-0-1.0.core1.ldn1.flagtel.com (62.216.128.57) 161.815 ms ge-2-0-1.0.cor e1.ldn1.flagtel.com (62.216.128.61) 161.869 ms ge-1-0-1.0.core1.ldn1.flagtel.co m (62.216.128.57) 188.040 ms 13 t3-0-2-0.0.core1.kar1.flagtel.com (62.216.128.210) 321.569 ms 322.287 ms 339.928 ms 14 62.216.145.154 (62.216.145.154) 345.317 ms 345.215 ms 344.144 ms 15 isbgw2.comsats.net.pk (210.56.8.131) 346.717 ms 345.276 ms 354.303 ms 16 pcncp29.ncp.edu.pk (210.56.13.124) 351.934 ms 349.288 ms 346.037 ms Also within a region can be big differences between sites/countries, due to service providers

Loss to Africa (example of variability)

Technology Achievement Index (TAI) TAI captures how well a country is creating and diffusing technology and building a human skills base. TAI from UNDP hdr.undp.org/reports/global/2001/en/pdf/techindex.pdf TAI top 12 Finland 0.744 US 0.733 Sweden 0.703 Japan 0.698 Korea Rep. of 0.666 Netherlands 0.630 UK 0.606 Canada 0.589 Australia 0.587 Singapore 0.585 Germany 0.583 Norway 0.579 US & Canada off-scale Spreadsheet is V:\SCS\networking\Netdev\iepm\tai-dec-3.xls TAI reflects a country's capacity to participate in the technological innovations of the network age. TAI aims to capture how well a country is creating and diffusing technology and building a human skill base TAI includes the following dimensions: Creation of technology (e.g. patents, royalty receipts); diffusion of recent innovations (Internet hosts/capita, high & medium tech exports as share of all exports); Diffusion of old innovations (log phones/capita, log of electric consumption/capita); Human skills (mean years of schooling, gross enrollment in tertiary level in science, math & engineering)

Summary Performance from U.S. & Europe is improving all over Performance to developed countries are orders of magnitude better than to developing countries Poorer regions 5-10 years behind Poorest regions Africa, Caucasus, Central & S. Asia Some regions are: catching up (SE Europe, Russia), keeping up (Latin America, Mid East, China), falling further behind (e.g. India, Africa)