1. End to end Internet Performance today
Prepared by Les Cottrell, SLAC, for the
Extending the Reach of Advanced Networking: Special International Workshop
Arlington, VA., April 22, 2004
PingER is arguably the most extensive set of measurements of the
end-to-end performance of the Internet going back almost ten years.
Measurements are available from over 30 sites in 13 countries to
sites in over 100 countries. We will use the PingER results
to: demonstrate how the Internet
performance to the regions of the world has evolved over the last 9
years; identify regions that have poor connectivity, how far
they are behind the developed world and whether they are catching up
or falling further behind; and illustrate the correlation
between the UN Technology Achievement Index and Internet performance.
Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2. Countries Monitored
Monitor site
Remote site
Monitoring hosts
35 hosts
13 Countries
Countries monitored contain over 90% of the world’s Internet connected population
Need >1 host/country to reduce anomalies.
Remote hosts
>105 countries
560 sites
880 hosts
3650pairs

3. Loss to world from US Loss Rate < 0.1 to 1 % 1 to 2.5 % 2.5 to 5 %
> 12 %
2001
Dec-2003
In 2001 <20% of the world’s population had Good or Acceptable Loss performance
BUT by December 2003 It had improved to 77%
Spreadsheet: v:Groups\scs\networking\netdev\tai-dec03.xls

4. Trends
C. Asia, Russia, S.E. Europe, L. America, M. East, China: 4-5 yrs behind
India, Africa: 7 yrs behind
S.E. Europe, Russia: catching up
Latin Am., Mid East, China: keeping up
India, Africa: falling behind
Spreadsheet \cottrell\iepm\esnet-to-all-longterm.xls
CERN data only goes back to Aug-01. It confirms S.E. Europe & Russia are catching up, and India & Africa are falling behind
Note for Africa only one host in Uganda. Actually have been adding hosts 5 countries), but there is considerable disparity in performance so as add hosts from less developed countries the aggregate performance measured to Africa is dropping!
Ghana, Nigeria and Uganda are all satellite links with ms RTTs. The losses to Ghana & Nigeria are 8-12% while to Uganda they are 1-3%. The routes are different. The route from SLAC to Ghana uses ESnet-Worldcom-UUNET, Nigeria goes CalREN-Qwest-Teiianet-New Skies satellite, Uganda goes Esnet-Level3-Intelsat. For both Ghana and Nigeria there are no losses (for 100 pings) until the last hop when over 40 of 100 packets were lost. For Uganda the losses (3 in 100 packets) also occur at the last hop.
Worksheet: for trends: \\Zwinsan2\c\cottrell\iepm\esnet-to-all-longterm.xls
for Africa: \\Zwinsan2\c\cottrell\iepm\africa.xls
traceroute to ( ): 1-30 hops, 38 byte packets
1 rtr-core1-nethub.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
2 rtr-dmz1-ger.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
( ) ms (ttl=252!) ms (ttl=252!) ms (ttl=252!)
4 snv-pos-slac.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms (ttl=251!) ms (ttl=251!) ms (ttl=251!)
5 snvrt1-ge0-snvcr1.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms (ttl=250!) ms (ttl=250!) ms (ttl=250!)
ATM1-0.BR2.SJC1.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms ms ms
ATM3-0.XR1.SJC1.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms ms ms
8 0.so XL1.SJC1.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms (ttl=247!) ms (ttl=247!) ms (ttl=247!)
9 0.so TL1.SAC1.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms (ttl=246!) ms (ttl=246!) ms (ttl=246!)
10 0.so IL1.NYC9.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms (ttl=242!) ms (ttl=242!) 74.4ms (ttl=242!)
11 0.so IR1.NYC12.ALTER.NET ( ) [AS701 - UUNET, An MCI Worldcom Company] ms (ttl=241!) ms (ttl=241!) ms (ttl=241!)
12 so TR1.CPH3.ALTER.NET ( ) [AS702 - UUNET, An MCI Worldcom Company] 171 ms (ttl=241!) 171 ms (ttl=241!) 171 ms (ttl=241!)
13 POS5-0.XR1.CPH3.ALTER.NET ( ) [AS702 - UUNET, An MCI Worldcom Company] 172 ms (ttl=241!) 173 ms (ttl=241!) 171 ms (ttl=241!)
14 POS4-0-0.CR1.CPH2.ALTER.NET ( ) [AS702 - UUNET, An MCI Worldcom Company] 171 ms (ttl=240!) 172 ms (ttl=240!) 172 ms (ttl=240!)
15 FastEthernet GW1.CPH2.ALTER.NET ( ) [AS702 - UUNET, An MCI Worldcom Company] 212 ms (ttl=239!) 347 ms (ttl=239!) 408 ms (ttl=239!)
16 satworks.gw.dk.uu.net ( ) [AS702 - UUNET DK Block 4] 226 ms (ttl=238!) 163 ms (ttl=238!) 163 ms (ttl=238!)
( ) [AS702 - UUNET DK Block 4] * 907 ms 865 ms# 43% loss on 100 pings (0 losses until this hop)
asoju.oauife.edu.ng
traceroute to asoju.oauife.edu.ng ( ): 1-30 hops, 38 byte packets
1 rtr-core1-nethub.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
2 rtr-dmz1-ger.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
3 i2-gateway.stanford.edu ( ) ms ms ms
4 STAN.POS.calren2.NET ( ) [AS32 - BN-CIDR ] ms ms ms
5 SUNV--STAN.POS.calren2.net ( ) [AS NET-C2-NORTH] ms ms ms
6 QSV-M10-2-C2.GE.calren2.net ( ) [AS CENIC-DCP] ms (ttl=249!) ms (ttl=249!) ms (ttl=249!)
( ) [AS209 - Qwest Communications] ms (ttl=247!) ms (ttl=247!) ms (ttl=247!)
( ) [AS209 - Qwest Communications] ms ms ms
( ) [AS209 - Qwest Communications] ms (ttl=248!) ms (ttl=248!) ms (ttl=248!)
( ) [AS209 - Qwest Communications] ms (ttl=245!) ms (ttl=245!) ms (ttl=245!)
11 sca-bb1-pos0-0-0.telia.net ( ) [AS TELIANET-BLK] ms (ttl=243!) ms (ttl=243!) ms (ttl=243!)
12 chi-bb1-pos1-0-0.telia.net ( ) [AS TELIANET-BLK] ms (ttl=242!) ms (ttl=242!) ms (ttl=242!)
13 nyk-bb1-pos0-1-0.telia.net ( ) [AS TELIANET-BLK] ms (ttl=238!) 76.1 ms (ttl=238!) ms (ttl=238!)
14 nyk-bb2-pos1-0-0.telia.net ( ) [AS TELIANET-BLK] ms (ttl=239!) ms (ttl=239!) ms (ttl=239!)
15 ldn-bb2-pos1-3-0.telia.net ( ) [AS TELIANET-BLK] 148 ms (ttl=237!) 148 ms (ttl=237!) 147 ms (ttl=237!)
16 ldn-b1-pos11-0.telia.net ( ) [AS TELIANET-BLK] 147 ms (ttl=236!) 148 ms (ttl=237!) 147 ms (ttl=236!)
17 ldn-th-i1-srp1-0.telia.net ( ) [AS TELIANET-BLK] 147 ms (ttl=234!) 147 ms (ttl=234!) 148 ms (ttl=234!)
18 new-skies ldn-th-i1.c.telia.net ( ) [AS TELIANET-BLK] 141 ms (ttl=242!) 141 ms (ttl=242!) 141 ms (ttl=242!)
19 rtr-cor01-pos6-0-0.cha.newskies.net ( ) [AS New Skies Satellites]142 ms (ttl=241!) 142 ms (ttl=241!) 142 ms (ttl=241!)
20 rtr-dvb01-gi cha.newskies.net ( ) [AS New Skies Satellites] 142 ms (ttl=240!) 142 ms (ttl=240!) 143 ms (ttl=240!)
21 * * *
reverse.newskies.net ( ) [AS701 - UUNET - AS 701] ms (ttl=238!) 970 ms (ttl=238!) 988 ms (ttl=238!)# 44% loss on 100 pings for this hop, 0 for others
mail2.starcom.co.ug
traceroute to mail2.starcom.co.ug ( ): 1-30 hops, 38 byte packets
1 rtr-core1-nethub.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
2 rtr-dmz1-ger.slac.stanford.edu ( ) [AS SU-SLAC] ms ms ms
( ) ms (ttl=252!) ms (ttl=252!) ms (ttl=252!)
4 snv-pos-slac.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms (ttl=251!) ms (ttl=251!) ms (ttl=251!)
5 snvrt1-ge0-snvcr1.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms (ttl=250!) ms (ttl=250!) ms (ttl=250!)
6 paix-pa-snv.es.net ( ) [AS293 - Energy Sciences Network (ESnet)] ms ms ms
7 gigabitethernet edge1.paix-sjo1.Level3.net ( ) [AS no more prtraceroute whiners ! Just kidding - we love you Nik.] ms ms ms
8 GigabitEthernet3-1.core1.SanJose1.Level3.net ( ) [AS no more prtraceroute whiners ! Just kidding - we love you Nik.] ms ms ms
9 ae0-55.mp1.SanJose1.Level3.net ( ) [AS no more prtraceroute whiners ! Just kidding - we love you Nik.] ms (ttl=246!) ms (ttl=246!) ms (ttl=246!)
( ) [AS no more prtraceroute whiners ! Just kidding - we love you Nik.] ms (ttl=245!) ms (ttl=245!) ms (ttl=245!)
11 so mp1.London2.Level3.net ( ) [AS Level 3 RIPE block] 152 ms (ttl=244!) 152 ms (ttl=244!) 153 ms (ttl=244!)
12 so mp1.London1.Level3.net ( ) [AS Level 3 RIPE block] 152 ms (ttl=243!) 152 ms (ttl=243!) 152 ms (ttl=243!)
13 so gar1.London1.Level3.net ( ) [AS Level 3 RIPE block] 158 ms (ttl=242!) 158 ms (ttl=242!) 158 ms (ttl=242!)
14 pos2-0.metro1-londencyh00.London1.Level3.net ( ) [AS Level 3 RIPE block] 158 ms 158 ms 160 ms
( ) [AS Level 3 (ex Businessnet)] 154 ms (ttl=240!) 153 ms (ttl=240!) 153 ms (ttl=240!)
16 fus-rt001-stm core.globalconnex.net ( ) [AS Intelsat Specific route within RIPE LIR allocation] 178 ms (ttl=239!) 177 ms (ttl=239!) 176 ms (ttl=239!)
17 fus-rt004-fe-0-0-v2.its-dvb.globalconnex.net ( ) [AS Intelsat Specific route within RIPE LIR allocation] 172 ms 171 ms 171 ms
18 * * *
19 * * *
20 * * *
21 mail2.starcom.co.ug ( ) ms ms ms # Loss of 3% for both 100 and 1400 byte packets
Many institutes in developing world have less performance than a household in N. America or Europe

5. Within Developing Regions
Not unusual for communications within developing regions to go via developed region, e.g.
Rio to Sao Paola goes directly within Brazil
But Rio to Buenos Aires goes via Florida
And…
NIIT – NSC (Rawalpindi – Islamabad) few miles apart,
No peering in Pakistan, can this be changed?
Route goes via England!!!!
Takes longer to go few miles than to SLAC!
cottrell]$ traceroute
-bash: traceroute: command not found
cottrell]$ /usr/sbin/traceroute (pcncp29.ncp.edu.pk)
traceroute to ( ), 30 hops max, 38 byte packets
( ) ms ms ms
2 s9-1-3.rwp44d2.pie.net.pk ( ) ms ms ms
3 f5-0-0.rwp44c1.pie.net.pk ( ) ms ms ms
4 p0-0.khi77c1.pie.net.pk ( ) ms ms ms
5 g3-0.khi77gw1.pie.net.pk ( ) ms ms ms
6 t2a4-p2-3.uk-lon2.concert.net ( ) ms ms
6 ms
7 t2c1-ge6-0.uk-lon2.concert.net ( ) ms ms 160.
948 ms
8 t2c2-ge4-0.uk-lon1.concert.net ( ) ms ms 258
.386 ms
9 t2a1-ge6-0.uk-lon1.concert.net ( ) msIcmp checksum is w
rong
Icmp checksum is wrong
msIcmp checksum is wrong ms
10 peer1.ldn1.flagtel.com ( ) ms ms ms
( ) ms ms ms
12 ge core1.ldn1.flagtel.com ( ) ms ge cor
e1.ldn1.flagtel.com ( ) ms ge core1.ldn1.flagtel.co
m ( ) ms
13 t core1.kar1.flagtel.com ( ) ms ms ms
( ) ms ms ms
15 isbgw2.comsats.net.pk ( ) ms ms ms
16 pcncp29.ncp.edu.pk ( ) ms ms ms
Also within a region can be big differences between sites/countries, due to service providers

6. Loss to Africa (example of variability)

7. Technology Achievement Index (TAI)
TAI captures how well a country is creating and diffusing technology and building a human skills base.
TAI from UNDP hdr.undp.org/reports/global/2001/en/pdf/techindex.pdf
TAI top 12
Finland US
Sweden
Japan
Korea Rep. of
Netherlands 0.630 UK
Canada
Australia
Singapore
Germany
Norway
US & Canada off-scale
Spreadsheet is V:\SCS\networking\Netdev\iepm\tai-dec-3.xls
TAI reflects a country's capacity to participate in the technological innovations of the network age.
TAI aims to capture how well a country is creating and diffusing technology and building a human skill base
TAI includes the following dimensions: Creation of technology (e.g. patents, royalty receipts); diffusion of recent innovations (Internet hosts/capita, high & medium tech exports as share of all exports); Diffusion of old innovations (log phones/capita, log of electric consumption/capita); Human skills (mean years of schooling, gross enrollment in tertiary level in science, math & engineering)

8. Summary Performance from U.S. & Europe is improving all over
Performance to developed countries are orders of magnitude better than to developing countries
Poorer regions 5-10 years behind
Poorest regions Africa, Caucasus, Central & S. Asia
Some regions are:
catching up (SE Europe, Russia),
keeping up (Latin America, Mid East, China),
falling further behind (e.g. India, Africa)