1. 1 Measuring the Digital Divide with PingER Prepared by Les Cottrell, SLAC, for the Round Table: Developing Countries Access to Scientific Knowledge, October 23-24, 2003, ICTP Trieste, Italy www.slac.stanford.edu/grp/scs/net/talk03/ictp-oct03.ppt Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2. 2 Outline How PingER makes the measurements Typical usage & benefits Set the stage for what values mean Application to Digital Divide How’s the world doing today, where is it going? Comparisons with development indices Summary

3. 3 Methodology Use ubiquitous ping Each 30 minutes from monitoring site to target : –1 ping to prime caches –by default send10x100Byte pkts –10x1000Byte pkts Record loss & RTT, (+ reorders, duplicates) Derive throughput, jitter, unreachability …

4. 4 Architecture Hierarchical vs. full mesh WWW Archive Monitoring Remote FNAL Reports & Data Cache Monitoring SLAC Ping HTTP Archive 1 monitor host remote host pair

5. 5 Countries Monitored

6. 6 Recent additions Added remote hosts in Albania, Macedonia, Serbia/Montenegro, Belarus, Turkey, Armenia, Mexico, Cuba, Azerbaijan, Tajikistan, Turkmenistan, Kyrgyzstan, Philippines & Namibia Contacts –Working with contacts in Vietnam, and Tunisia –Working with Iran site to set up monitor host Increased hosts monitored from CERN to give better European view –Now monitoring 60 countries from CERN –~80 from SLAC

7. 7 Countries Monitored Used to monitor Only 1 host Need > 1 host to reduce anomalies Country Ho st sCountry Ho stsCountry Ho st sCountryHosts Albania 1 Estonia 1 Latvia 1 Slovakia 2 Argentina 6 Finland 1 Lithuania 1 Slovenia 1 Armenia 2 France 11 Macedonia 2 S Africa 3 Australia 4 Georgia 2 Malaysia 3 Spain 6 Austria 2 Germany 13 Mexico 5 Sweden 4 Azerbaijan 2 Ghana 1 Moldova 2 Switzerland 8 Bangladesh 1 Greece 1 Mongolia 1 Taiwan 1 Belarus 2 Guatemala 2 Netherlands 12 Tajikstan 1 Belgium 3 Hungary 5 New-Zealand 4 Thailand 0 Brazil 21 Iceland 3 Nigeria 1 Turkey 2 Bulgaria 1 India 10 Norway 2 Turkmenistan 1 Canada 11 Indonesia 0 Pakistan 1 Uganda 1 Chile 4 Iran 4 Peru 1 Ukraine 2 China 6 Ireland 2 Philippines 1 UK 36 Colombia 4 Israel 5 Poland 4 US 208 Costa-Rica 1 Italy 13 Portugal 2 Uruguay 3 Croatia 5 Japan 11 Romania 1 Uzbekistan 2 Cuba 2 Jordan 1 Russia 12 Venezuela 2 Czech-Rep 3 Kazakhstan 2 Saudi Arabia 2 Vietnam 0 Denmark 1 Korea 2 Serbia & Montenegro 2 Egypt 1 Kyrghzstan 1 Singapore 1 ~ 80 countries  480 sites  800 hosts  3600 pairs

8. 8 PingER Benefits Provides quantitative historical (> 8yrs) and near real-time information –Aggregate by regions, affiliations etc. –How bad is performance to various regions, rank countries? –Trends: who is catching up, falling behind, is progress being made? –Compare vs. economic, development indicators etc. Use for trouble shooting setting expectations, identify needed upgrades, choosing a provider, presenting to policy makers, funding bodies Monitoring site vs. Remote sites screen shot Aimed at: end-user (net-admin & sophisticated user), planners Measures analyzes & reports round-trip times, losses, availability, throughput... –Uses ubiquitous ping, no special host, or software to install/configure at remote sites –Low impact on network < 100bits/s, important for many DD sites –Covers ~80 countries (99% of Internet connected population)

9. 9 What do values mean? Loss – most important single metric –< 1% good, (throughput ~ 1/sqrt(loss)) –> 1% voice over IP slightly annoying –>4-6% non-native language speakers unable to communicate, video conferencing becomes irritating, hard to edit files remotely (characters on screen lag keyboarding) Significant degradation of TCP performance –>10-12% TCP sessions time out, FTP fails, mail still works (keeps retrying) RTT (Round Trip Time) –Remote instrument control needs low RTT –>500ms significant interactive voice problems, typing/echo problems

10. 10 Throughput - meaning Derived throughput ~ 1460 Bytes / (RTT * sqrt(loss)) 56 - 64 kbps home modem limit, Internet backbone 1980 128 kbps ISDN Home DSL / cable modems low limits 200-500kbps Internet backbone late 80’s 1.54 Mbps Internet backbone early 90’s 45Mbps Internet backbone today 10Gbits, big sites 155Mbps – 1Gbps Testbeds/cutting edge end to end TCP throughput (Land Speed Record, Guinness Records Book) –2001 1Gbps –2003 2.4Gbps/s (February) => 5.6Gbps (October) Example of the increases in capacity of a well connected site in US

11. 11 Usage Examples Ten-155 became operational on December 11. murf Filters Smurf Filters installed on NORDUnet’s US connection. To North America To Western Europe Peering problems Upgrades & ping filtering

12. 12 Usage Examples Selecting ISPs for DSL/Cable services for home users –Monitor accessibility of routers etc. from site –Long term and changes Trouble shooting –Identifying problem reported is probably network related –Identify when it started and if still happening or fixed –Look for patterns: Step functions Periodic behavior, e.g. due to congestion Multiple sites with simultaneous problems, e.g. common problem link/router … –Provide quantitative information to ISPs Identify need to upgrade and effects BW increase by factor 300 Multiple sites track Xmas & summer holiday

13. 13 Rate Limiting RTTLoss 2 hosts at same site see sudden step- like increase in loss from < 1% to 20- 30% at similar time boromir.nask.waw.pl gollum.nask.pl www.pol34.pl Another host in Poland sees no problems, i.e. helps to have another nearby host RTT Loss Similar effects for Greek (uoa.gr), Bulgarian (acad.bg), Kazakhstan (president.kz), Moldovan (asm.md) and Turkish (metud.edu.tr) sites If no step function or nearby host may not notice, so also compare synack vs ping Can ping routers along path to see where onset occurs At any given time, about 5% of monitored hosts are doing this, most in developing countries. Recently (August 2003) seen an increase in ping rate limiting RTTLoss boromir.nask.waw.pl gollum.nask.waw.pl www.pol34.pl RTT Loss

14. 14 Digital Divide Regions Design regions –to match well known world regions and –to have similar connectivity within region Then order by derived throughput –Derived throughput ~ MSS [=1460Bytes] / (RTT*sqrt(loss)) Want to show general behavior & variability (outliers) Developed: –U. S.+Canada, Japan+Taiwan+Singapore+Korea, Australia+NZ, Europe (excl. SE Europe, Russia) Developing (Digital Divide): –Africa, S. America, C. America, C. Asia, China, S. Asia, Caucasus, M. East, SE Europe, Russia Israel has much better connectivity than neighbors in Mid East so distorts Mid East results, move to Europe?! Greece is part of Europe, should it be part of S. E. Europe, choice varies with time…

15. 15 Region Map Also have affinity groups, e.g. AMPATH, Silk Road, CMS, XIWT and can select multiple groups

16. 16 Current State – Aug ‘03 (throughput) Within region performance better –E.g. Ca|EDU|GOV-NA, Hu-SE Eu, Eu-Eu, Jp-E Asia, Au-Au, Ru- Ru|Baltics Africa, Caucasus, Central & S. Asia all bad Bad < 200kbits/s < DSL Poor > 200 < 500kbits/s Acceptable > 500kbits/s, < 1000kbits/s Good > 1000kbits/s

17. 17 RegionCountries# Africa Ghana, Namibia, Nigeria, Namibia, S. Africa, Uganda 6 C Asia Kazakhstan, Kyrghzstan, Mongolia, Tajikistan, Turkmenistan, Uzbekistan 9 S Asia Bangladesh, India, Indonesia, Malaysia, Pakistan, Thailand, (Vietnam) 16 M East Egypt, Iran, Israel, Jordan, Saudi Arabia, Turkey 10 Caucasus Armenia, Azerbaijan, Georgia 5 S America Argentina, Brazil, Columbia, Chile, Peru, Uruguay, Venezuela 13 China China including Hong Kong 5 Russia 5 C America Costa Rica, Cuba, Guatemala, Mexico 4 SE Europe (Albania), Bulgaria, Croatia, Macedonia, Moldova, Romania, Serbia/Montenegro, Slovenia 13

18. 18 Trends Africa shown for only Uganda seen from SLAC, since adding new countries with very different throughputs distorts result S.E. Europe, Russia: catching up Latin Am., Mid East, China: keeping up India, Africa: falling behind Derived throughput~MSS/(RTT*sqrt(loss))

19. 19 Russia E.g. Upgrade to KEK-BINP link from 128kbps to 512kbps, May ’02: improved from few % loss to ~0.1% loss Russian losses improved by factor 5 in last 2 years, due to multiple upgrades

20. 20 Loss Comparisons with Development (UNDP) Positive correlation with Human Development or GDP

21. 21 Derived throughput~MSS [=1460Bytes] / (RTT*sqrt(loss)) Europe Netherlands Turkey Belgium

22. 22 Network Readiness Index: How Ready to Use Modern ICTs [*]? Market Infrastructure Political/Regulatory Individual Readiness Gov’t Readiness Business Readiness Individual Usage Gov’t Usage Business Usage (FI) (SG) (US) (IC) (SG) (US) (SG) (US) (FI) (DE ) (KR) ( ): Which Country is First From the 2002-2003 Global Information Technology Report. See http://www.weforum.org Network Readiness Index Environment Readiness Usage Slide prepared by Harvey Newman, Caltech for ICFA

23. 23 Network Readiness NRI from Center for International Development, Harvard U. http://www.cid.harvard.edu/cr/pdf/gitrr2002_ch02.pdf http://www.cid.harvard.edu/cr/pdf/gitrr2002_ch02.pdf Using derived throughput ~ MSS[=1460B] / (RTT * sqrt(loss)) –Fit to exponential is better Internet for all focus A&R focus NRI Top 14 Finland 5.92 US 5.79 Singapore 5.74 Sweden 5.58 Iceland 5.51 Canada 5.44 UK5.35 Denmark 5.33 Taiwan5.31 Germany5.29 Netherlands 5.28 Israel 5.22 Switzerland 5.18 Korea 5.10

24. 24 Challenges Effort: –Usually negligible for remote hosts –Monitoring host: < 1 day to install and configure, occasional updates to remote host tables and problem response –Archive host: 20% FTE, code stable, could do with upgrade, contact monitoring sites whose data is inaccessible –Analysis: your decision, usually for long term details download & use Excel –Trouble-shooting: usually re-active, user reports, then look at PingER data have played with automating alerts, data will/is available via web services Ping blocking –Complete block easy to ID, then contact site to try and by-pass, can be frustrating –Partial blocks trickier, compare with synack or TCP/ping Derived throughputs poor for well connected sites (<0.1% loss) Funding to sustain efforts

25. 25 35+ monitoring sites in 15 countries –Plan to add ICTP Trieste if funded –Other projects used toolkit, e.g. XIWT, PPCNG/EDG … SLAC with help from FNAL Digital Divide collaboration (MOU) with ICTP, Trieste –eJDS –looking for a EU grant for eJDS and PingER Need funding for coming year: –Working with DoE, NSF, Pew Charitable Foundation … –Tasks: (0.5 FTE) ongoing maintain data collection, explain needs, reopen connections, open firewall blocks, find replacement hosts, make limited special analyses, prepare & make presentations, respond to questions (+ 0.5 FTE) extend the code for new environment (more countries, more data collections), fix known non-critical bugs, improve visualization, automate some of reports generated by hand today, find new country site contacts, add route histories and visualization, automate alarms, detect rate limiting earlier, update web site for better navigation, add more DD monitoring sites/countries, improve code portability, understand regions better Also looking for small grants for helpers in developing countries ICFA: show importance to policy makers, funding agencies, identify sympathetic contacts at agencies, get support Collaborations & Funding

26. 26 Futures More work on understanding regions Better/quicker detection of rate limiting Extend deployment, in particular Africa: Demonstration in coordination with WSIS in Geneva Dec 2003

27. 27 Summary Valuable light-weight tool for end-to-end performance Good for trouble-shooting, planning, setting expectations World wide coverage Performance from U.S. is improving all over Performance to developed countries are orders of magnitude better than to developing countries Poorer regions 5-10 years behind Poorest regions Africa, Caucasus, Central & S. Asia Some regions are: –catching up (SE Europe, Russia), –keeping up (Latin America, Mid East, China), –falling further behind (e.g. India, Africa)

28. 28 More Information PingER: –www-iepm.slac.stanford.edu/pinger/www-iepm.slac.stanford.edu/pinger/ MonaLisa –monalisa.cacr.caltech.edu/monalisa.cacr.caltech.edu/ GGF/NMWG –www-didc.lbl.gov/NMWG/www-didc.lbl.gov/NMWG/ ICFA/SCIC Network Monitoring report, Jan03 –www.slac.stanford.edu/xorg/icfa/icfa-net-paper-dec02www.slac.stanford.edu/xorg/icfa/icfa-net-paper-dec02 Monitoring the Digital Divide, CHEP03 paper –arxiv.org/ftp/physics/papers/0305/0305016.pdfarxiv.org/ftp/physics/papers/0305/0305016.pdf Human Development Index –www.undp.org/hdr2003/pdf/hdr03_backmatter_2.pdfwww.undp.org/hdr2003/pdf/hdr03_backmatter_2.pdf Network Readiness Index –www.weforum.org/site/homepublic.nsf/Content/Initiatives+subhomewww.weforum.org/site/homepublic.nsf/Content/Initiatives+subhome

29. 29 Africa – Getting Started Recommendation on Monitoring from Open Round Table, Trieste 2002 –Devote resources to monitor in real time the connectivity of research & educational institutions in developing countries MOU signed between SLAC & eJDS/ICTP –Use & extend SLAC PingER/project for monitoring eJDS “network” of participants Find contacts at potential sites –Dec ’02 send emails to eJDS participants, ICFA/SCIC representatives Explain value to participants, needs for hosts to monitor, contacts –Contact contacts, explain again When get host, check it is pingable Could be multi-month process ~75% contacts provided successful hosts –25% pings blocked and unable to resolve, emails exchange petered out Still need contacts for many countries

30. 30 Africa Oct ‘03 Hosts in: Ife-Ife/Nigeria, Accra/Ghana, Kampala/Uganda, Windhoek/Namibia, UCT/ZA, Johannesburg/ZA, Mosselbay/ZA Carriers: –GH uses UUNET/Satworks, NA uses UUNET/xantic, NG uses TELIANET/NewSkies, UG uses Level(3)/globalconnex –ZA varies from site to site: UUNET/ALTERNET, C&W Telecom S. Africa, CAIS telcom S. Africa UG, NA, NG, GH use satellites (> 600ms) ZA uses landlines

31. 31 Africa RTT Monitored from N. America & Europe –Depends on remote site (not monitoring site) –Satellite for all except S. Africa –Ghana problems

32. 32 West Africa Loss Ghana very poor performance –Sudden increase in losses on August 18th –Not rate limiting according to synack –Sometimes get down to a few % –Route ESnet-UUNET/ALTER.NET –Losses appear on last 2 hops in Ghana Nigeria better –Route via TELIANET/newskies

33. 33 Africa Derived Throughput S. Africa (UCT) best, followed by Uganda, Nigeria and Ghana Throughput to Nigeria site == home DSL/cable Throughput to Ghana site == modem dialup Uganda site == SLAC late 1980s Derived throughput ~ MSS[=1460Bytes] RTT * sqrt(loss)

34. 34 Summary 1/2 Beware limited number of countries monitored and even then limited number of sites in each country. Need more contacts: –Countries with no sites: e.g. Algeria, Cameroon, Kenya, Malawi, Mozambique, Senegal, Tanzania, Tunisia, Zambia –Countries with only one (possibly anomalous) site: Uganda, Ghana, Nigeria, Namibia Africa poorest performance region –Factor 30 behind Europe in performance today –ZA and Uganda better performers –“Best” Uganda site, 8 years behind Europe –Ghana bad, Nigeria poor, similar to connectivity to homes in Developed nations Overall Africa not catching up Ghana, Nigeria falling behind

35. 35 Summary 2/2 Little uniformity in routes, many carriers –Unlike S. America & Caucasus where AMPATH & Virtual Silk Highway have improved performance Hopefully Africa ONE project will help –Undersea fiber to link countries of Africa together and to one another

36. 36 Extra Slides

37. 37 Visualization Keep it simple, enable user to do their own by making data available Tables –Time series ( www-iepm.slac.stanford.edu/cgi- wrap/pingtable.pl ): www-iepm.slac.stanford.edu/cgi- wrap/pingtable.pl select metric (loss, RTT etc.), time ticks, packet size, aggregations from/to, etc. Color code numbers, provide sort, drill down to graphs, download data (TSV), statistical summaries –Monitoring site vs. Remote sites ( www- iepm.slac.stanford.edu/cgi-wrap/table.pl ): www- iepm.slac.stanford.edu/cgi-wrap/table.pl Select metric, region aggregations Drill down to time series, download data Graphs –Select source(s)/destination(s), metric, time window, SQL selects, graph type

38. 38 Publish information #!/usr/bin/perl use SOAP::Lite; my $characteristic = SOAP::Lite -> service(‘http://www-iepm.slac.stanford.edu/tools/soap/wsdl/profile_06.wsdl') -> pathDelayOneWay("tt81.ripe.net:tt28.ripe.net”); print $characteristic->{NetworkTestTool}->{toolName},"\n"; print $characteristic->{NetworkPathDelayStatistics}->{value},"\n"; www.slac.stanford.edu/cgi- wrap/pingtable.pl => tabular reports, also download datawww.slac.stanford.edu/cgi- wrap/pingtable.pl Data accessible from MonaLisa Implementing web services access prototype –Includes: PingER, IEPM-BE, RIPE-tt, I2 E2Epi OWAMP –Use GGF/NMWG schema/profile, e.g. path.delay.roundTrip

39. 39 Rate Limiting Moldova cni.md lises.asm.md RTTLoss Moldova Bulgaria

40. 40

41. 41 Loss Comparisons with Development (UNDP) Weaker with education & literacy Positive correlation with Human Development or GDP

42. 42 NREN Core Network Size (Mbps-km) 10M 1M 100K 10K 1K 100 2000 2001 Leading Advanced In transition Source: From slide prepared by Harvey Newman, presented by David Williams ICFA/SCIC talk on Serenate report. Data from the TERENA Compendium Lagging Derived throughput~MSS/(RTT*sqrt(loss)) Europe Netherlands Turkey Belgium