1. 1 How is the Internet Performing? Les Cottrell – SLAC Lecture # 2 presented at the Workshop on Scientific Information in the Digital Age: Access and Dissemination ICTP, Trieste, Italy October, 2009 www.slac.stanford.edu/grp/scs/net/talk09/ictp-perform.ptt

2. 2 Overview Internet characteristics –Users, capacities, satellites, packet sizes, protocols, routing, flows How is it used apps etc. How the Internet worldwide is performing as seen by various measurements and metrics Application requirements Comparisons with Development Indices

3. USERS 3

4. Internet Usage growth ‘95-’10 4 95 00 05 09 Millions of Users Penetration % 0 200 400 600 Asia Europe N Amer L Amer Africa M East Austrlasia N America Australia Europe L America M East Asia Africa World 0 20% 50% 80% Millions of users Year 1500 1000 500

5. Example: China China not connected to the Internet until May 1994 –1 st permanent IHEP/Beijing used satellite via SLAC –www.computerworld.com.au/article/128099/china_cel ebrates_10_years_being_connected_internetwww.computerworld.com.au/article/128099/china_cel ebrates_10_years_being_connected_internet 5

6. Where are they 6 Internet city connections Internet Users 2002 2.8% growth/year ~¼ world pop uses Internet Developed world saturating Developing catching up 73% penetration US 43% users from Asia

7. Capacities 7

8. What have they got? 8 Capacity From Telegeography Capacity

9. Who is still on Satellite 9 Terrestrial GEOS Min RTT (ms) GEOS (Geostationary Earth Orbit Satellite) good coverage, but expensive in $/Mbps broadband costs 50 times that in US, >800% of monthly salary c.f. 20% in US AND long delays min RTT > 450ms, usually much larger due to congestion Easy to spot Clear signature

10. Packet sizes & types 10

11. 11 Packet size primarily 3 sizes: WHY? Packet size (bytes) Cu,mulative probability % Packets Bytes Mean ~ 420Bytes, median ~ 80Bytes Measured Feb 2000 at Ames Internet eXchange ~ 84M packets, < 0.05% fragmented close to minimum=telnet and ACKs, 1500 (max Ethernet payload, e.g. FTP, HTTP); ~ 560Bytes for TCP implementations not using max transmission unit discovery

12. 12 Internet protocol use There are 3 main protocols in use on the Internet: –UDP (connectionless datagrams, best effort delivery), –TCP (Connection oriented, “guaranteed” delivery in order) –ICMP (Control Message protocol) Time Feb-May 2001 Flows/10min In Out TCP dominates today SLAC protocol flows TCP UDP ICMP

13. Routing 13

14. 14 Hops Hop counts seen from 4 Skitter sites (Japan, S. Cal, N. Cal, E. Canada, i.e. 10-15 hops on average Hop Count Weak RTT dependence on hop count 95% 50% 5% RTT Hops

15. 15 Richness of connectivity Angle = longitude of AS HQ in whois records Radius=1-log(outdegree(AS)+1)/(maxoutdegree + 1) –Outdegree = number of next Hops As’ accepting traffic Deeper blue & red more connections All except 1 of top 15 AS’ are in US, exception in Canada Few links between ISPs in Europe and Asia

16. Today’s routing less via US www.nytimes.com/2008/08/30/business/30pipes.html 16 Invented in US 1 st 30 yrs most traffic thru US 70%=>20% in 10yrs No central control Patriot act=>store info outside US China, India, Japan making larger investments More level playing field Harder for CIA!

17. 17 Routes are not symmetric Min, 50% & 90% RTT measured by Surveyor Notice big differences in RTTs May be due to different paths in the 2 directions or to different loading Advanced to U. Chicago RTT ms U. Chicago to Advanced

18. Flows 18

19. 19 Flow sizes Heavy tailed, in ~ out, UDP flows shorter than TCP, packet~bytes 75% TCP-in < 5kBytes, 75% TCP-out < 1.5kBytes (<10pkts) UDP 80% < 600Bytes (75% < 3 pkts), ~10 * more TCP than UDP Top UDP = AFS (>55%), Real(~25%), SNMP(~1.4%) Can roughly characterize as power law with slope & intercept SNMP Real A/V AFS file server

20. 20 Flow lengths 60% of TCP flows less than 1 second Would expect TCP streams longer lived –But 60% of UDP flows over 10 seconds, maybe due to heavy use of AFS at SLAC –Another (CAIDA) study indicates UDP flows are shorter than TCP flows TCP outbound flows Active time in secs Measured by Netflow flows tied off at 30 mins

21. Applications 21

22. Usage P2p hit by RIAA law suits Moving to video, social networking –Video on demand double/2 years ’08-’13 iPhones (only peripherally a phone) –Mobile traffic doubles each year 22 Yahoo Google Facebook YouTube

23. How it is used & when 23 Enterprise & tier 1 asert.arbornetworks.com/2009/08/the- internet-after-dark

24. 24 Web use characteristics Size of web objects varies from site to site, server to server and by time of day. –Typical medians in 2000 varied from 1500 to 4000 bytes Also varies by object type, e.g. medians for –movies few 100KB to MBs, postscript & audio few 100KB, text, html, applets and images few thousand KB Bytes Size of average web page tripled in 5 years 2003-2008 www.websiteoptimization.com/speed/tweak/av erage-web-page/www.websiteoptimization.com/speed/tweak/av erage-web-page/

25. Why increasing New users (easier for user, more coverage) New apps: You-Tube, climate modeling … New tools: manual(hand tuned)  Automatic generation –Web 2: Ajax, Javascript, CSS Broadband  more elaborate/attractive designs possible 25 desktop to web apps e.g. mail, calendars, photo albums, games...

26. 26 Log Plot of ESnet Monthly Accepted Traffic, January 1990 – December 2008 Impact on backbones: e.g. Current and Historical ESnet Traffic Patterns Terabytes / month Oct 1993 1 TBy/mo. Aug 1990 100 MBy/mo. Jul 1998 10 TBy/mo. 38 months 57 months 40 months Nov 2001 100 TBy/mo. Apr 2006 1 PBy/mo. 53 months  ESnet Traffic Increases by 10X Every 47 Months, on Average July 2010 10 PBy/mo.

27. Performance by Metric 27

28. 28 What does performance depend on? End-to end internet performance seen by applications depends on: –round trip times –packet loss –jitter –reachability –bottleneck bandwidth –implementation/configurations –application requirements Data transmitted in packets

29. msec. ITU G.114 300 ms RTT limit for voice 29 RTT from SLAC to the World RTT ~ distance/(0.6*c) + hops * router delay Router delay = queuing + clocking in & out + processing 2/3 countries of world Ok for voice, rest mainly in Africa What is the problem with > 300ms?

30. 30 RTT from California to world Longitude (degrees) 300ms RTT (ms.) Frequency RTT (ms) Source = Palo Alto CA, W. Coast E. Coast US W. Coast US Europe & S. America Europe 0.3*0.6c Brazil E. Coast Data from CAIDA Skitter project WHY these distributions?

31. Jitter Variability of RTT, many ways to measure “Jitter” = IQR(ipdv); ipdv(i) =RTT(i) – RTT(i-1) Usually at edges, so ~distance independent Impacts smooth flows e.g. VoIP, video, real-time Haptics (surgery) < 1ms; H.323 <40ms with buffer 31 Internet Jitter seen from SLAC to World Sep’08 Can improve voice with de-jitter buffer, e.g. 70ms to smooth the flow But….

32. Losses On good lines usually congestion Wireless dB loss, net devices Usually last mile 32 Distance independent Big effect Realtime, games, Voice, typing echo 1% loss VoIP annoying

33. Derived Throughput 33 Behind Europe 5 Yrs: Russia, Latin America, Mid East 6 Yrs: SE Asia 9 Yrs: South Asia 12 Yrs: Cent. Asia 16 Yrs: Africa Central Asia, and Africa are in Danger of Falling Even Farther behind In 10 years at the current rate Africa will be 1000 times worse than Europe Derived throughput ~ 8 * 1460 /(RTT * sqrt(loss)) Mathis et. al 1993

34. Where is best Throughput? 34

35. Voice over IP Affected by: –Loss, RTT, Jitter, Quality measured by Mean Opinion Score (MOS) 35 –Can convert from RTT, loss & jitter to MOS –MOS values: 1=bad; 2=poor; 3=fair; 4=good; 5=excellent. –Typical reasonable range for Voice over IP (VoIP) is 3.5 to 4.2. –Russia and L.America improved dramatically in 2000-2002 as moved from GEOS to terrestrial. –US, Europe, E. Asia, Russia and the M East (all above MOS = 3.5) good. S.E. Asia marginal, S. Asia need a lot of patience –C. Asia and Africa are pretty much out of the question in general.

36. 36 Application requirements Based on ITU Y1541 & Stanford (Haptics) The VoIP loss of 10^-3 used to be 0.25 but that assumed random flat loss –actual loss is often bursty Tail drop in routers Sync loss in circuits, bridge spanning tree reconfiguration, route changes Applicatio n Real time VoIPWAN connectivity Web free services Stream video Haptics (remote surgery) 1 way delay150ms 1000msundefined400ms160ms ‘jitter”50ms1000msundefined17ms1ms Loss10 -3 undefined10 -5 0.1

37. Compare with Development Indices Abv.NameOrganizationCountries Date of Data GDP Gross Domestic Product per capita CIA 2292001-2006 HDIHuman Development IndexUNDP1752004 DAIDigital Access IndexITU 180 1995-2003 NRINetwork Readiness Index World Economic Forum 1202007 TAI Technology Achievement Index UNDP721995-2000 DOIDigital IndexITU1802004-2005 OIIndexITU1391996-2003 CPICorruption Perception Index Transparency Organization 1802007 37 Choose most: up-to-date, countries, important factors HDI & DOI

38. Human Development Index A long and healthy life, as measured by life expectancy at birth Knowledge, as measured by the adult literacy rate (with two-thirds weight) and the combined primary, secondary and tertiary education gross enrollment ratio (with one-third weight) A decent standard of living, as measured by GDP per capita (or Purchasing Power Parity (PPP) in US$). http://www-iepm.slac.stanford.edu/pinger/pinger- metrics-motion-chart.htmlhttp://www-iepm.slac.stanford.edu/pinger/pinger- metrics-motion-chart.html 38

39. HDI vs Throughput Dot size=population, color =region Bottom left = bad (Africa, blue), top right=good 39 HDI Throughput (kbps) 0.4 0.6 0.8 1 100 1000 10000

40. Digital Opportunity Index DOI tracks infrastructure, opportunity & utilization Strong Correlation, less subjective PingER throuphut Quicker/easier to update cf DOI 40 R 2 = 0.67 Throughput (kbps) DOI 0 0.2 0.4 0.6 0.8

41. What’s next Mobile devices 40G (transAtlantic, US) & 100Gb backbones On demand dynamic dedicated services (layers 1 & 2) –Reserve a path at some bandwidth for some time –Use QoS to deliver –HEP, Radio Astronomy, climate research IPv6 41

42. Questions & more study www.internetworld.stats.comwww.internetworld.stats.com www-iepm.slac.stanford.edu/pingerwww-iepm.slac.stanford.edu/pinger www.slac.stanford.edu/comp/net/wan- mon/tutorial.htmlwww.slac.stanford.edu/comp/net/wan- mon/tutorial.html www.slac.stanford.edu/xorg/icfa/icfa-net-paper- jan09/report-jan09.docwww.slac.stanford.edu/xorg/icfa/icfa-net-paper- jan09/report-jan09.doc Question: why a Dolphin for a PingER Logo 42

43. 43 IP Addresses pingable June 2003 43 From CAIDA, SDSC Grey= not allocated Black= not pingable Companies own class A

44. 44 Growth 2003-2006 44 June 2003 Nov 2006 More areas allocated, Existing areas more colorful

45. 45 Cumulative RTT distributions Gives quality measure Seen from San Diego, US Skitter Steeper = less jitter, i.e. better Small values better RTT ms Cumulative %

46. 46 Throughput also depends on window Optimal window size depends on: –Bandwidth end to end, i.e. min(BW links ) AKA bottleneck bandwidth –Round Trip Time (RTT) –For TCP keep pipe full Window (sometime called pipe) ~ RTT*BW –Can increase bandwidth by orders of magnitude If no loss Throughput ~ Window/RTT Src Rcv ACK t = bits in packet/link speed RTT

47. 47 Loss seen from US to groups of Sites ETSI DTR/TIPHON-05001 V1.2.5 threshold for good speech 50% improvement / year

48. 48 Detailed example of improvements Increase of bandwidth by factor of 460 in 6 years, more than kept pace - factor of 50 times improvement in loss Note valleys when students on vacation

49. 49 Loss to world from US Using year 2000, fraction of world’s population/country from www.nua.ie/surveys/how_many_online/

50. 50 How are the U.S. Nets doing? In general performance is good (i.e. <= 1%) ESnet holding steady, still better than others Edu (vBNS/Abilene) &.com improving

51. 51 Losses for 28 days in May 2001 Measured by MIDS to 583 DNS services, 383 Web services, 1367 Internet (ping) hosts, & 1225 ISPs (routers) DNS WWW Internet ISP % Loss

52. 52 Losses between Regions

53. 53 Bulk throughput Important for long TCP flows where we want to copy large amounts of data from one site to another in a relatively short time, e.g. file transfer Depends on RTT, loss, timeouts, window sizes

54. 54 Throughput quality TCP BW < 1/(RTT*sqrt(loss)) Note E. Europe catching up Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), July 1997

55. 55 “Jitter” from N. America to W. Europe “Jitter” = IQR(ipdv), where ipdv(i) =RTT(i) – RTT(i-1) 214 pairs ETSI: DTR/TIPHON-05001 V1.2.5 (1998-09) good speech < 75ms jitter

56. 56 “Jitter” between regions 75ms=Good 125ms=Med225ms=Poor ETSI: DTR/TIPHON-05001 V1.2.5 (1998-09) Jitter varies with loading

57. 57 SLAC-CERN Jitter ETSI/TIPHON delay jitter threshold (75 ms)

58. 58 Reachability Within N. America, & W. Europe loss, RTT and jitter is acceptable for VoIP But what about reachability

59. 59 Reachability – Outage Probability Surveyor probes randomly 2/second Measure time (Outage length) consecutive probes don’t get through Heavy tailed outage lengths (packet loss not Poisson) http://www-iepm.slac.stanford.edu/monitoring/surveyor/outage.html

60. 60 Europe seen from U.S. 650ms 200 ms 7% loss 10% loss 1% loss Monitor site Beacon site (~10% sites) HENP country Not HENP Not HENP & not monitored

61. 61 Asia seen from U.S. 3.6% loss 10% loss 0.1% loss 640 ms 450 ms 250ms

62. 62 Latin America, Africa & Australasia 4% Loss 2% Loss 350 ms 700ms 170 ms 220 ms

63. 63 Animated monthly 2000 20% loss 200ms RTT 20% unreachable Big is Bad

64. 64 RTT worldwide from the Matrix

65. 65 More Information IEEE Communications, May 2000, Vol 38, No 5, pp 120-159 IEPM/PingER home site –www-iepm.slac.stanford.edu/www-iepm.slac.stanford.edu/ CAIDA/Skitter home site –www.caida.org/home/www.caida.org/home/ Matrix Net home site –www.matrix.net/index.htmlwww.matrix.net/index.html Surveyor home site: –www.advanced.org/csg-ippm/www.advanced.org/csg-ippm/

66. 66 Country dispersion Seen from Japan After 3 to 4 hops most goes to US. –In some cases goes US & back to jp –Some goes to UK & onto other European countries Hops Probes

67. 67 Route maps Simple routes from TRIUMF, Canada to several sites already gets quite complex TRIUMF SLAC KEK UW FNAL DESY CERN

68. 68 Getting more complex PingER Beacon sites in US seen from TRIUMF, Vancouver (from Andrew Daviel, TRIUMF)

69. 69 Connections by country Unknown US UK NL DE IT JP RU

70. 70 Autonomous Systems (AS) Disperson Color indicates the AS responsible for the router at the hop, height is number of probes for that route Seen by Skitter at Palo Alto US (F root name server) Hop number

71. 71 Notes: Many.com are in N. America S. Asia = in (36K), pk (6K), lk, bd E. Asia= jp, cn, my, sg, tw, hk, th, id, bn, mm Mid East= il, kw, lb, ae, tr, sa TLDs with hosts~238 Total TLDs~258 Hosts by regions Jan 2001, 109 Million hosts http://www.internetworldstats.com/stats.htm –Source: Internet Software Consortium (www.isc.org) see web site also for hosts/population

72. 72 Backbone utilization Shows utilization of I2/Abilene backbone links, NB Backbone < 30% loaded Most losses at exchange points & edges

73. 73 Typical Internet traffic by Application CERFnet link Dominated by WWW (http) WWW FTP RealAudio Mail http://www.datacenterknowled ge.com/archives/2007/06/22/y outube-10-percent-of-all- internet-traffic/ http://www.cisco.com/en/US/s olutions/collateral/ns341/ns52 5/ns537/ns705/ns827/white_p aper_c11- 481360_ns827_Networking_S olutions_White_Paper.html

74. 74 SLAC Traffic profile SLAC offsite links: OC3 to ESnet, 1Gbps to Stanford U & thence OC12 to I2 OC48 to NTON Profile bulk-data xfer dominates SSH FTP HTTP Mbps in Mbps out Last 6 months 2 Days bbftp iperf

75. 75 SLAC Internet Application usage Ames IXP: approximately 60-65% was HTTP, about 13% was NNTP Uwisc: 34% HTTP, 24% FTP, 13% Napster

76. 76 RTT Region to Region OK White 0-64ms Green 64-128ms Yellow 128-256ms NOT OK Pink 256-512ms Red > 512ms OK within regions, N. America OK with Europe, Japan

77. 77 Longitude RTT(ms) Seen from Japan RTT from Japan to world