1. Part III: Measuring Inter- domain Paths

2. March 8, 20042 Packet forwarding path Internet Source Destination IP traffic Forwarding path - the path packets traverse through the Internet from a source to a destination

3. March 8, 20043 An inter-domain level view Internet Source Destination AS A AS B AS C AS D IP traffic An IP forwarding path often span across multiple Autonomous Systems.

4. March 8, 20044 Why do we care?  Characterize end-to-end network paths  Diagnose routing anomalies  Discover Internet topology

5. March 8, 20045 Why do we care?  Characterize end-to-end network paths  Latency  Capacity  Link utilization  Loss rate.  Diagnose routing anomalies  Discover Internet topology

6. March 8, 20046 Varies link capacity Internet Source Destination

7. March 8, 20047 Different loss rate Internet Source Destination

8. March 8, 20048 Traffic engineering Internet Source Destination Customer service enhancement

9. March 8, 20049 Why do we care?  Characterize end-to-end network paths  Diagnose routing anomalies  Forwarding loop, black holes, routing changes, unexpected paths, main component of end-to-end latency.  Discover Internet topology

10. March 8, 200410 Forwarding loops Internet Source Destination

11. March 8, 200411 Black holes Internet Source Destination

12. March 8, 200412 Routing changes Internet Source Destination

13. March 8, 200413 Unexpected routes Internet Source Destination

14. March 8, 200414 Performance bottleneck Internet Source Destination

15. March 8, 200415 Why do we care?  Characterize end-to-end network paths  Diagnose routing anomalies  Discover Internet topology  Server placement

16. March 8, 200416 Internet topology Internet Client Server Client

17. March 8, 200417 Server placement Internet Client Server Client Proxy

18. March 8, 200418 Key challenge  Need to understand how packets flow through the Internet without real-time access to proprietary routing data from each domain.  Identify accurate packet forwarding paths  Characterize the performance metrics of each hop along the paths

19. March 8, 200419 Identify forwarding path  Traceroute gives IP level forwarding path  IP address of the router interfaces on a forwarding path  RTT statistics for each hop along the way

20. March 8, 200420 Traceroute from UC Berkeley to www.cnn.com 1 169.229.62.1 2 169.229.59.225 3 128.32.255.169 4 128.32.0.249 5 128.32.0.66 6 209.247.159.109 7 * 8 64.159.1.46 9 209.247.9.170 10 66.185.138.33 11 * 12 66.185.136.17 13 64.236.16.52 inr-daedalus-0.CS.Berkeley.EDU soda-cr-1-1-soda-br-6-2 vlan242.inr-202-doecev.Berkeley.EDU gigE6-0-0.inr-666-doecev.Berkeley.EDU qsv-juniper--ucb-gw.calren2.net POS1-0.hsipaccess1.SanJose1.Level3.net ? pos8-0.hsa2.Atlanta2.Level3.net pop2-atm-P0-2.atdn.net ? pop1-atl-P4-0.atdn.net www4.cnn.com Traceroute output: (hop number, IP address, DNS name) 1 169.229.62.1 2 169.229.59.225 3 128.32.255.169 4 128.32.0.249 5 128.32.0.66 6 209.247.159.109 7 * 8 64.159.1.46 9 209.247.9.170 10 66.185.138.33 11 * 12 66.185.136.17 13 64.236.16.52 inr-daedalus-0.CS.Berkeley.EDU soda-cr-1-1-soda-br-6-2 vlan242.inr-202-doecev.Berkeley.EDU gigE6-0-0.inr-666-doecev.Berkeley.EDU qsv-juniper--ucb-gw.calren2.net POS1-0.hsipaccess1.SanJose1.Level3.net ? pos8-0.hsa2.Atlanta2.Level3.net pop2-atm-P0-2.atdn.net ? pop1-atl-P4-0.atdn.net www4.cnn.com

21. March 8, 200421 Traceroute from AT&T Research to www.cnn.com traceroute to cnn.com (64.236.24.12), 30 hops max, 40 byte packets 1 oden (135.207.16.1) 1 ms 1 ms 1 ms 2 * * * 3 attlr-gate (192.20.225.1) 2 ms 2 ms 2 ms 4 12.119.155.157 (12.119.155.157) 3 ms 4 ms 4 ms 5 gbr6-p52.n54ny.ip.att.net (12.123.192.18) 4 ms 4 ms 4 ms 6 tbr2-p012401.n54ny.ip.att.net (12.122.11.29) 4 ms (ttl=249!) 5 ms (ttl=249!) 5 ms (ttl=249!) 7 ggr2-p390.n54ny.ip.att.net (12.123.3.62) 4 ms 5 ms 4 ms 8 att-gw.ny.aol.net (192.205.32.218) 4 ms 4 ms 4 ms 9 bb2-nye-P1-0.atdn.net (66.185.151.66) 4 ms 4 ms 4 ms 10 bb2-vie-P8-0.atdn.net (66.185.152.201) 13 ms (ttl=245!) 12 ms (ttl=245!) 12 ms (ttl=245!) 11 bb1-vie-P11-0.atdn.net (66.185.152.206) 10 ms 10 ms 10 ms 12 bb1-cha-P7-0.atdn.net (66.185.152.28) 20 ms 20 ms 20 ms 13 bb1-atm-P6-0.atdn.net (66.185.152.182) 25 ms 25 ms 25 ms 14 pop1-atl-P4-0.atdn.net (66.185.136.17) 25 ms (ttl=243!) 24 ms (ttl=243!) 24 ms (ttl=243!) 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Who is responsible for the forwarding problem? Destination unreachable!

22. March 8, 200422 Need to know Inter-domain level path Internet AT&T Research www.cnn.com AS A AS B AS C AS D Routing loop in AS C!

23. March 8, 200423 How to obtain AS level paths  BGP AS path  Traceroute AS path

24. March 8, 200424 BGP AS path AS A AS B AS C Prefix d Forwarding path: data traffic Signaling path: control traffic d: path=[C] d: path=[BC] PrefixAS path dA B C… Is BGP AS path the answer?No!

25. March 8, 200425 BGP AS path is not the answer  Requires timely access to BGP data  Signaling path may differ from forwarding path  Route aggregation and filtering  Routing anomalies: e.g., deflections, loops [Griffin2002]  BGP misconfigurations: e.g., incorrect AS prepending Two paths may differ precisely when operators most need accurate data to diagnose a problem!

26. March 8, 200426 AS AAS BAS CAS D Traceroute AS path  Obtain IP level path using traceroute  Map IP addresses to ASes Is traceroute AS path the answer?NO! SourceDestination a bcde

27. March 8, 200427 Example: UC Berkeley to CNN 1 169.229.62.1 2 169.229.59.225 3 128.32.255.169 4 128.32.0.249 5 128.32.0.66 6 209.247.159.109 7 * 8 64.159.1.46 9 209.247.9.170 10 66.185.138.33 11 * 12 66.185.136.17 13 64.236.16.52 Traceroute output: (hop number, IP) AS25 AS11423 AS3356 AS1668 AS5662 Berkeley CNN Calren Level3 GNN

28. March 8, 200428 Traceroute AS path is not the answer  Identifying ASes along forwarding path is surprisingly difficult!  Internet route registry  Origin AS in BGP routes

29. March 8, 200429 Internet route registry  Whois database  E.g. NANOG traceroute, prtraceroute  Out-of-date, incomplete  Address allocation to customers  Acquisition, mergers, break-ups

30. March 8, 200430 Origin AS in BGP routes  Last AS in the AS path for each prefix  More accurate and complete than whois data PrefixAS path dA B C ……

31. March 8, 200431 Limitations of BGP origin AS  Multiple Origin AS (MOAS)  Infrastructure addresses may not be advertised  Addresses announced by someone else

32. March 8, 200432 Limitations of BGP origin AS  Multiple Origin AS (MOAS)  Multi-homing  Misconfiguration  Internet eXchange Points (IXPs)  Infrastructure addresses may not be advertised  Addresses announced by someone else

33. March 8, 200433 Limitations of BGP origin AS  Multiple Origin AS (MOAS)  Infrastructure addresses may not be advertised  Does not require to be announced publicly  Security concerns  Addresses announced by someone else

34. March 8, 200434 Limitations of BGP origin AS  Multiple Origin AS (MOAS)  Infrastructure addresses may not be advertised  Addresses announced by someone else  Static routed customers  Shared equipments at boundary between ASes Need accurate IP-to-AS mapping!

35. March 8, 200435 Accurate AS-level traceroute Combine BGP and traceroute data to find a better answer!

36. March 8, 200436 Assumptions  IP-to-AS mapping  Mappings from BGP tables are mostly correct.  Change slowly  BGP paths and forwarding paths mostly match.  70% of the BGP path and traceroute path match

37. March 8, 200437 BGP path and traceroute path could differ!  Inaccurate IP-to-AS mapping  Traceroute problems  Legitimate mismatches

38. March 8, 200438 BGP path and traceroute path could differ!  Inaccurate IP-to-AS mapping  Internet eXchange Points (IXPs)  Sibling ASes  Unannounced infrastructure addresses  Traceroute problems  Legitimate mismatches

39. March 8, 200439 Internet eXchange Points (IXPs)  Shared infrastructure connected to multiple service providers  Exchange BGP routes and data traffic  May have its own AS number or announced by participating ASes  Dedicated BGP sessions between pairs of participating ASes  E.g., Mae-East, Mae-West, PAIX.

40. March 8, 200440 IXPs cause extra AS hop  Extra AS hop in traceroute path  Large number of fan-in and fan-out ASes  Non-transit AS, small address block, likely MOAS

41. March 8, 200441 IXPs cause extra AS hop A B C D E F G Traceroute AS pathBGP AS path B C F G AE

42. March 8, 200442 Sibling ASes  Single organization owns and manages multiple ASes  May share address space  Large fan-in and fan-out for the “sibling AS pair”

43. March 8, 200443 Sibling ASes cause extra AS hop  Large fan-in and fan-out for the “sibling AS pair” Traceroute AS path BGP AS path A B C D E F G H A B C D E F G

44. March 8, 200444 Unannounced infrastructure addresses  ASes do not necessarily announce infrastructure via BGP  Lead to “unmapped” addresses  Sometimes fall into supernet announced by AS’s provider or sibling

45. March 8, 200445 Unannounced infrastructure addresses 1. A,C AS A AS B AS C 2. A 3. B,A4. A,C,A Extra AS hop in traceroute path Missing AS hop in traceroute path Substitute AS hop AS loop in traceroute path

46. March 8, 200446 BGP path and traceroute path could differ!  Inaccurate IP-to-AS mapping  Traceroute problems  Forwarding path changing during traceroute  Interface numbering at AS boundaries  ICMP response refers to outgoing interface  Legitimate mismatches

47. March 8, 200447 Forwarding path changing during traceroute AS AAS BAS C AS AAS C AS DAS E AS D AS hop B is substituted by AS D in the traceroute path Route flaps between A B C and A D E

48. March 8, 200448 Interface numbering at AS boundaries AS AAS BAS C AS AAS C Missing AS hop B in traceroute path

49. March 8, 200449 ICMP response refers to outgoing interface AS B AS AAS C ICMP message Extra AS hop B in traceroute path

50. March 8, 200450 BGP path and traceroute path could differ!  Inaccurate IP-to-AS mapping  Traceroute problems  Legitimate mismatches  Route aggregation and filtering  Routing anomalies, e.g., deflections

51. March 8, 200451 Route aggregation/filtering 8.0.0.0/8 B C8.0.0.0/8 C 8.64.0.0/16 C D AS BAS CAS A Extended traceroute path due to filtering by AS B

52. March 8, 200452 Mismatch patterns and causes Extra AS Miss AS AS Loop Subst AS Other IXPX Sibling ASesXXXX Unannounced IPXXXX Aggregation/ filteringX Inter-AS interfaceXX ICMP source addressXXXX Routing anomalyXXXXX

53. March 8, 200453 BGP and traceroute data collection Initial mappings from origin AS of a large set of BGP tables Traceroute paths from multiple locations Compare Look for known causes of mismatches (e.g., IXP, sibling ASes) Edit IP-to-AS mappings (a single change explaining a large number of mismatches) For each location: Combine all locations: Local BGP pathsTraceroute AS paths For each location: (Ignoring unstable paths)

54. March 8, 200454 Experimental methodology 200,000 destinations: d 0, d 1, d 2, d 3, d 4, … d 200,000 For each d i -Traceroute path -BGP path

55. March 8, 200455 Measurement setup  Eight vantage points  Upstream providers: US-centric tier-1 ISPs  Sweep all routable IP address space  About 200,000 IP addresses, 160,000 prefixes, 15,000 destination ASes

56. March 8, 200456 Eight vantage points OrganizationLocationUpstream provider AT&T ResearchNJ, USUUNET, AT&T UC BerkeleyCA, USQwest, Level3, Internet 2 PSG home networkWA, USSprint, Verio Univ of WashingtonWA, USVerio, Cable&Wireless ArosNetUT, USUUNET NortelON, CanadaAT&T Canada Vineyard.NETMA, USUUNET, Sprint, Level3 Peak Web HostingCA, USLevel 3, Global Crossing, Teleglobe Many thanks to people who let us collect data!

57. March 8, 200457 Preprocessing BGP paths  Discard prefixes with BGP paths containing  Routing changes based on BGP updates  Private AS numbers (64512 - 65535)  Empty AS paths (local destinations)  AS loops from misconfiguration  AS SET instead of AS sequence  Less than 1% prefixes affected

58. March 8, 200458 Preprocessing traceroute paths  Resolving incomplete traceroute paths  Unresolved hops within a single AS map to that AS  Unmapped hops between ASes  Try match to neighboring AS using DNS, Whois  Trim unresponsive (*) hops at the end  Compare with the beginning of local BGP paths  MOAS at the end of paths  Assume multi-homing without BGP  Validation using AT&T router configurations  More than 98% cases validated

59. March 8, 200459 Initial IP-to-AS Mapping WhoisCombined BGP tables Resolving incompletes Match44.7%73.2%78.0% Mismatch29.4%8.3%9.0% Ratio1.58.89.0

60. March 8, 200460 Heuristics to improve mappings  Overall modification to mappings  10% IP-to-AS mappings modified  25 IXPs identified  28 pairs of sibling ASes found  1150 of the /24 prefixes shared

61. March 8, 200461 Heuristics to improve mappings IXPsSibling ASes Unannounced address space Match84.4%85.9%90.6% Mismatch8.7%7.8%3.5% Ratio9.711.026.0

62. March 8, 200462 Systematic optimization  Dynamic-programming and iterative improvement  Initial IP-to-AS mapping derived from BGP routing tables  Identify a small number of modifications that significantly improve the match rate.  95% match ratio, less than 3% changes, very robust

63. March 8, 200463 Optimization results Mismatch ratio Full initial Mapping5.23% Heuristically optimized mapping3.08% Omit 10% initial mapping6.57% Omit 4 probing sources6.34% Omit probing destinations (one probe per unique BGP path) 7.12%

64. March 8, 200464 Validation  Public data  Whois/DNS data  pch.net for known IXPs  Private data  AS 7018

65. March 8, 200465 Validations – IXP heuristic  25 inferences: 19 confirmed  Whois/DNS data confirm 18 of 25 inferences  AS5459 -- “London Internet Exchange”  198.32.176.0/24: part of “Exchange Point Blocks” DNS name: sfba-unicast1-net.eng.paix.net  Known list from pch.net confirm 16 of 25  Missing 13 known IXPs due to  Limited number of measurement locations  Mostly tier-1 US-centric providers

66. March 8, 200466 Validations – Sibling heuristic  28 inferences: all confirmed  Whois for organization names (15 cases)  E.g., AS1299 and AS8233 are TeliaNet  MOAS origin ASes for several address blocks (13 cases)  E.g., 148.231.0.0/16 has MOAS: AS5677 and AS7132 (Pacific Bell Internet Services and SBC Internet Services)

67. March 8, 200467 Summary  Identify accurate AS level forwarding path  improve infrastructure IP to AS mappings  Heuristics and Dynamic programming optimization  Match/mismatch ratio improvement: 8-12 to 25-35  Reduction of incomplete paths: 18-22% to 6-7%

68. March 8, 200468 Summary  Dependence on operational realities  Most BGP routes are relatively stable  Few private ASes, AS_SETs  Public, routable infrastructure addresses  Routers respond with ICMP replies http://www.research.att.com/~jiawang/as_traceroute