Part III: Measuring Inter- domain Paths
March 8, Packet forwarding path Internet Source Destination IP traffic Forwarding path - the path packets traverse through the Internet from a source to a destination
March 8, An inter-domain level view Internet Source Destination AS A AS B AS C AS D IP traffic An IP forwarding path often span across multiple Autonomous Systems.
March 8, Why do we care? Characterize end-to-end network paths Diagnose routing anomalies Discover Internet topology
March 8, Why do we care? Characterize end-to-end network paths Latency Capacity Link utilization Loss rate. Diagnose routing anomalies Discover Internet topology
March 8, Varies link capacity Internet Source Destination
March 8, Different loss rate Internet Source Destination
March 8, Traffic engineering Internet Source Destination Customer service enhancement
March 8, Why do we care? Characterize end-to-end network paths Diagnose routing anomalies Forwarding loop, black holes, routing changes, unexpected paths, main component of end-to-end latency. Discover Internet topology
March 8, Forwarding loops Internet Source Destination
March 8, Black holes Internet Source Destination
March 8, Routing changes Internet Source Destination
March 8, Unexpected routes Internet Source Destination
March 8, Performance bottleneck Internet Source Destination
March 8, Why do we care? Characterize end-to-end network paths Diagnose routing anomalies Discover Internet topology Server placement
March 8, Internet topology Internet Client Server Client
March 8, Server placement Internet Client Server Client Proxy
March 8, Key challenge Need to understand how packets flow through the Internet without real-time access to proprietary routing data from each domain. Identify accurate packet forwarding paths Characterize the performance metrics of each hop along the paths
March 8, Identify forwarding path Traceroute gives IP level forwarding path IP address of the router interfaces on a forwarding path RTT statistics for each hop along the way
March 8, Traceroute from UC Berkeley to * * inr-daedalus-0.CS.Berkeley.EDU soda-cr-1-1-soda-br-6-2 vlan242.inr-202-doecev.Berkeley.EDU gigE6-0-0.inr-666-doecev.Berkeley.EDU qsv-juniper--ucb-gw.calren2.net POS1-0.hsipaccess1.SanJose1.Level3.net ? pos8-0.hsa2.Atlanta2.Level3.net pop2-atm-P0-2.atdn.net ? pop1-atl-P4-0.atdn.net www4.cnn.com Traceroute output: (hop number, IP address, DNS name) * * inr-daedalus-0.CS.Berkeley.EDU soda-cr-1-1-soda-br-6-2 vlan242.inr-202-doecev.Berkeley.EDU gigE6-0-0.inr-666-doecev.Berkeley.EDU qsv-juniper--ucb-gw.calren2.net POS1-0.hsipaccess1.SanJose1.Level3.net ? pos8-0.hsa2.Atlanta2.Level3.net pop2-atm-P0-2.atdn.net ? pop1-atl-P4-0.atdn.net www4.cnn.com
March 8, Traceroute from AT&T Research to traceroute to cnn.com ( ), 30 hops max, 40 byte packets 1 oden ( ) 1 ms 1 ms 1 ms 2 * * * 3 attlr-gate ( ) 2 ms 2 ms 2 ms ( ) 3 ms 4 ms 4 ms 5 gbr6-p52.n54ny.ip.att.net ( ) 4 ms 4 ms 4 ms 6 tbr2-p n54ny.ip.att.net ( ) 4 ms (ttl=249!) 5 ms (ttl=249!) 5 ms (ttl=249!) 7 ggr2-p390.n54ny.ip.att.net ( ) 4 ms 5 ms 4 ms 8 att-gw.ny.aol.net ( ) 4 ms 4 ms 4 ms 9 bb2-nye-P1-0.atdn.net ( ) 4 ms 4 ms 4 ms 10 bb2-vie-P8-0.atdn.net ( ) 13 ms (ttl=245!) 12 ms (ttl=245!) 12 ms (ttl=245!) 11 bb1-vie-P11-0.atdn.net ( ) 10 ms 10 ms 10 ms 12 bb1-cha-P7-0.atdn.net ( ) 20 ms 20 ms 20 ms 13 bb1-atm-P6-0.atdn.net ( ) 25 ms 25 ms 25 ms 14 pop1-atl-P4-0.atdn.net ( ) 25 ms (ttl=243!) 24 ms (ttl=243!) 24 ms (ttl=243!) 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Who is responsible for the forwarding problem? Destination unreachable!
March 8, Need to know Inter-domain level path Internet AT&T Research AS A AS B AS C AS D Routing loop in AS C!
March 8, How to obtain AS level paths BGP AS path Traceroute AS path
March 8, BGP AS path AS A AS B AS C Prefix d Forwarding path: data traffic Signaling path: control traffic d: path=[C] d: path=[BC] PrefixAS path dA B C… Is BGP AS path the answer?No!
March 8, BGP AS path is not the answer Requires timely access to BGP data Signaling path may differ from forwarding path Route aggregation and filtering Routing anomalies: e.g., deflections, loops [Griffin2002] BGP misconfigurations: e.g., incorrect AS prepending Two paths may differ precisely when operators most need accurate data to diagnose a problem!
March 8, AS AAS BAS CAS D Traceroute AS path Obtain IP level path using traceroute Map IP addresses to ASes Is traceroute AS path the answer?NO! SourceDestination a bcde
March 8, Example: UC Berkeley to CNN * * Traceroute output: (hop number, IP) AS25 AS11423 AS3356 AS1668 AS5662 Berkeley CNN Calren Level3 GNN
March 8, Traceroute AS path is not the answer Identifying ASes along forwarding path is surprisingly difficult! Internet route registry Origin AS in BGP routes
March 8, Internet route registry Whois database E.g. NANOG traceroute, prtraceroute Out-of-date, incomplete Address allocation to customers Acquisition, mergers, break-ups
March 8, Origin AS in BGP routes Last AS in the AS path for each prefix More accurate and complete than whois data PrefixAS path dA B C ……
March 8, Limitations of BGP origin AS Multiple Origin AS (MOAS) Infrastructure addresses may not be advertised Addresses announced by someone else
March 8, Limitations of BGP origin AS Multiple Origin AS (MOAS) Multi-homing Misconfiguration Internet eXchange Points (IXPs) Infrastructure addresses may not be advertised Addresses announced by someone else
March 8, Limitations of BGP origin AS Multiple Origin AS (MOAS) Infrastructure addresses may not be advertised Does not require to be announced publicly Security concerns Addresses announced by someone else
March 8, Limitations of BGP origin AS Multiple Origin AS (MOAS) Infrastructure addresses may not be advertised Addresses announced by someone else Static routed customers Shared equipments at boundary between ASes Need accurate IP-to-AS mapping!
March 8, Accurate AS-level traceroute Combine BGP and traceroute data to find a better answer!
March 8, Assumptions IP-to-AS mapping Mappings from BGP tables are mostly correct. Change slowly BGP paths and forwarding paths mostly match. 70% of the BGP path and traceroute path match
March 8, BGP path and traceroute path could differ! Inaccurate IP-to-AS mapping Traceroute problems Legitimate mismatches
March 8, BGP path and traceroute path could differ! Inaccurate IP-to-AS mapping Internet eXchange Points (IXPs) Sibling ASes Unannounced infrastructure addresses Traceroute problems Legitimate mismatches
March 8, Internet eXchange Points (IXPs) Shared infrastructure connected to multiple service providers Exchange BGP routes and data traffic May have its own AS number or announced by participating ASes Dedicated BGP sessions between pairs of participating ASes E.g., Mae-East, Mae-West, PAIX.
March 8, IXPs cause extra AS hop Extra AS hop in traceroute path Large number of fan-in and fan-out ASes Non-transit AS, small address block, likely MOAS
March 8, IXPs cause extra AS hop A B C D E F G Traceroute AS pathBGP AS path B C F G AE
March 8, Sibling ASes Single organization owns and manages multiple ASes May share address space Large fan-in and fan-out for the “sibling AS pair”
March 8, Sibling ASes cause extra AS hop Large fan-in and fan-out for the “sibling AS pair” Traceroute AS path BGP AS path A B C D E F G H A B C D E F G
March 8, Unannounced infrastructure addresses ASes do not necessarily announce infrastructure via BGP Lead to “unmapped” addresses Sometimes fall into supernet announced by AS’s provider or sibling
March 8, Unannounced infrastructure addresses 1. A,C AS A AS B AS C 2. A 3. B,A4. A,C,A Extra AS hop in traceroute path Missing AS hop in traceroute path Substitute AS hop AS loop in traceroute path
March 8, BGP path and traceroute path could differ! Inaccurate IP-to-AS mapping Traceroute problems Forwarding path changing during traceroute Interface numbering at AS boundaries ICMP response refers to outgoing interface Legitimate mismatches
March 8, Forwarding path changing during traceroute AS AAS BAS C AS AAS C AS DAS E AS D AS hop B is substituted by AS D in the traceroute path Route flaps between A B C and A D E
March 8, Interface numbering at AS boundaries AS AAS BAS C AS AAS C Missing AS hop B in traceroute path
March 8, ICMP response refers to outgoing interface AS B AS AAS C ICMP message Extra AS hop B in traceroute path
March 8, BGP path and traceroute path could differ! Inaccurate IP-to-AS mapping Traceroute problems Legitimate mismatches Route aggregation and filtering Routing anomalies, e.g., deflections
March 8, Route aggregation/filtering /8 B C /8 C /16 C D AS BAS CAS A Extended traceroute path due to filtering by AS B
March 8, Mismatch patterns and causes Extra AS Miss AS AS Loop Subst AS Other IXPX Sibling ASesXXXX Unannounced IPXXXX Aggregation/ filteringX Inter-AS interfaceXX ICMP source addressXXXX Routing anomalyXXXXX
March 8, BGP and traceroute data collection Initial mappings from origin AS of a large set of BGP tables Traceroute paths from multiple locations Compare Look for known causes of mismatches (e.g., IXP, sibling ASes) Edit IP-to-AS mappings (a single change explaining a large number of mismatches) For each location: Combine all locations: Local BGP pathsTraceroute AS paths For each location: (Ignoring unstable paths)
March 8, Experimental methodology 200,000 destinations: d 0, d 1, d 2, d 3, d 4, … d 200,000 For each d i -Traceroute path -BGP path
March 8, Measurement setup Eight vantage points Upstream providers: US-centric tier-1 ISPs Sweep all routable IP address space About 200,000 IP addresses, 160,000 prefixes, 15,000 destination ASes
March 8, Eight vantage points OrganizationLocationUpstream provider AT&T ResearchNJ, USUUNET, AT&T UC BerkeleyCA, USQwest, Level3, Internet 2 PSG home networkWA, USSprint, Verio Univ of WashingtonWA, USVerio, Cable&Wireless ArosNetUT, USUUNET NortelON, CanadaAT&T Canada Vineyard.NETMA, USUUNET, Sprint, Level3 Peak Web HostingCA, USLevel 3, Global Crossing, Teleglobe Many thanks to people who let us collect data!
March 8, Preprocessing BGP paths Discard prefixes with BGP paths containing Routing changes based on BGP updates Private AS numbers ( ) Empty AS paths (local destinations) AS loops from misconfiguration AS SET instead of AS sequence Less than 1% prefixes affected
March 8, Preprocessing traceroute paths Resolving incomplete traceroute paths Unresolved hops within a single AS map to that AS Unmapped hops between ASes Try match to neighboring AS using DNS, Whois Trim unresponsive (*) hops at the end Compare with the beginning of local BGP paths MOAS at the end of paths Assume multi-homing without BGP Validation using AT&T router configurations More than 98% cases validated
March 8, Initial IP-to-AS Mapping WhoisCombined BGP tables Resolving incompletes Match44.7%73.2%78.0% Mismatch29.4%8.3%9.0% Ratio
March 8, Heuristics to improve mappings Overall modification to mappings 10% IP-to-AS mappings modified 25 IXPs identified 28 pairs of sibling ASes found 1150 of the /24 prefixes shared
March 8, Heuristics to improve mappings IXPsSibling ASes Unannounced address space Match84.4%85.9%90.6% Mismatch8.7%7.8%3.5% Ratio
March 8, Systematic optimization Dynamic-programming and iterative improvement Initial IP-to-AS mapping derived from BGP routing tables Identify a small number of modifications that significantly improve the match rate. 95% match ratio, less than 3% changes, very robust
March 8, Optimization results Mismatch ratio Full initial Mapping5.23% Heuristically optimized mapping3.08% Omit 10% initial mapping6.57% Omit 4 probing sources6.34% Omit probing destinations (one probe per unique BGP path) 7.12%
March 8, Validation Public data Whois/DNS data pch.net for known IXPs Private data AS 7018
March 8, Validations – IXP heuristic 25 inferences: 19 confirmed Whois/DNS data confirm 18 of 25 inferences AS “London Internet Exchange” /24: part of “Exchange Point Blocks” DNS name: sfba-unicast1-net.eng.paix.net Known list from pch.net confirm 16 of 25 Missing 13 known IXPs due to Limited number of measurement locations Mostly tier-1 US-centric providers
March 8, Validations – Sibling heuristic 28 inferences: all confirmed Whois for organization names (15 cases) E.g., AS1299 and AS8233 are TeliaNet MOAS origin ASes for several address blocks (13 cases) E.g., /16 has MOAS: AS5677 and AS7132 (Pacific Bell Internet Services and SBC Internet Services)
March 8, Summary Identify accurate AS level forwarding path improve infrastructure IP to AS mappings Heuristics and Dynamic programming optimization Match/mismatch ratio improvement: 8-12 to Reduction of incomplete paths: 18-22% to 6-7%
March 8, Summary Dependence on operational realities Most BGP routes are relatively stable Few private ASes, AS_SETs Public, routable infrastructure addresses Routers respond with ICMP replies