Part III: Measuring Inter- domain Paths

March 8, Packet forwarding path Internet Source Destination IP traffic Forwarding path - the path packets traverse through the Internet from a source to a destination

March 8, An inter-domain level view Internet Source Destination AS A AS B AS C AS D IP traffic An IP forwarding path often span across multiple Autonomous Systems.

March 8, Why do we care?  Characterize end-to-end network paths  Diagnose routing anomalies  Discover Internet topology

March 8, Why do we care?  Characterize end-to-end network paths  Latency  Capacity  Link utilization  Loss rate.  Diagnose routing anomalies  Discover Internet topology

March 8, Varies link capacity Internet Source Destination

March 8, Different loss rate Internet Source Destination

March 8, Traffic engineering Internet Source Destination Customer service enhancement

March 8, Why do we care?  Characterize end-to-end network paths  Diagnose routing anomalies  Forwarding loop, black holes, routing changes, unexpected paths, main component of end-to-end latency.  Discover Internet topology

March 8, Forwarding loops Internet Source Destination

March 8, Black holes Internet Source Destination

March 8, Routing changes Internet Source Destination

March 8, Unexpected routes Internet Source Destination

March 8, Performance bottleneck Internet Source Destination

March 8, Why do we care?  Characterize end-to-end network paths  Diagnose routing anomalies  Discover Internet topology  Server placement

March 8, Internet topology Internet Client Server Client

March 8, Server placement Internet Client Server Client Proxy

March 8, Key challenge  Need to understand how packets flow through the Internet without real-time access to proprietary routing data from each domain.  Identify accurate packet forwarding paths  Characterize the performance metrics of each hop along the paths

March 8, Identify forwarding path  Traceroute gives IP level forwarding path  IP address of the router interfaces on a forwarding path  RTT statistics for each hop along the way

March 8, Traceroute from UC Berkeley to * * inr-daedalus-0.CS.Berkeley.EDU soda-cr-1-1-soda-br-6-2 vlan242.inr-202-doecev.Berkeley.EDU gigE6-0-0.inr-666-doecev.Berkeley.EDU qsv-juniper--ucb-gw.calren2.net POS1-0.hsipaccess1.SanJose1.Level3.net ? pos8-0.hsa2.Atlanta2.Level3.net pop2-atm-P0-2.atdn.net ? pop1-atl-P4-0.atdn.net www4.cnn.com Traceroute output: (hop number, IP address, DNS name) * * inr-daedalus-0.CS.Berkeley.EDU soda-cr-1-1-soda-br-6-2 vlan242.inr-202-doecev.Berkeley.EDU gigE6-0-0.inr-666-doecev.Berkeley.EDU qsv-juniper--ucb-gw.calren2.net POS1-0.hsipaccess1.SanJose1.Level3.net ? pos8-0.hsa2.Atlanta2.Level3.net pop2-atm-P0-2.atdn.net ? pop1-atl-P4-0.atdn.net www4.cnn.com

March 8, Traceroute from AT&T Research to traceroute to cnn.com ( ), 30 hops max, 40 byte packets 1 oden ( ) 1 ms 1 ms 1 ms 2 * * * 3 attlr-gate ( ) 2 ms 2 ms 2 ms ( ) 3 ms 4 ms 4 ms 5 gbr6-p52.n54ny.ip.att.net ( ) 4 ms 4 ms 4 ms 6 tbr2-p n54ny.ip.att.net ( ) 4 ms (ttl=249!) 5 ms (ttl=249!) 5 ms (ttl=249!) 7 ggr2-p390.n54ny.ip.att.net ( ) 4 ms 5 ms 4 ms 8 att-gw.ny.aol.net ( ) 4 ms 4 ms 4 ms 9 bb2-nye-P1-0.atdn.net ( ) 4 ms 4 ms 4 ms 10 bb2-vie-P8-0.atdn.net ( ) 13 ms (ttl=245!) 12 ms (ttl=245!) 12 ms (ttl=245!) 11 bb1-vie-P11-0.atdn.net ( ) 10 ms 10 ms 10 ms 12 bb1-cha-P7-0.atdn.net ( ) 20 ms 20 ms 20 ms 13 bb1-atm-P6-0.atdn.net ( ) 25 ms 25 ms 25 ms 14 pop1-atl-P4-0.atdn.net ( ) 25 ms (ttl=243!) 24 ms (ttl=243!) 24 ms (ttl=243!) 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Who is responsible for the forwarding problem? Destination unreachable!

March 8, Need to know Inter-domain level path Internet AT&T Research AS A AS B AS C AS D Routing loop in AS C!

March 8, How to obtain AS level paths  BGP AS path  Traceroute AS path

March 8, BGP AS path AS A AS B AS C Prefix d Forwarding path: data traffic Signaling path: control traffic d: path=[C] d: path=[BC] PrefixAS path dA B C… Is BGP AS path the answer?No!

March 8, BGP AS path is not the answer  Requires timely access to BGP data  Signaling path may differ from forwarding path  Route aggregation and filtering  Routing anomalies: e.g., deflections, loops [Griffin2002]  BGP misconfigurations: e.g., incorrect AS prepending Two paths may differ precisely when operators most need accurate data to diagnose a problem!

March 8, AS AAS BAS CAS D Traceroute AS path  Obtain IP level path using traceroute  Map IP addresses to ASes Is traceroute AS path the answer?NO! SourceDestination a bcde

March 8, Example: UC Berkeley to CNN * * Traceroute output: (hop number, IP) AS25 AS11423 AS3356 AS1668 AS5662 Berkeley CNN Calren Level3 GNN

March 8, Traceroute AS path is not the answer  Identifying ASes along forwarding path is surprisingly difficult!  Internet route registry  Origin AS in BGP routes

March 8, Internet route registry  Whois database  E.g. NANOG traceroute, prtraceroute  Out-of-date, incomplete  Address allocation to customers  Acquisition, mergers, break-ups

March 8, Origin AS in BGP routes  Last AS in the AS path for each prefix  More accurate and complete than whois data PrefixAS path dA B C ……

March 8, Limitations of BGP origin AS  Multiple Origin AS (MOAS)  Infrastructure addresses may not be advertised  Addresses announced by someone else

March 8, Limitations of BGP origin AS  Multiple Origin AS (MOAS)  Multi-homing  Misconfiguration  Internet eXchange Points (IXPs)  Infrastructure addresses may not be advertised  Addresses announced by someone else

March 8, Limitations of BGP origin AS  Multiple Origin AS (MOAS)  Infrastructure addresses may not be advertised  Does not require to be announced publicly  Security concerns  Addresses announced by someone else

March 8, Limitations of BGP origin AS  Multiple Origin AS (MOAS)  Infrastructure addresses may not be advertised  Addresses announced by someone else  Static routed customers  Shared equipments at boundary between ASes Need accurate IP-to-AS mapping!

March 8, Accurate AS-level traceroute Combine BGP and traceroute data to find a better answer!

March 8, Assumptions  IP-to-AS mapping  Mappings from BGP tables are mostly correct.  Change slowly  BGP paths and forwarding paths mostly match.  70% of the BGP path and traceroute path match

March 8, BGP path and traceroute path could differ!  Inaccurate IP-to-AS mapping  Traceroute problems  Legitimate mismatches

March 8, BGP path and traceroute path could differ!  Inaccurate IP-to-AS mapping  Internet eXchange Points (IXPs)  Sibling ASes  Unannounced infrastructure addresses  Traceroute problems  Legitimate mismatches

March 8, Internet eXchange Points (IXPs)  Shared infrastructure connected to multiple service providers  Exchange BGP routes and data traffic  May have its own AS number or announced by participating ASes  Dedicated BGP sessions between pairs of participating ASes  E.g., Mae-East, Mae-West, PAIX.

March 8, IXPs cause extra AS hop  Extra AS hop in traceroute path  Large number of fan-in and fan-out ASes  Non-transit AS, small address block, likely MOAS

March 8, IXPs cause extra AS hop A B C D E F G Traceroute AS pathBGP AS path B C F G AE

March 8, Sibling ASes  Single organization owns and manages multiple ASes  May share address space  Large fan-in and fan-out for the “sibling AS pair”

March 8, Sibling ASes cause extra AS hop  Large fan-in and fan-out for the “sibling AS pair” Traceroute AS path BGP AS path A B C D E F G H A B C D E F G

March 8, Unannounced infrastructure addresses  ASes do not necessarily announce infrastructure via BGP  Lead to “unmapped” addresses  Sometimes fall into supernet announced by AS’s provider or sibling

March 8, Unannounced infrastructure addresses 1. A,C AS A AS B AS C 2. A 3. B,A4. A,C,A Extra AS hop in traceroute path Missing AS hop in traceroute path Substitute AS hop AS loop in traceroute path

March 8, BGP path and traceroute path could differ!  Inaccurate IP-to-AS mapping  Traceroute problems  Forwarding path changing during traceroute  Interface numbering at AS boundaries  ICMP response refers to outgoing interface  Legitimate mismatches

March 8, Forwarding path changing during traceroute AS AAS BAS C AS AAS C AS DAS E AS D AS hop B is substituted by AS D in the traceroute path Route flaps between A B C and A D E

March 8, Interface numbering at AS boundaries AS AAS BAS C AS AAS C Missing AS hop B in traceroute path

March 8, ICMP response refers to outgoing interface AS B AS AAS C ICMP message Extra AS hop B in traceroute path

March 8, BGP path and traceroute path could differ!  Inaccurate IP-to-AS mapping  Traceroute problems  Legitimate mismatches  Route aggregation and filtering  Routing anomalies, e.g., deflections

March 8, Route aggregation/filtering /8 B C /8 C /16 C D AS BAS CAS A Extended traceroute path due to filtering by AS B

March 8, Mismatch patterns and causes Extra AS Miss AS AS Loop Subst AS Other IXPX Sibling ASesXXXX Unannounced IPXXXX Aggregation/ filteringX Inter-AS interfaceXX ICMP source addressXXXX Routing anomalyXXXXX

March 8, BGP and traceroute data collection Initial mappings from origin AS of a large set of BGP tables Traceroute paths from multiple locations Compare Look for known causes of mismatches (e.g., IXP, sibling ASes) Edit IP-to-AS mappings (a single change explaining a large number of mismatches) For each location: Combine all locations: Local BGP pathsTraceroute AS paths For each location: (Ignoring unstable paths)

March 8, Experimental methodology 200,000 destinations: d 0, d 1, d 2, d 3, d 4, … d 200,000 For each d i -Traceroute path -BGP path

March 8, Measurement setup  Eight vantage points  Upstream providers: US-centric tier-1 ISPs  Sweep all routable IP address space  About 200,000 IP addresses, 160,000 prefixes, 15,000 destination ASes

March 8, Eight vantage points OrganizationLocationUpstream provider AT&T ResearchNJ, USUUNET, AT&T UC BerkeleyCA, USQwest, Level3, Internet 2 PSG home networkWA, USSprint, Verio Univ of WashingtonWA, USVerio, Cable&Wireless ArosNetUT, USUUNET NortelON, CanadaAT&T Canada Vineyard.NETMA, USUUNET, Sprint, Level3 Peak Web HostingCA, USLevel 3, Global Crossing, Teleglobe Many thanks to people who let us collect data!

March 8, Preprocessing BGP paths  Discard prefixes with BGP paths containing  Routing changes based on BGP updates  Private AS numbers ( )  Empty AS paths (local destinations)  AS loops from misconfiguration  AS SET instead of AS sequence  Less than 1% prefixes affected

March 8, Preprocessing traceroute paths  Resolving incomplete traceroute paths  Unresolved hops within a single AS map to that AS  Unmapped hops between ASes  Try match to neighboring AS using DNS, Whois  Trim unresponsive (*) hops at the end  Compare with the beginning of local BGP paths  MOAS at the end of paths  Assume multi-homing without BGP  Validation using AT&T router configurations  More than 98% cases validated

March 8, Initial IP-to-AS Mapping WhoisCombined BGP tables Resolving incompletes Match44.7%73.2%78.0% Mismatch29.4%8.3%9.0% Ratio

March 8, Heuristics to improve mappings  Overall modification to mappings  10% IP-to-AS mappings modified  25 IXPs identified  28 pairs of sibling ASes found  1150 of the /24 prefixes shared

March 8, Heuristics to improve mappings IXPsSibling ASes Unannounced address space Match84.4%85.9%90.6% Mismatch8.7%7.8%3.5% Ratio

March 8, Systematic optimization  Dynamic-programming and iterative improvement  Initial IP-to-AS mapping derived from BGP routing tables  Identify a small number of modifications that significantly improve the match rate.  95% match ratio, less than 3% changes, very robust

March 8, Optimization results Mismatch ratio Full initial Mapping5.23% Heuristically optimized mapping3.08% Omit 10% initial mapping6.57% Omit 4 probing sources6.34% Omit probing destinations (one probe per unique BGP path) 7.12%

March 8, Validation  Public data  Whois/DNS data  pch.net for known IXPs  Private data  AS 7018

March 8, Validations – IXP heuristic  25 inferences: 19 confirmed  Whois/DNS data confirm 18 of 25 inferences  AS “London Internet Exchange”  /24: part of “Exchange Point Blocks” DNS name: sfba-unicast1-net.eng.paix.net  Known list from pch.net confirm 16 of 25  Missing 13 known IXPs due to  Limited number of measurement locations  Mostly tier-1 US-centric providers

March 8, Validations – Sibling heuristic  28 inferences: all confirmed  Whois for organization names (15 cases)  E.g., AS1299 and AS8233 are TeliaNet  MOAS origin ASes for several address blocks (13 cases)  E.g., /16 has MOAS: AS5677 and AS7132 (Pacific Bell Internet Services and SBC Internet Services)

March 8, Summary  Identify accurate AS level forwarding path  improve infrastructure IP to AS mappings  Heuristics and Dynamic programming optimization  Match/mismatch ratio improvement: 8-12 to  Reduction of incomplete paths: 18-22% to 6-7%

March 8, Summary  Dependence on operational realities  Most BGP routes are relatively stable  Few private ASes, AS_SETs  Public, routable infrastructure addresses  Routers respond with ICMP replies