The Stable Paths Problem As A Model Of BGP Routing NJIT April 24, 2002 Timothy G. Griffin AT&T Research

Outline Part I: The glue that holds the Internet together : interdomain routing with The Border Gateway Protocol (BGP) Part II: A formal model of BGP routing policies Joint work with Bruce Shepherd and Gordon Wilfong (Bell Labs)

Architecture of Dynamic Routing AS 1 AS 2 BGP EGP = Exterior Gateway Protocol IGP = Interior Gateway Protocol Metric based: OSPF, IS-IS, RIP, EIGRP (cisco) Policy based: BGP The Routing Domain of BGP is the entire Internet OSPF EIGRP

Topology information is flooded within the routing domain Best end-to-end paths are computed locally at each router. Best end-to-end paths determine next-hops. Based on minimizing some notion of distance Works only if policy is shared and uniform Examples: OSPF, IS-IS Each router knows little about network topology Only best next-hops are chosen by each router for each destination network. Best end-to-end paths result from composition of all next-hop choices Does not require any notion of distance Does not require uniform policies at all routers Examples: RIP, BGP Link StateVectoring Technology of Distributed Routing

The Gang of Four Link StateVectoring EGP IGP BGP RIP IS-IS OSPF

6 Many Routing Processes Can Run on a Single Router Forwarding Table OSPF Domain RIP Domain BGP OS kernel OSPF Process OSPF Routing tables RIP Process RIP Routing tables BGP Process BGP Routing tables Forwarding Table Manager

AS Numbers (ASNs) ASNs are 16 bit values through are “private” Yale: 29 MIT: 3 Harvard: 11 Genuity: 1 AT&T: 7018, 6341, 5074, … UUNET: 701, 702, 284, 12199, … Sprint: 1239, 1240, 6211, 6242, … … ASNs represent units of routing policy Currently over 12,000 in use.

Autonomous Routing Domains Don’t Always Need BGP or an ASN Qwest Yale University Nail up default routes /0 pointing to Qwest Nail up routes /16 pointing to Yale /16 Static routing is the most common way of connecting an autonomous routing domain to the Internet. This helps explain why BGP is a mystery to many …

ASNs Can Be “Shared” (RFC 2270) AS 701 UUNet ASN 7046 is assigned to UUNet. It is used by Customers single homed to UUNet, but needing BGP for some reason (load balancing, etc..) [RFC 2270] AS 7046 Crestar Bank AS 7046 NJIT AS 7046 Hood College /16

How Many ASNs are there? Thanks to Geoff Huston. on June 23, 2001

AS Graphs Can Be Fun The subgraph showing all ASes that have more than 100 neighbors in full graph of 11,158 nodes. July 6, Point of view: AT&T route-server

BGP Table Growth Thanks to Geoff Huston. on August 8, 2001

13 Nontransit vs. Transit ASes ISP 1 ISP 2 Nontransit AS might be a corporate or campus network. Could be a “content provider” NET A Traffic NEVER flows from ISP 1 through NET A to ISP 2 (At least not intentionally!) IP traffic Internet Service providers (often) have transit networks

14 Selective Transit NET B NET C NET A provides transit between NET B and NET C and between NET D and NET C NET A NET D NET A DOES NOT provide transit Between NET D and NET B Most transit networks transit in a selective manner… IP traffic

Customers and Providers Customer pays provider for access to the Internet provider customer IP traffic provider customer

The Peering Relationship peer customerprovider Peers provide transit between their respective customers Peers do not provide transit between peers Peers (often) do not exchange $$$ traffic allowed traffic NOT allowed

Peering Provides Shortcuts Peering also allows connectivity between the customers of “Tier 1” providers. peer customerprovider

18 BGP-4 BGP = Border Gateway Protocol Is a Policy-Based routing protocol Is the de facto EGP of today’s global Internet Relatively simple protocol, but configuration is complex and the entire world can see, and be impacted by, your mistakes : BGP-1 [RFC 1105] –Replacement for EGP (1984, RFC 904) 1990 : BGP-2 [RFC 1163] 1991 : BGP-3 [RFC 1267] 1995 : BGP-4 [RFC 1771] –Support for Classless Interdomain Routing (CIDR)

19 BGP Operations (Simplified) Establish session on TCP port 179 Exchange all active routes Exchange incremental updates AS1 AS2 While connection is ALIVE exchange route UPDATE messages BGP session

20 Four Types of BGP Messages Open : Establish a peering session. Keep Alive : Handshake at regular intervals. Notification : Shuts down a peering session. Update : Announcing new routes or withdrawing previously announced routes. announcement = prefix + attributes values

BGP Attributes Value Code Reference ORIGIN [RFC1771] 2 AS_PATH [RFC1771] 3 NEXT_HOP [RFC1771] 4 MULTI_EXIT_DISC [RFC1771] 5 LOCAL_PREF [RFC1771] 6 ATOMIC_AGGREGATE [RFC1771] 7 AGGREGATOR [RFC1771] 8 COMMUNITY [RFC1997] 9 ORIGINATOR_ID [RFC2796] 10 CLUSTER_LIST [RFC2796] 11 DPA [Chen] 12 ADVERTISER [RFC1863] 13 RCID_PATH / CLUSTER_ID [RFC1863] 14 MP_REACH_NLRI [RFC2283] 15 MP_UNREACH_NLRI [RFC2283] 16 EXTENDED COMMUNITIES [Rosen] reserved for development From IANA: Most important attributes Not all attributes need to be present in every announcement

Attributes are Used to Select Best Routes /24 pick me! /24 pick me! /24 pick me! /24 pick me! Given multiple routes to the same prefix, a BGP speaker must pick at most one best route (Note: it could reject them all!)

23 BGP Route Processing Best Route Selection Apply Import Policies Best Route Table Apply Export Policies Install forwarding Entries for best Routes. Receive BGP Updates Best Routes Transmit BGP Updates Apply Policy = filter routes & tweak attributes Based on Attribute Values IP Forwarding Table Apply Policy = filter routes & tweak attributes Open ended programming. Constrained only by vendor configuration language

Route Selection Summary Highest Local Preference Shortest ASPATH Lowest MED i-BGP < e-BGP Lowest IGP cost to BGP egress Lowest router ID traffic engineering Enforce relationships Throw up hands and break ties

Tweak Tweak Tweak For inbound traffic –Filter outbound routes –Tweak attributes on outbound routes in the hope of influencing your neighbor’s best route selection For outbound traffic –Filter inbound routes –Tweak attributes on inbound routes to influence best route selection outbound routes inbound routes inbound traffic outbound traffic In general, an AS has more control over outbound traffic

26 ASPATH Attribute AS /16 AS Path = 6341 AS 1239 Sprint AS 1755 Ebone AT&T AS 3549 Global Crossing /16 AS Path = /16 AS Path = AS /16 AT&T Research Prefix Originated AS RIPE NCC RIS project AS 1129 Global Access /16 AS Path = /16 AS Path = /16 AS Path = /16 AS Path =

AS Graphs Do Not Show Topology! The AS graph may look like this. Reality may be closer to this… BGP was designed to throw away information!

AS Graphs Depend on Point of View This explains why there is no UUNET (701) Sprint (1239) link on previous slide! peer customer provider

In fairness: could you do this “right” and still scale? Exporting internal state would dramatically increase global instability and amount of routing state Shorter Doesn’t Always Mean Shorter AS 4 AS 3 AS 2 AS 1 Mr. BGP says that path 4 1 is better than path Duh!

30 Shedding Inbound Traffic with ASPATH Padding Hack Padding will (usually) force inbound traffic from AS 1 to take primary link AS /24 ASPATH = customer AS 2 provider /24 backupprimary /24 ASPATH = 2

31 Padding May Not Shut Off All Traffic AS /24 ASPATH = customer AS 2 provider /24 ASPATH = 2 AS 3 provider AS 3 will send traffic on “backup” link because it prefers customer routes and local preference is considered before ASPATH length! Padding in this way is often used as a form of load balancing backupprimary

32 COMMUNITY Attribute to the Rescue! AS 1 customer AS 2 provider /24 ASPATH = 2 AS 3 provider backupprimary /24 ASPATH = 2 COMMUNITY = 3:70 Customer import policy at AS 3: If 3:90 in COMMUNITY then set local preference to 90 If 3:80 in COMMUNITY then set local preference to 80 If 3:70 in COMMUNITY then set local preference to 70 AS 3: normal customer local pref is 100, peer local pref is 90

33 Hot Potato Routing: Go for the Closest Egress Point / IGP distances egress 1 egress 2 This Router has two BGP routes to /24. Hot potato: get traffic off of your network as Soon as possible. Go for egress 1!

34 Getting Burned by the Hot Potato High bandwidth Provider backbone Low bandwidth customer backbone Heavy Content Web Farm Many customers want their provider to carry the bits! tiny http request huge http reply SFFNYC San Diego

35 Cold Potato Routing with MEDs (Multi-Exit Discriminator Attribute) Heavy Content Web Farm /24 MED = /24 MED = 56 This means that MEDs must be considered BEFORE IGP distance! Prefer lower MED values Note1 : some providers will not listen to MEDs Note2 : MEDs need not be tied to IGP distance

Policies Can Interact Strangely (“Route Pinning” Example) backup Disaster strikes primary link and the backup takes over Primary link is restored but some traffic remains pinned to backup Install backup link using community customer

News at 11:00h BGP is not guaranteed to converge on a stable routing. Policy interactions could lead to “livelock” protocol oscillations. See “Persistent Route Oscillations in Inter-domain Routing” by K. Varadhan, R. Govindan, and D. Estrin. ISI report, 1996 Corollary: BGP is not guaranteed to recover from network failures.

PART II

What Problem is BGP solving? Underlying problem Shortest Paths Distributed means of computing a solution. X? RIP, OSPF, IS-IS BGP aid in the design of policy analysis algorithms and heuristics aid in the analysis and design of BGP and extensions help explain some BGP routing anomalies provide a fun way of thinking about the protocol X could Can we model BGP?

Separate dynamic and static semantics SPVP = Simple Path Vector Protocol = a distributed algorithm for solving SPP BGP SPVP Booo Hooo, Many, many complications... BGP Policies Stable Paths Problem (SPP) static semantics dynamic semantics

1 An instance of the Stable Paths Problem (SPP) A graph of nodes and edges, Node 0, called the origin, For each non-zero node, a set or permitted paths to the origin. This set always contains the “null path”. A ranking of permitted paths at each node. Null path is always least preferred. (Not shown in diagram) When modeling BGP : nodes represent BGP speaking routers, and 0 represents a node originating some address block most preferred … least preferred Yes, the translation gets messy!

A Solution to a Stable Paths Problem node u’s assigned path is either the null path or is a path uwP, where wP is assigned to node w and {u,w} is an edge in the graph, each node is assigned the highest ranked path among those consistent with the paths assigned to its neighbors. A Solution need not represent a shortest path tree, or a spanning tree. A solution is an assignment of permitted paths to each node such that

An SPP may have multiple solutions First solution Second solution DISAGREE

Multiple solutions can result in “Route Triggering” Remove primary linkRestore primary link primary link backup link

BAD GADGET : No Solution Persistent Route Oscillations in Inter-Domain Routing. Kannan Varadhan, Ramesh Govindan, and Deborah Estrin. Computer Networks, Jan. 2000

SURPRISE : Beware of Backup Policies Becomes a BAD GADGET if link (4, 0) goes down. BGP is not robust : it is not guaranteed to recover from network failures.

PRECARIOUS As with DISAGREE, this part has two distinct solutions This part has a solution only when node 1 is assigned the direct path (1 0). Has a solution, but can get “trapped”

Solving an SPP Just enumerate all path assignments And check stability of each…. Exponential complexity But, in worst case you (probably) can’t do any better…

Use 3-SAT… Variables V = {X1, X2, …, Xn} Clauses C1 = X17 or ~X23 or ~X3, C2 = ~X2 or X3 or ~X12 …. Cm = X6 or ~X7 or X18 Question Is there an variable assignment A : V {true, false} such that each clause C1, …,Cm is true? 3-SAT is NP-complete

Modeling assignment to variable X X 0 X X = trueX = false X0 X X0 X X X 0 X 0 X X 0 X 0

SPP Solvability is NP-complete BAD GADGET 0 X5 X7 X3 C X7 or X5 or X3 C X7 0 C X5 0 C X3 0

SPVP protocol process spvp[u] { receive P from w  { rib-in(u  w) := u P if rib(u) != best(u) { rib(u) := best(u) foreach v in peers(u) { send rib(u) to v } Pick the best path available at any given time…

SPVP wanders around assignment space = assignment = solution

Distributed algorithms to solve SPP? OSPF-like : –Distribute topology, path ranks –Solve SPP locally –Exponential worst case –How can loops be avoided when multiple solutions exist? RIP-like: –Pick the best path from the set of your neighbor’s paths, tell your neighbors when you change your mind –Can diverge –Not guaranteed to find a solution, even when one exists –Even when converges, no bound on convergence time This is BGP…

A sufficient condition for sanity Static (SPP) solvable Dynamic (SPVP) unique solution safe (can’t diverge) predictable restoration If an instance of SPP has an acyclic dispute digraph, then all sub-problems uniquely solvable robust with respect to link/node failures

Dispute Digraph … (u v)P … (u v)Q... … Q … P... uv0 P Q Q (u v)P Gives the dispute arc

Dispute Digraph (cont.) … (u,v)P... … P... uv0 P P (u,v)P Gives the transmission arc

Dispute Digraph Example BAD GADGET II CYCLE

What is to be done? Static Approach Inter-AS coordination Automated Analysis of Routing Policies Dynamic Approach Extend BGP with a dynamic means of detecting and suppressing policy-based oscillations? These approaches are complementary

Some Applications SPP Theory A Safe Path Vector Protocol. Timothy G. Griffin, Gordon Wilfong. INFOCOM 2001 –Dynamic solution for SPVP based on histories (dynamically constructed dispute cycles). Inherently safe backup routing with BGP. Lixin Gao, Timothy G. Griffin, Jennifer Rexford. INFOCOM 2001 Show that if customer/provider peer/peer model is followed, then all is well, –Show that this can be exteded with complex backup policies and remain safe. –Analysis of “cold potato” routing problems (MED oscillation). Griffin and Wilfong. Work in progress –MED requires a modification to SPP model –Analysis of Internal BGP (IBGP) configuration. Griffin and Wilfong. Work in progress.

A Few Research Topics Dynamic Behavior of BGP –Convergence time, message overhead BGP Security –S-BGP defined, but not deployed. Is it a good solution. –Need an “interdomain trust model” Beyond BGP? –When will it break? What will replace it?

Selected Papers on BGP Sanity Persistent Route Oscillations in Inter-Domain Routing. Kannan Varadhan, Ramesh Govindan, and Deborah Estrin. Computer Networks, Jan (Also USC Tech Report, Feb. 1996) –Shows that BGP is not guaranteed to converge An Architecture for Stable, Analyzable Internet Routing. Ramesh Govindan, Cengiz Alaettinoglu, George Eddy, David Kessens, Satish Kumar, and WeeSan Lee. IEEE Network Magazine, Jan-Feb –Use RPSL to specify policies. Store them in registries. Use registry for conguration generation and analysis. An Analysis of BGP Convergence Properties. Timothy G. Griffin, Gordon Wilfong. SIGCOMM 1999 –Model BGP, shows static analysis of divergence in policies is NP complete Policy Disputes in Path Vector Protocols. Timothy G. Griffin, F. Bruce Shepherd, Gordon Wilfong. ICNP 1999 –Define Stable Paths Problem and develop sufficient condition for “sanity” A Safe Path Vector Protocol. Timothy G. Griffin, Gordon Wilfong. INFOCOM 2001 –Dynamic solution for SPVP based on histories Stable Internet Routing without Global Coordination. Lixin Gao, Jennifer Rexford. SIGMETRICS 2000 –Show that if certain guidelines are followed, then all is well. Inherently safe backup routing with BGP. Lixin Gao, Timothy G. Griffin, Jennifer Rexford. INFOCOM 2001 –Use SPP to study complex backup policies

Pointers Links on Interdomain routing and BGP: SIGCOMM 2001 Tutorial on BGP: _tutorial/abstract.html Papers on BGP theory:

Thank You!