1. Lecture 14: Inter-domain Routing Stability CS 268 class March 8 th, 2004 (slides from Timothy Griffin’s tutorial and Craig Labovitz’s NANOG talk)

2. Outline of the Today’s class An Introduction to BGP BGP and the Stable Paths problem Convergence of BGP in the real world Conclusions and Open Issues

3. Inter-domain Routing basics Internet is composed of over 16000 autonomous systems BGP = Border Gateway Protocol –Is a Policy-Based routing protocol –Is the de facto inter-domain routing protocol of today ’ s global Internet Relatively simple protocol, but configuration is complex and the entire world can see, and be impacted by, your mistakes.

4. BGP Operations (Simplified) Establish session on TCP port 179 Exchange all active routes Exchange incremental updates AS1 AS2 While connection is ALIVE exchange route UPDATE messages BGP session

5. Four Types of BGP Messages Open : Establish a peering session. Keep Alive : Handshake at regular intervals. Notification : Shuts down a peering session. Update : Announcing new routes or withdrawing previously announced routes. announcement = prefix + attributes values

6. Two Types of BGP Neighbor Relationships External Neighbor (eBGP) in a different Autonomous Systems Internal Neighbor (iBGP) in the same Autonomous System AS1 AS2 eBGP iBGP iBGP is routed (using IGP!)

7. iBGP Peers Must be Fully Meshed iBGP neighbors do not announce routes received via iBGP to other iBGP neighbors. eBGP update iBGP updates iBGP is needed to avoid routing loops within an AS Injecting external routes into IGP does not scale and causes BGP policy information to be lost BGP does not provide “shortest path” routing Is iBGP an IGP? NO!

8. Important BGP attributes LocalPREF –Local preference policy to choose “most” preferred route Multi-exit Discriminator –Which peering point to choose? Import Rules –What route advertisements do I accept? Export Rules –Which routes do I forward to whom?

9. Route Selection Summary Highest Local Preference Shortest ASPATH Lowest MED i-BGP < e-BGP Lowest IGP cost to BGP egress Lowest router ID traffic engineering Enforce relationships Throw up hands and break ties

10. Implementing Customer/Provider and Peer/Peer relationships Enforce transit relationships –Outbound route filtering Enforce order of route preference –provider < peer < customer Two parts:

11. Import Routes From peer From peer From provider From provider From customer From customer provider routecustomer routepeer routeISP route

12. Export Routes To peer To peer To customer To customer To provider From provider provider routecustomer routepeer routeISP route filters block

13. Outline of the Today’s class An Introduction to BGP BGP and the Stable Paths problem Convergence of BGP in the real world Conclusions and Open Issues

14. This talk What Problem is BGP solving? Underlying problem Shortest Paths Distributed means of computing a solution. X? RIP, OSPF, IS-IS BGP aid in the design of policy analysis algorithms and heuristics, aid in the analysis and design of BGP and extensions, help explain some BGP routing anomalies, provide a fun way of thinking about the protocol Having an X can

15. 1 Q : How simple can X get? A: The Stable Paths Problem (SPP) 2 5 5 2 1 0 0 2 1 0 2 0 1 3 0 1 0 3 0 4 2 0 4 3 0 3 4 2 1 A graph of nodes and edges, Node 0, called the origin, For each non-zero node, a set or permitted paths to the origin. This set always contains the “ null path ”. A ranking of permitted paths at each node. Null path is always least preferred. (Not shown in diagram) An instance of the SPP : When modeling BGP : nodes represent BGP speaking border routers, and 0 represents a node originating some address block most preferred … least preferred (not null) Yes, the translation gets messy!

16. 5 5 2 1 0 1 A Solution to a Stable Paths Problem 2 0 2 1 0 2 0 1 3 0 1 0 3 0 4 2 0 4 3 0 3 4 2 1 node u ’ s assigned path is either the null path or is a path uwP, where wP is assigned to node w and {u,w} is an edge in the graph, each node is assigned the highest ranked path among those consistent with the paths assigned to its neighbors. A Solution need not represent a shortest path tree, or a spanning tree. A solution is an assignment of permitted paths to each node such that

17. A Stable Paths Problem may have multiple solutions First solution 102 1 2 0 1 0 102102 2 1 0 2 0 1 2 0 1 0 2 1 0 2 0 1 2 0 1 0 2 1 0 2 0 Second solution DISAGREE

18. Multiple sets of BGP routing policies can map down to the same Stable Paths Problem : DISAGREE in RPSL (Version I) 102 1 2 0 1 0 2 1 0 2 0 import : from AS1 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS2 announce ANY; import : from AS2 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS1 announce ANY; export : to AS1, AS2 announce AS0;

19. DISAGREE in RPSL (Version II) 102 1 2 0 1 0 2 1 0 2 0 import : from AS-ANY action pref = 0; accept community.contains(1:1); from AS-ANY action pref = 10; accept ANY; export : to AS2 announce ANY; export : to AS1 set community.append(2:1); announce AS0; to AS2 set community.append(1:1); announce AS0 import : from AS-ANY action pref = 0; accept community.contains(2:1); from AS-ANY action pref = 10; accept ANY; export : to AS1 announce ANY; Assume AS1 and AS2 use “neighbor send-community” command ….

20. DISAGREE in RPSL (Version III) 1 0 2 1 2 0 1 0 2 1 0 2 0 import : from AS-ANY accept ANY; export : to AS2 announce ANY; export : to AS1 action aspath.prepend(AS0, AS0, AS0); announce AS0; to AS2 announce AS0 import : from AS1 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS1 announce ANY; The interaction of all BGP policies is directly represented in SPP

21. Multiple solutions can result in “ Route Triggering ” 1023 1 0 1 2 3 0 2 3 0 2 1 0 3 2 1 0 3 0 1 02 3 1 02 3 Remove primary linkRestore primary link 1 0 1 2 3 0 2 3 0 3 1 0 3 2 1 0 3 0 primary link backup link

22. SPP helps explain possibility of BGP divergence BGP is not guaranteed to converge to a stable routing. Policy inconsistencies can lead to “ livelock ” protocol oscillations. See “ Persistent Route Oscillations in Inter-domain Routing ” by K. Varadhan, R. Govindan, and D. Estrin. ISI report, 1996 SolvableCan Diverge must converge must diverge The SPP view :

23. 2 0 3 1 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 4 4 2 0 4 3 0 BAD GADGET : No Solution With a BGP-like protocol, each node will do the best it can, so at least one node will always have the opportunity to improve its path. Result : persistent oscillation.

24. SURPRISE : Beware of Backup Policies 2 0 3 1 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 4 4 0 4 2 0 4 3 0 Becomes BAD GADGET if link (4, 0) goes down. BGP is not robust : it is not guaranteed to recover from network failures.

25. PRECARIOUS 1 0 2 1 2 0 1 0 2 1 0 2 0 3 4 5 6 5 3 1 0 5 6 3 1 2 0 5 3 1 2 0 6 3 1 0 6 4 3 1 2 0 6 3 1 2 0 4 3 1 0 4 5 3 1 2 0 4 3 1 2 0 3 1 0 3 1 2 0 As with DISAGREE, this part has two distinct solutions This part has a solution only when node 1 is assigned the direct path (1 0). Has a solution, but can get “trapped”

26. What is to be done? Static Approach Inter-AS coordination Automated Analysis of Routing Policies (This is very hard). Dynamic Approach Extend BGP with a dynamic means of detecting and suppressing policy-based oscillations? These approaches are complementary

27. Research papers on SPP “An Analysis of BGP Convergence Properties” Timothy G. Griffin, Gordon Wilfong “Policy Disputes in Path Vector Protocols” Timothy G. Griffin, F. Bruce Shepherd, Gordon Wilfong SIGCOMM’99 INFOCOM’00 ICNP ‘99 Model BGP, show static analysis is hard Define Stable Paths Problem and develop sufficient condition for “sanity” Dynamic solution based on histories “A Safe Path Vector Protocol” Timothy G. Griffin, Gordon Wilfong “Stable Internet Routing without Global Coordination” Lixin Gao, Jennifer Rexford SIGMETRICS’00 Show that if certain guidelines are followed, then all is well. Rule: Do not forward route advertisements from peers or Providers to other peers or providers.

28. Outline of the Today’s class An Introduction to BGP BGP and the Stable Paths problem Convergence of BGP in the real world Conclusions and Open Issues

29. Convergence in the real-world? [Labovitz99] Experimental results from two year study which measured 150,000 BGP faults injected into peering sessions at several IXPs Found –Internet averages 3 minutes to converge after failover –Some multihomed failovers (short to long ASPath) require 15 minutes

30. Problems with Distance Vector Distance vector protocols (e.g. RIP) suffer routing table loops –Counting-to-infinity –Routing table loops –Bouncing problem BGP uses path vector to “solve” problems seen with RIP and other Bellman-Ford derived protocols

31. Counting to Infinity A B 2 B2 R1+2=3 A2R1A2R1 R 1 R5+2=7 2+3=5 R7+2=9

32. Taming Infinity Routing Information Protocol (RIP) solved counting to infinity problem by re-defining infinity. –Added speedups: poison reverse, split horizon, triggered updates. –Strictly increasing O(N) ASPath limits “infinity” to the width of the Internet (an ASPath through all your neighbors) –Monotonically increasing –Upper bound?

33. BGP Convergence Example R AS0 AS1 AS2 AS3 *B Rvia AS3 B R via AS1,AS3 B R via AS2,AS3 *B Rvia AS3 B R via AS0,AS3 B R via AS2,AS3 *B Rvia AS3 B R via AS1,AS3 B R via AS2,AS3 AS0AS1AS2 *** *B R via 203 *B R via 031 B R via 103

34. N > 4? AS1673 AS237 AS5696 AS2497 AS1239 AS6453 AS701 AS2914 AS6461 AS5000 AS6113 AS1 2914 237 1 5696 237 1239 5696 237 2497 5696 237 701 6461 5696 237 6461 5696 237 237 5696 237 5000 237 6113 2914 237 1673 5696 237 6453 1239 5696 237

35. The Problem with BGP If we assume 1.unbounded delay on BGP processing and propagation 2.Full BGP mesh BGP peers 3.Constrained shortest path first selection algorithm BGP is O(N!), where N number of default- free BGP speakers There exists possible ordering of messages such that BGP will explore all possible ASPaths of all possible lengths

36. BGP and RIP RIP precisely monotonically increasing. Can explore metrics (1…N) BGP monotonically increasing. Multiple (N!) ways to represent a path metric of N. BGP “solved” RIP routing table loop problem by making it exponentially worse… 2117 5696 2129 2117 1 5696 2129 2117 2041 3508 3508 4540 7037 1239 5696 2129 2117 1 2041 3508 3508 4540 7037 1239 5696 2129 2117 2041 3508 3508 4540 7037 1239 6113 5696 2129 2117 1 2041 3508 3508 4540 7037 1239 6113 5696 2129

37. BGP Best Case What is the best we can expect from BGP? What is the best we can expect from BGP? Implementation of MinRouteAdver timer leads to 30 second rounds Implementation of MinRouteAdver timer leads to 30 second rounds Time complexity is O(n-3)*30 seconds Time complexity is O(n-3)*30 seconds State/Computational complexity O(n) State/Computational complexity O(n) At its best, BGP performs as well as RIP2 (but uses exponentially more memory in the process) At its best, BGP performs as well as RIP2 (but uses exponentially more memory in the process)

38. MinRouteAdver Minimum interval between successive updates sent to a peer for a given prefix –Allow for greater efficiency/packing of updates –Rate throttle Applied only to announcements (at least according to BGP RFC) Applied on (prefix destination, peer) basis, but implemented on (peer) basis

39. MinRouteAdver 30*(N-3) delay due to creation mutual dependencies. Provide proof that N-3 rounds necessarily created during bounded BGP MinRouteAdver convergence Rounds due to –Ambiguity in the BGP RFC and lack receiver loop detection –Inclusion of BGP withdrawals with MinRouteAdver (in violation of RFC)

40. Conclusions Internet routing has serious convergence problems –Result 1 [Griffin et al.]: BGP does not satisfy the stable paths problem. –Result 2 [Rexford et al.]: If every AS follows a set of guidelines then Internet routing should not have convergence problems. –Result 3 [Labovitz et al.]: An extensive measurement study shows that Internet convergence can be in the order of several minutes.

41. Open issues? Convergence analysis (lower,upper) bounds are very weak are “worst-worst” case scenarios. Can we design a cleaner protocol that has provably good convergence properties? –What about link-state routing? Should we really care about convergence? –Routes to popular prefixes are stable [IMC03]