1. Stable Internet Routing Without Global Coordination Jennifer Rexford Princeton University http://www.cs.princeton.edu/~jrex Joint work with Lixin Gao, Michael Schapira, and Yi Wang

2. What is an Internet?  A “network of networks” –Networks run by different institutions  Autonomous System (AS) –Collection of routers run by a single institution  ASes have different goals –Different views of which paths are good  Interdomain routing is what reconciles those views –To compute end-to-end paths through the Internet Wonderful problem setting for game theory and mechanism design

3. An Open Question Can we have all three? Under what conditions? Evolvable Protocols (under-specified, programmable) Autonomy (autonomous parties, with different economic objectives) Global Properties (stability, scalability, reliability, security, managability, …) ?

4. Autonomous Systems (ASes) 1 2 3 4 5 6 7 Client Web server Path: 6, 5, 4, 3, 2, 1

5. Interdomain Routing: Border Gateway Protocol  ASes exchange info about who they can reach –Destination: block of IP addresses (an “IP prefix”) –AS path: sequence of ASes along the path  Policies configured by the AS’s network operator –Path selection: which of the paths to use? –Path export: which neighbors to tell? 1 23 d “I can reach d” “I can reach d via AS 1” data traffic

6. Interdomain Routing Convergence Challenges  Must scale –Address blocks: 300,000 and growing –Autonomous Systems: around 35,000  Must support flexible policy –Path selection: which path your AS wants to use –Path export: who can send packets through your AS  Must converge, and quickly –Routing convergence can take several minutes –… and the system doesn’t necessarily converge at all! Goal: Guaranteed convergence of the global routing system with purely local control.

7. Stable Paths Problem (SPP) Model  Model of routing policy –Each AS has a ranking of the permissible paths  Model of path selection –Pick the highest-ranked path consistent with neighbors  Flexibility is not free –Global system converges slowly, or not at all –Depending on the way the ASes rank their paths 1 2 d 1 d 2 3 d 2 d 3 1 d 3 d 1 3 2 d

8. Conflicting Policies Cause Convergence Problems 0 1 2 3 1 2 0 1 0 2 3 0 2 0 3 1 0 3 0 Pick the highest-ranked path consistent with your neighbors’ choices. Only choice! Top choice! Only choice! Better choice! Only choice! Better choice!

9. Global Control is Not Workable  Create a global Internet routing registry –Keeping the registry up-to-date would be difficult  Require each AS to publish its routing policies –ASes may be unwilling to reveal BGP policies  Check for conflicting policies, and resolve conflicts –Checking for convergence problems is NP-complete –Link/router failure may result in an unstable system Need a solution that does not require global coordination.

10. Think Globally, Act Locally  Key features of a good solution –Flexibility: allow diverse local policies for each AS –Privacy: do not force ASes to divulge their policies –Backwards-compatibility: no changes to BGP –Guarantees: convergence even when system changes  Restrictions based on AS relationships –Path selection rules: which route you prefer –Export policies: who you tell about your route –AS graph structure: who is connected to who

11. Customer-Provider Relationship  Customer pays provider for access to the Internet –Provider exports its customer’s routes to everybody –Customer exports provider’s routes only to downstream customers d d provider customer provider Traffic to the customerTraffic from the customer advertisements traffic

12. Peer-Peer Relationship  Peers exchange traffic between their customers –AS exports only customer routes to a peer –AS exports a peer’s routes only to its customers peer Traffic to/from the peer and its customers d advertisements traffic

13. Hierarchical AS Relationships  Provider-customer graph is a directed, acyclic graph –If u is a customer of v and v is a customer of w –… then w is not a customer of u u v w

14. Valid and Invalid Paths 2 3 1 d 4 5 6 7 8 Provider-Customer Peer-Peer Valid paths: “1 2 d” and “7 d” Invalid path: “5 8 d” Valid paths: “6 4 3 d” and “8 5 d” Invalid paths: “6 5 d” and “1 4 3 d”

15. Act Locally, Prove Globally  Route export –Do not export routes learned from a peer or provider –… to another peer or provider  Global topology –Provider-customer relationship graph is acyclic –E.g., my customer’s customer is not my provider  Route selection –Prefer routes through customers –… over routes through peers and providers  Guaranteed to converge to unique, stable solution

16. Our Local Path Selection Rules  Classify routes based on next-hop AS –Customer routes, peer routes, and provider routes  Rank routes based on classification –Prefer customer routes over peer and provider routes  Allow any ranking of routes within a class –E.g., can rank one customer route higher than another –Gives network operators the flexibility they need  Consistent with traffic engineering practices –Customers pay for service, and providers are paid –Peer relationship contingent on balanced traffic load

17. Solving the Convergence Problem  Result –Safety: guaranteed convergence to unique stable solution –Inherent safety: holds under failures and policy changes  Definitions –System state: current best route at each AS –Activating AS: re-do decision based on neighbor choices  Sketch of (constructive) proof –Find an activation sequence that leads to a stable state –Any “fair” sequence (eventually) includes this sequence

18. Rough Sketch of the Proof  Two phases –Walking up the customer-provider hierarchy –Walking down the provider-customer hierarchy 2 3 1 d 4 5 6 7 8 Provider-Customer Peer-Peer

19. Economic Incentives Affect Protocol Behavior  ASes already follow our rules, so system is stable –High-level argument »Export and topology assumptions are reasonable »Path selection rule matches with financial incentives –Empirical results »BGP routes for popular destinations are stable for ~10 days »Most instability from failure/recovery of a few destinations  ASes should follow our rules to make system stable –Need to encourage operators to obey these guidelines –… and provide ways to verify the network configuration –Need to consider more complex relationships and graphs

20. Playing One Condition Off Against Another  All three conditions are important –Path ranking, export policy, and graph structure  Allowing more flexibility in ranking routes –Allow same preference for peer and customer routes –Never choose a peer route over a shorter customer route  … at the expense of stricter AS graph assumptions –Hierarchical provider-customer relationship (as before) –No private peering with (direct or indirect) providers Peer-peer

21. Extension to Backup Relationships  Backups: more liberal export policies, and different ranking –The motivation is increased reliability –…but ironically it may cause routing instability!  Generalize rule: prefer routes with fewest backup links –Need to maintain a count of the # of backup links in the path backup path primary provider backup provider failure Backup Provider backup path failure peer provider Peer-Peer Backup [RFC 1998]

22. Results Hold Under More Complex Scenarios  Complex AS relationships –AS pair with different relationship for different prefixes –AS pair with both a backup and a peer relationships –AS providing transit service between two peer ASes  Stability under changing AS relationships –Customer-provider to/from peer-peer –Customer-provider to/from provider-customer

23. Extensions of the Work  Influence of AS relationships on BGP convergence –Algebraic framework and design principles for policy languages –Fundamental limits on relaxing the assumptions  Application of the idea to internal BGP inside an AS –Sufficient conditions for iBGP convergence inside an AS –“What-if” tool for traffic engineering inside an AS  AS-level analysis of the Internet topology –Inference of AS relationships and policies from routing data –Characterization of AS-level topology and growth  Practical applications of knowing AS relationships –Analyzing your competitors’ business relationships –Identifying BGP routes that violate export conditions

24. A Case For Customized Route Selection  ISPs usually have multiple paths to the destination  Different paths have different properties  Different neighbors may prefer different routes Bank VoIP provider School Most secure Shortest latency Lowest cost

25. Neighbor-Specific Route Selection  A node has a ranking function per neighbor is node i’s ranking function for neighbor node j.

26. Stability Conditions for NS-BGP  Surprisingly, NS-BGP improves stability! –Neighbor-specific selection is more flexible –Yet, requires less restrictive stability conditions  “Prefer customer” assumption is not needed –Choose any “permissible” route per neighbor  That is, need just two assumptions –No cycle of provider-customer relationships –Do not export routes learned from one peer/provider to other peers/providers

27. Why Do Weaker Conditions Work?  An AS always tells its neighbor a route –If it has any route that is permissible for that neighbor 0 1 2 3 1 2 0 1 0 2 3 0 2 0 3 1 0 3 0

28. Deploying NS-BGP  An AS can deploy NS-BGP alone –Without upgrading their routers –Without coordinating with all their neighbors  Three aspects to the solution –Disseminating extra BGP routes –Customized route selection –Directing traffic from ingress to egress  Can be done exploiting existing mechanisms –Designed for Virtual Private Networks (VPNs)

29. Disseminating Extra BGP Routes  Advertising more than one BGP route –Route distinguisher feature for VPNs –Multiple internal BGP sessions –ADD-PATHs extensions to internal BGP

30. Customized Route Selection  Multiple virtual routing and forwarding tables –Cisco: Virtual Routing and Forwarding (VRF) –Juniper: Virtual Router D: (red path): R6 D: (blue path): R7 R3’s forwarding table (FIB) entries

31. Directing Traffic from Ingress to Egress  Tunnels from ingress to egress –IP-in-IP tunneling –MPLS ?

32. Customized Route Selection  Customized route selection as a service –Select a different best route for different neighbors  Different menu options –Cheapest route (e.g., “prefer customer”) –Best performing routes, or most secure routes –Routes that avoid undesirable ASes (e.g., censorship)  Nice practical features of NS-BGP –An individual AS can deploy NS-BGP alone –… without compromising global stability!

33. Conclusions  Avoiding convergence problems –Hierarchical of provider-customer relationships –Export policies based on commercial relationships –(Path ranking based on AS relationships)  Salient features –No global coordination (locally implementable) –No changes to BGP protocol or decision process –Guaranteed convergence, even under failures –Guidelines consistent with financial incentives

34. References Related to This Talk  “The stable paths problem and interdomain routing” –Tim Griffin, Bruce Shepherd, and Gordon Wilfong –http://portal.acm.org/citation.cfm?id=508332http://portal.acm.org/citation.cfm?id=508332  “Stable Internet routing without global coordination” –Lixin Gao and Jennifer Rexford –http://www.cs.princeton.edu/~jrex/papers/sigmetrics00.long.pdfhttp://www.cs.princeton.edu/~jrex/papers/sigmetrics00.long.pdf  “Inherently Safe Backup Routing with BGP” –Lixin Gao, Tim Griffin, and Jennifer Rexford –http://www.cs.princeton.edu/~jrex/papers/infocom01.pdfhttp://www.cs.princeton.edu/~jrex/papers/infocom01.pdf  “Neighbor-Specific BGP: More flexible routing policies while improving global stability” –Yi Wang, Michael Schapira, and Jennifer Rexford –http://www.cs.princeton.edu/~jrex/papers/nsbgp_sigmetrics09.pdfhttp://www.cs.princeton.edu/~jrex/papers/nsbgp_sigmetrics09.pdf

35. Other Related Research Papers  Inherently Safe Backup Routing with BGP –http://www.cs.princeton.edu/~jrex/papers/infocom01.pdfhttp://www.cs.princeton.edu/~jrex/papers/infocom01.pdf  Design Principles of Policy Languages for Path Vector Protocols –http://conferences.sigcomm.org/sigcomm/2003/papers/p61-griffin.pdfhttp://conferences.sigcomm.org/sigcomm/2003/papers/p61-griffin.pdf  Implications of Autonomy for the Expressiveness of Policy Routing –http://conferences.sigcomm.org/sigcomm/2005/paper-FeaBal.pdfhttp://conferences.sigcomm.org/sigcomm/2005/paper-FeaBal.pdf  Meta-routing –http://conferences.sigcomm.org/sigcomm/2005/paper-GriSob.pdfhttp://conferences.sigcomm.org/sigcomm/2005/paper-GriSob.pdf  An Algebraic Theory of Interdomain Routing –http://portal.acm.org/citation.cfm?id=1103561http://portal.acm.org/citation.cfm?id=1103561  Searching for Stability In Interdomain Routing –http://www.cs.yale.edu/homes/schapira/PID808559.pdfhttp://www.cs.yale.edu/homes/schapira/PID808559.pdf

36. 36 Related Papers With Game Theory  Interdomain Routing and Games –http://www.cs.huji.ac.il/~mikesch/routing_games-full.pdfhttp://www.cs.huji.ac.il/~mikesch/routing_games-full.pdf  Rationality and Traffic Attraction: Incentives for Honest Path Announcements in BGP –http://ccr.sigcomm.org/online/?q=node/395http://ccr.sigcomm.org/online/?q=node/395  Incentive-Compatible Interdomain Routing –http://cs-www.cs.yale.edu/homes/jf/FRS.pdfhttp://cs-www.cs.yale.edu/homes/jf/FRS.pdf  Mechanism Design for Policy Routing –http://cs-www.cs.yale.edu/homes/jf/FSS.pdfhttp://cs-www.cs.yale.edu/homes/jf/FSS.pdf  The Complexity of Game Dynamics: BGP Oscillations, Sink Equlibria, and Beyond –http://www.cs.berkeley.edu/~alexf/papers/fp08.pdfhttp://www.cs.berkeley.edu/~alexf/papers/fp08.pdf  Specification Faithfulness in Networks with Rational Nodes –http://www.eecs.harvard.edu/econcs/pubs/podc04.pdfhttp://www.eecs.harvard.edu/econcs/pubs/podc04.pdf  Distributed Algorithmic Mechanism Design –http://cs-www.cs.yale.edu/homes/jf/AGTchapter14.pdfhttp://cs-www.cs.yale.edu/homes/jf/AGTchapter14.pdf  Partially Optimal Routing –http://www.stanford.edu/~rjohari/pubs/por.pdfhttp://www.stanford.edu/~rjohari/pubs/por.pdf

37. 37 Background on Interdomain Economics  http://drpeering.net/a/Home.html http://drpeering.net/a/Home.html  http://www.fcc.gov/Bureaus/OPP/working_papers/oppwp32.pdf http://www.fcc.gov/Bureaus/OPP/working_papers/oppwp32.pdf  http://www.potaroo.net/papers/1999-6-peer/peering.pdf http://www.potaroo.net/papers/1999-6-peer/peering.pdf  http://www.cisco.com/en/US/about/ac123/ac147/ac174/ac201/about_ cisco_ipj_archive_article09186a00800c83a5.html http://www.cisco.com/en/US/about/ac123/ac147/ac174/ac201/about_ cisco_ipj_archive_article09186a00800c83a5.html  http://www.cisco.com/en/US/about/ac123/ac147/ac174/ac200/about_ cisco_ipj_archive_article09186a00800c8900.html http://www.cisco.com/en/US/about/ac123/ac147/ac174/ac200/about_ cisco_ipj_archive_article09186a00800c8900.html  http://www.vjolt.net/vol3/issue/vol3_art8.html http://www.vjolt.net/vol3/issue/vol3_art8.html