Internet Routing (COS 598A) Today: Interdomain Traffic Engineering Jennifer Rexford Tuesdays/Thursdays 11:00am-12:20pm

Outline Border Gateway Protocol –BGP protocol –BGP policies –BGP decision process BGP traffic engineering for outbound traffic –Predicting effects of policy changes –Identifying good policy changes What about inbound traffic? –AS prepending and MEDs –Inter-AS negotiation

Interdomain Routing: Border Gateway Protocol ASes exchange info about who they can reach –IP prefix: block of destination IP addresses –AS path: sequence of ASes along the path Policies configured by the AS’s operator –Path selection: which of the paths to use? –Path export: which neighbors to tell? “ /24: path (2,1)” “ /24: path (1)” data traffic

Components of BGP BGP protocol –Definition of how two BGP neighbors communicate –Message formats, state machine, route attributes, etc. –Standardized by the IETF Policy specification –Flexible language for filtering and manipulating routes –Indirectly affects the selection of the best route –Varies across vendors, though constructs are similar BGP decision process –Complex sequence of rules for selecting the best route –De facto standard applied by router vendors –Being codified in a new RFC for BGP coming soon

BGP Protocol: BGP Sessions Establish session on TCP port 179 Exchange all active routes Exchange incremental updates AS1 AS2 While connection is ALIVE exchange route UPDATE messages BGP session

BGP Protocol: Update Messages Update messages –Advertisement New route for the prefix (e.g., /24) Attributes such as the AS path (e.g., “2 1”) –Withdrawal Announcing that the route is no longer available Numerous BGP attributes –AS path –Next-hop IP address –Local preference –Multiple-Exit Discriminator –…

BGP Policy: Applying Policy to Routes Import policy –Filter unwanted routes from neighbor E.g. prefix that your customer doesn’t own –Manipulate attributes to influence path selection E.g., assign local preference to favored routes Export policy –Filter routes you don’t want to tell your neighbor E.g., don’t tell a peer a route learned from other peer –Manipulate attributes to control what they see E.g., make a path look artificially longer than it is

BGP Policy: Influencing Decisions Best Route Selection Apply Import Policies Best Route Table Apply Export Policies Install forwarding Entries for best Routes. Receive BGP Updates Best Routes Transmit BGP Updates Apply Policy = filter routes & tweak attributes Based on Attribute Values IP Forwarding Table Apply Policy = filter routes & tweak attributes Open ended programming. Constrained only by vendor configuration language

BGP Decision Process: Path Selection on a Router Routing Information Base –Store all BGP routes for each destination prefix –Withdrawal message: remove the route entry –Advertisement message: update the route entry Selecting the best route –Consider all BGP routes for the prefix –Apply rules for comparing the routes –Select the one best route Use this route in the forwarding table Send this route to neighbors

BGP Decision Process: Multiple Steps Highest local preference –Set by import policies upon receiving advertisement Shortest AS path –Included in the route advertisement Lowest origin type –Included in advertisement or reset by import policy Smallest multiple exit discriminator –Included in the advertisement or reset by import policy Smallest internal path cost to the next hop –Based on intradomain routing protocol (e.g., OSPF) Smallest next-hop router id –Final tie-break

BGP Decision Process in Action “(2, 1)” “(3, 4, 1)” “(2, 1)” But, what if the path “(3,4,1)” would be better?

Manipulating Policy to Move the Traffic Assign local preference to… –Prefer one neighbor over another for a prefix –Prefer certain AS paths over others Router configuration languages –Specifying rules for setting local-pref attribute –“if path(3, *, 1), then local-pref=110” –“else, local-pref=100” Allow policy to over-ride shortest AS path –Indirect way of making one path look better or worse than another –Main way to do BGP traffic engineering today

BGP Traffic Engineering

Limitations of intradomain traffic engineering –Alleviating congestion on edge links –Making use of new or upgraded edge links –Influencing choice of end-to-end path Extra flexibility by allowing changes to BGP policies –Direct traffic toward/from certain edge links –Change the set of egress links for a destination

BGP Modeling for Traffic Engineering Predict effects of changes to import policies –Inputs: routing, traffic, and configuration data –Outputs: flow of traffic through the network Topology BGP policy configuration eBGP routes Offered traffic BGP routing model Flow of traffic through the network

Steps in the Model Effects of import policy on the routes –Inputs: BGP routes, and import policies –Output: BGP routes modified by policies Set of best routes for the network –Inputs: BGP routes modified by policies –Output: set of best routes (max local-pref, min AS path,…) Best route per router –Inputs: set of BGP routes, and intradomain costs –Outputs: best route (closest egress) per prefix per router Link-level paths –Inputs: best route per router, and intradomain costs –Outputs: link-level shortest path(s) from entry to egress

But, Hard to Select Good Import Policy Changes Can’t enumerate all choices –Many eBGP sessions –Around 100K prefixes –Highly programmable policies Risk of creating side-effects –Overhead of BGP routing changes –Unpredictable behavior of others Vulnerability to changes –New eBGP routes from neighbors –Shifts in the traffic volume per prefix

Traffic Engineering Guidelines Predictability –Ensure the BGP decision process is deterministic –Assume that BGP updates are (relatively) stable Outbound traffic (import policy, local preference) –Easier to control how traffic leaves the network –Cooperate with neighbor ASes for inbound traffic Limit overhead introduced by routing changes –Minimize frequency of changes to routing policies –Limit number of prefixes affected by changes Limit impact on how traffic enters the network –Avoid new routes that might change neighbor’s mind –Select route with same attributes, or at least path length

Measurement Data Analysis of peering links –Links connecting AT&T to other large providers –Relatively small # of high-end routers BGP routing tables –Log daily output of “show ip bgp” command –Extract BGP routes for each prefix Cisco Netflow –Collect continuous flow-level measurement –Aggregate the traffic by prefix –Focus on outbound traffic to peers

Controlling the Scale Destination prefixes –More than 90,000 destination prefixes Don’t want to have per-prefix routing policies –Small fraction of prefixes contribute most of the traffic Focus on the small number of heavy hitters –Define routing policies for selected prefixes Routing choices –About 27,000 unique “routing choices” Help in reducing the scale of the problem –Small fraction of “routing choices” contribute most traffic Focus on the very small number of “routing choices” –Define routing policies on common attributes

Avoid Impacting Downstream Neighbors Will traffic volume change???

Predictable Routing Change Predictability –Do not change the route sent to downstream neighbor –Focus on prefixes where all “best” routes are identical –Neighbors do not even receive a new BGP advertisement! Example application –Multiple links to same peer, with one congested link –Assign lower local-pref at that link for some prefixes Empirical results –83.5% of prefixes have shortest AS paths that are all identical (same next-hop AS, same AS path, etc.) –These prefixes are responsible for 45% of the traffic –Plenty of scope to move traffic in a predictable fashion

Semi-Predictable Routing Change Semi-predictability –Do not change the length of the AS path sent to neighbor –Neighbors receive new advertisement with same length –(Hopefully) they still make the same routing decision Example application –Need to move some traffic from one peer to another –Find prefixes with “best” paths via both neighbors –Assign lower local-pref at some links for some prefixes Empirical results –10-15% of prefixes have shortest paths with 2 next hops –These prefixes contribute 35-40% of the traffic –Potential to move traffic in a semi-predictable fashion

Influence of AS Path Length AS path length –Plays a significant role in the BGP decision process –All “best” routes must have the same AS path length –10% of prefixes have choices with different path lengths An idea: a more flexible approach –Possible to disable consideration of AS path length, and incorporate AS path length in local-pref assignment –E.g., treat paths of length 3 and 4 as equally good AS prepending by other ASes –Inflating AS path length by adding fake hops –E.g., “ ” instead of “701 80” –18% of routes had some form of AS prepending

Inbound Traffic

Why Inbound Traffic is Hard to Manage Other ASes decide how to send to you –Destination-based routing –Other ASes decide which path to take –Based on their own policies AS 2 doesn’t know how AS 1 will send traffic toward p p

AS Prepending Artificial inflate AS path length –Prepend your own AS in the path –E.g., turn “3 4 5” into “ ” –Hope to make the path less attractive 1 3 “ ” “3 4 5”

Multiple Exit Discriminator (MED) Tell your neighbor what you want –MED attribute to indicate receiver preference –Decision process picks route with smallest MED –Can use MED for “cold potato” routing –But, have to get your neighbor to accept MEDs 1 3 “3 4 5” with MED=2 “3 4 5” with MED=1

Inter-AS Negotiation Better to cooperate? –Negotiate where to send –Inbound and outbound –Mutual benefits But, how to do it? –What info to exchange? –How to prioritize the many choices? –How prevent cheating? Open research territory Customer A Customer B multiple peering points Provider A Provider B Early-exit routing

Next Time: Multi-Homed Customers Two papers –“A Measurement-Based Analysis of Multi-homing” (SIGCOMM’03) –“Optimizing Cost and Performance for Multihoming” (SIGCOMM’04) Reviews of both papers –Brief summary of the paper –Reasons to accept the paper –Reasons to reject the paper –Suggestions for future research directions