1. Inter-domain Routing Protocol
2007

2. Overview An Introduction to BGP BGP and the Stable Paths problem
Convergence of BGP in the real world
The End-to-End Effects of Internet Path Selection

3. The Internet: Zooming In
ASes: Independently owned & operated commercial entities
Autonomous Systems (ASes)
BGP
Abilene
Comcast
CMU
(PSC)
AT&T
Cogent
IGPs
(OSPF, etc)
All ASes are not equal

4. Internet Routing Internet organized as a two level hierarchy
First level – autonomous systems (AS’s)
AS – region of network under a single administrative domain
AS’s run an intra-domain routing protocols
Distance Vector, e.g., RIP
Link State, e.g., OSPF
Between AS’s runs inter-domain routing protocols, e.g., Border Gateway Routing (BGP)
De facto standard today, BGP-4

5. Inter-domain Routing basics
Internet is composed of over AS’s
All AS’s must cooperate to ensure reachability
BGP = Border Gateway Protocol
Policy-Based routing protocol
De facto inter-domain routing protocol of Internet
Relatively simple protocol, but…
Extremely complex configuration flexibility: entire world can see, and be impacted by, configuration mistakes
Many open research problems in policy, scalability, failover, configuration, correctness, security.

6. BGP Basics Use TCP - AS’s exchange
Border Gateway Protocol (BGP), based on Bellman-Ford path vector
AS’s exchange reachability information through their BGP routers, only when routes change
BGP routing information – a sequence of AS’s indicating the path traversed by a route
General operations of a BGP router:
Learns multiple paths
Picks best path according to its AS policies
Install best pick in IP forwarding tables

7. BGP Operations (Simplified)
AS1
Establish session on
TCP port 179
BGP session
Exchange all
active routes
AS2
While connection
is ALIVE exchange
route UPDATE messages
Exchange incremental
updates

8. Various Types of ISPs
Tier-1 ISPs have “default-free” routing tables (i.e., they don’t have any default routes), and typically have global reachability information. There are a handful of these today (about five or so).
Tier 1 ISP
Tier 1 ISP
Tier 2
Tier 2
Tier 2
Tier 3
Tier 2: Regional/National
Tier 3: Local

9. AS relationships Very complex economic landscape. Simplifying a bit:
Transit: “I pay you to carry my packets to everywhere” (provider-customer)
Peering: “For free, I carry your packets to my customers only.” (peer-peer)
Technical defn of tier-1 ISP: In the “default-free” zone. No transit.
Note that other “tiers” are marketing, but convenient. “Tier 3” may connect to tier-1.

10. Customers and Providers
IP traffic
provider
customer
customer
Customer pays provider for access to the Internet

11. The “Peering” Relationship
Peers provide transit between
their respective customers
Peers do not provide transit
between peers
Peers (often) do not exchange $$$
peer
customer
provider
traffic
allowed
traffic NOT
allowed

12. Peering Provides Shortcuts
customer
provider
Peering also allows connectivity between
the customers of “Tier 1” providers.

13. Peering Wars Peer Don’t Peer Reduces upstream transit costs
Can increase end-to-end performance
May be the only way to connect your customers to some part of the Internet (“Tier 1”)
You would rather have customers
Peers are usually your competitors
Peering relationships may require periodic renegotiation
Peering struggles are by far the most
contentious issues in the ISP world!
Peering agreements are often confidential.

14. Architecture of Dynamic Routing
OSPF
BGP
AS 1
EIGRP
IGP = Interior Gateway
Protocol
AS 2
Metric based: OSPF, IS-IS, RIP, EIGRP (cisco)
EGP = Exterior Gateway Protocol
Policy based: BGP

15. AS-Path Sequence of AS’s a route traverses
Used for loop detection and to apply policy
AS-3
AS-4
/16
/16
AS-5
AS-2
/16
/16 AS-2 AS-3 AS-4
AS-1
/16 AS-2 AS-3
/16 AS-2 AS-5

16. Four Types of BGP Messages
Open : Establish a peering session.
Keep Alive : Handshake at regular intervals.
Notification : Shuts down a peering session.
Update : Announcing new routes or withdrawing previously announced routes.
announcement =
prefix + attributes values

17. Two Types of BGP Neighbor Relationships
External Neighbor (eBGP) in a different Autonomous Systems
Internal Neighbor (iBGP) in the same Autonomous System
AS1
iBGP is routed using Interior Gateway Protocol (IGP) such as OSPF/IS-IS
eBGP
iBGP
AS2

18. iBGP Peers Must be Fully Meshed
iBGP is needed to avoid routing loops within an AS
Injecting external routes into IGP does not scale and causes BGP policy information to be lost
BGP does not provide “shortest path” routing
eBGP update
iBGP updates
iBGP neighbors do not announce
routes received via iBGP to other iBGP neighbors.

19. Important BGP attributes
Local PREF
Local preference policy to choose “most” preferred route
Multi-exit Discriminator (MED)
Which peering point to choose?
Import Rules
What route advertisements do I accept?
Export Rules
Which routes do I forward to whom?

20. Route Selection Summary
Highest Local
Preference
Enforce relationships
Shortest ASPATH
Lowest MED
i-BGP < e-BGP
Traffic engineering
Lowest IGP cost
to BGP egress
Throw up hands and
break ties
Lowest router ID

21. Import Routes provider route customer route peer route ISP route From

22. Export Routes provider route peer route customer route ISP route To
From
provider To
peer To
peer To
customer To
customer
filters
block

23. Overview An Introduction to BGP BGP and the Stable Paths problem
Convergence of BGP in the real world
The End-to-End Effects of Internet Path Selection

24. What Problem is BGP solving?
Underlying problem
Distributed means of
computing a solution
Shortest Paths
RIP, OSPF, IS-IS X?
BGP
Having an X can
aid in the design of policy analysis algorithms& heuristics
aid in the analysis and design of BGP and extensions
help explain some BGP routing anomalies
provide a fun way of thinking about the protocol
Our
focus

25. Q : How simple can X get? A: The Stable Paths Problem (SPP)
2 1 0
2 0 2 5
An instance of the SPP : 2
graph of nodes and edges
node 0, called the origin
for each non-zero node, a set or permitted paths to the origin—this set always contains the “null path”
ranking of permitted paths at each node—null path is always least preferred
4 2 0
4 3 0 4 1 3
3 0
1 3 0
1 0
most preferred …
least preferred (not null) 1
When modeling BGP : nodes represent BGP speaking border routers, and 0 represents a node originating some address block

26. A Solution to a Stable Paths Problem2
A solution is an assignment of
permitted paths to each node
such that
2 1 0
2 0 5 2
node u’s assigned path is either the null path or is a path uwP, where wP is assigned to node w and {u,w} is an edge
in the graph,
each node is assigned the highest ranked path among those consistent with the paths assigned to its neighbors.
4 2 0
4 3 0 4 1 3
3 0 1
1 3 0
1 0
A Solution need not represent a shortest path tree, or a spanning tree.

27. Example: SHORTEST1
2 0
2 1 0
1 0
1 3 0 1 2
4 3 0
4 2 0 4 3
3 0

28. Example: SHORTEST1 (Solution)
2 0
2 1 0
1 0
1 3 0 1 2
4 3 0
4 2 0 4 3
3 0

29. Example: SHORTEST2
2 0
2 1 0
1 0
1 3 0 1 2
4 2 0
4 3 0 4 3
3 0

30. Example: SHORTEST2 (Solution)
2 0
2 1 0
1 0
1 3 0 1 2
4 2 0
4 3 0 4 3
3 0

31. Example: GOOD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 3 0
4 2 0 4 3
3 0

32. Example: GOOD GADGET (Solution)
2 1 0
2 0
1 3 0
1 0 1 2
4 3 0
4 2 0 4 3
No another solution:
1 chooses (1 0)
2 chooses (2 1 0)
3 chooses (3 0)
4 chooses (4 3 0)
3 0

33. A Stable Paths Problem may have multiple solutions
1 2 0
1 0
1 2 0
1 0
1 2 0
1 0 1 1 1 2 2 2
2 1 0
2 0
2 1 0
2 0
2 1 0
2 0
DISAGREE
First solution
Second solution

34. Example: NAUGHTY GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 3 0
4 2 0 4 3
3 0

35. Example: NAUGHTY GADGET (Solution 1)
2 1 0
2 0
1 3 0
1 0 1 2
4 3 0
4 2 0 4
3 0 3

36. Example: NAUGHTY GADGET (Solution 2)
2 1 0
2 0
1 3 0
1 0 1 2
4 3 0
4 2 0 4
3 0 3

37. SPP explains possibility of BGP divergence
BGP is not guaranteed to converge to a stable routing. Policy inconsistencies can lead to “livelock” protocol oscillations.
See “Persistent Route Oscillations in Inter-domain Routing” by K. Varadhan, etc. ISI report, 1996
The SPP view :
Solvable
Can Diverge
If you can’t solve this problem, then you don’t understand the protocol
must diverge
must converge

38. Example: NAUGHTY GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 3 0
4 2 0 4 3
3 0

39. That was one round of oscillation!
Example: BAD GADGET 1 2 4 3
2 1 0
2 0
1 3 0
1 0
4 2 0
4 3 0
3 0
That was one round of oscillation!

40. Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)

41. Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 2 chooses (2 1 0) …

42. Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 4 chooses ( ) …

43. Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 3 chooses (3 0) …

44. Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 4 chooses (4 3 0) …

45. Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 1 chooses (1 3 0) …

46. Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 2 chooses (2 0) …

47. Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 4 chooses (4 2 0) …

48. That was one round of oscillation!
Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 3 chooses ( ) …
That was one round of oscillation!

49. That was one round of oscillation!
Example: BAD GADGET
2 1 0
2 0
1 3 0
1 0 1 2
4 2 0
4 3 0 4 3
3 0
2 chooses (2 0)
4 chooses (4 2 0)
3 chooses ( )
1 chooses (1 0)
… 1 chooses (1 0) …
That was one round of oscillation!

50. BAD GADGET : No Solution1 2 4 3
2 1 0
2 0
1 3 0
1 0
4 2 0
4 3 0
3 0
In a BGP-like protocol
each node local decisions
at least one node can always improve its path
Result:
persistent oscillation

51. SURPRISE : Beware of Backup Policies1 2 3
2 1 0
2 0
1 3 0
1 0
3 0 4
4 0
4 2 0
4 3 0
BGP is not robust :
it is not
guaranteed
to recover from
network failures.
Becomes BAD GADGET
if link (4, 0) goes down !

52. PRECARIOUS Has a solution, but can get “trapped” 4 1 3 6 5 2 1 2
1 2 0
1 0
2 1 0
2 0
3 1 0 3 6 5
This part has a solution only
when node 1 is assigned the
direct path (1 0).
As with DISAGREE, this part
has two distinct solutions

53. Theoretical Results
The problem of determining whether an instance of stable paths problem is solvable is NP-complete
Shortest path route selection is provably safe

54. What is to be done? Inter-AS coordination
Static
Approach
Dynamic
Approach
Extend BGP with
a dynamic means of
detecting and suppressing
policy-based oscillations?
Inter-AS
coordination
Automated Analysis
of Routing Policies
(This is very hard).
These approaches are complementary

55. Overview An Introduction to BGP BGP and the Stable Paths problem
Convergence of BGP in the real world
The End-to-End Effects of Internet Path Selection

56. Convergence in the real-world?
[Labovitz99] Experimental results from two year study which measured 150,000 BGP faults injected into peering sessions at several ISPs
Found:
Internet averages 3 minutes to converge after failover
Some multihomed failovers (short to long ASPath) require 15 minutes

57. BGP Convergence Example
AS0
AS1
AS2
AS3
*B R via AS3
B R via AS0,AS3
B R via AS2,AS3
B R via AS0,AS3
B R via AS2,AS3
B R via AS1,AS3 *
*B R via 203
*B R via 013
B R via 103

58. Convergence Result
If we assume
unbounded delay on BGP processing and propagation
Full BGP mesh BGP peers
Constrained shortest path first selection algorithm
Convergence time of BGP is O(N!), where N number of BGP speakers
There exists possible ordering of messages such that BGP will explore all possible ASPaths of all possible lengths

59. Overview An Introduction to BGP BGP and the Stable Paths problem
Convergence of BGP in the real world
The End-to-End Effects of Internet Path Selection

60. End-to-end effects of Path Selection
Goal of study: Quantify and understand the impact of path selection on end-to-end performance
Basic metric
Let X = performance of default path
Let Y = performance of best path
Y-X = cost of using default path
Technical issues
How to find the best path?
How to measure the best path?

61. Approximating the best path
Key Idea
Use end-to-end measurements to extrapolate potential alternate paths
Rough Approach
Measure paths between pairs of hosts
Generate synthetic topology – full NxN mesh
Conservative approximation of best path
Question: Given a selection of N hosts, how crude is this approximation?

62. Methodology For each pair of end-hosts, calculate:
Average round-trip time
Average loss rate
Average bandwidth
Generate synthetic alternate paths (based on long-term averages)
For each pair of hosts, graph difference between default path and alternate path

63. Courtesy:
Stefan Savage

64. Courtesy:
Stefan Savage

65. Courtesy:
Stefan Savage

66. Courtesy:
Stefan Savage

67. Why Path Selection is imperfect?
Technical Reasons
Single path routing
Non-topological route aggregation
Coarse routing metrics (AS_PATH)
Local policy decisions
Economic Reasons
Disincentive to offer transit
Minimal incentive to optimize transit traffic