1. The Impact of Policy and Topology on Internet Routing Convergence NANOG 20 October 23, 2000
Abha Ahuja
InterNap
Craig Labovitz
Microsoft Research
*In collaboration with Roger Wattenhofer, Srinivasan Venkatachary, Madan Musuvathi

2. Background
In NANOG 19, we showed BGP exhibits poor convergence behavior:
Measured convergence times of up to 20 minutes for BGP path changes/failures
Factorial (N!) theoretic upper bound on BGP convergence complexity (explore all paths of all possible lengths)
Open question: In practice, what topological and policy factors impact convergence delay ?

3. This Talk
Goal: Understand BGP convergence behavior under real topologies/policies
Given a physical topology and ISP policies, can we estimate the time required for convergence?
Do convergence behaviors of ISPs differ?
How does steady-state topology compare to paths explored during failure?
Can we change policies/topology to improve BGP convergence times?

4. Experiments
Analyzed secondary paths between between 20 source/destination AS pairs
Inject and monitor BGP faults
Survey providers to determine policies behind paths
To provide intuition, we will focus on faults injected into three ISPs at Mae-West
Observed faults via fourth ISP (in Japan)
Three ISPs roughly map onto tier1, tier2, tier3 providers
Results from these three ISPs representative of all data

5. Comparing ISP Convergence Latencies
CDF of faults injected into three Mae-West providers and observed at Japanese ISP
Significant variations between providers
Not related to geography

6. Observed Fault Injection Topologies
ISP 4
Steady State
ISP 1 R1
FAULT
ISP 2 R2
FAULT
Steady State
ISP 3 R3
FAULT
Steady State
MAE-WEST
In steady-state, topologies between ISP1, ISP2, ISP3 similar – all direct BGP peers of ISP4. Does not explain variation on previous slide…

7. Factors Impacting BGP Propagation
Topology and policy impact graph (usually DAG)
Each AS router adds between 0-45 seconds of MinRouteAdver Delay
iBGP/Route Reflector
MinRouteAdver and path race conditions affect which routes chosen as backup routes
iBGP D C B A

8. ISP1-ISP4 Paths During Failure
96% Average: 92 (min/max 63/140) seconds
Announce AS4 AS5 AS (44 seconds)
Withdraw (92 seconds)
4% Average: 32 (min/max 27/38) seconds
Withdraw (32 seconds)
Steady State
FAULT R1
ISP 1
Only one back up path (length 3)

9. ISP2-ISP4 Paths During Failure
Vagabond P4
ISP 10
ISP 11
ISP 12
ISP 13
63% Average: 79 (min/max 44/208) seconds
AS4 AS5 AS2 (35 seconds)
Withdraw (79 seconds)
7% Average: 88 (min/max 80/94) seconds
Announce AS4 AS5 AS (33 seconds)
Announce AS4 AS6 AS5 AS2 (61 seconds)
Withdraw (88 seconds)
7% Average: 54 (min/max 29/9) seconds
Withdraw (54 seconds)
23% Other P2
ISP 5 P3
ISP 6
Steady State
FAULT R2
ISP 2

10. ISP3-ISP4 Paths During Failure
36% Average: 110 (min/max 78/135) seconds
Announce AS4 AS5 AS (52 seconds)
Withdraw (110 seconds)
35% Average: 107 (min/max 91/133) seconds
Announce AS4 AS1 AS3 (39 seconds)
Announce AS4 AS5 AS3 (68 seconds)
Withdraw (107 seconds)
2% Average: (min/max 120/142)
Announce AS4 AS5 AS8 AS7 AS (27) Announce AS4 AS5AS9 AS8 AS7 AS3 (86)
Withdraw (140 seconds)
27% Other P6 P7 P4 P5
ISP 7
ISP 9
ISP 8
ISP 1 P3 P2
ISP 5
Steady State
FAULT R3
ISP 3

11. Why the Different Levels of Complexity?
Provider relationship taxonomy
Transit relationships
customer/provider
customer sends their customer routes
provider sends default-free routing info (or default)
Peer relationships
Bilateral exchange of customer routes
Back-up transit
peer relationship becomes transit relationship based on failure
These relationships constrain topology (no N! states) and determine number of possible backup paths

12. Convergence in the Real World3
customer
peer 1 2 X 4 5
Longest path:
Possible paths for node 3:
2 1 x
4 2 1 x
( x)
Possible paths for node 4:
2 1 x
3 2 1 x
5 2 1 x

13. Convergence in the Real World
Hierarchy eliminates some states 3
customer
peer 2 X 1 4 5
Longest path:
Tier 1?
Possible paths for node 3:
2 1 x x
Possible paths for node 4:
3 2 1 x
5 2 1 x

14. Policy and Convergence
Strict hierarchical relationships eliminate exploring some extra states
Policy controls the number of possible paths to explore.
But turns out the number of paths does not matter…

15. Relationship Between Backup Paths and Convergence
Longest Observed ASPath Between AS Pair
Convergence related to length longest possible backup ASPath between two nodes

16. So, what does all of this mean for convergence time?
Convergence time is related to the length of the longest path that needs to be explored
Before fail-over, need to withdraw all alternative paths
This is bounded O(n) by length of the longest alternative path in the system
This longest path is related to policy

17. Towards Millisecond BGP Convergence
Three possible solutions
Entirely new protocol
Turn off MinRouteAdver timer
“Tag” BGP updates
Provide hint so nodes can detect bogus state information

18. Further Information
C. Labovitz, R. Wattenhofer, A. Ahuja, S. Venkatachary, “The Impact of Topology and Policy on Delayed Internet Routing Convergence”. MSR Technical Report (number pending). June, 2000.
C. Labovitz, A. Ahuja, A. Bose, F. Jahanian, “Internet Delayed Routing Convergence.” To appear in Proceedings of ACM SIGCOMM. August, 2000.
Send to for more information or to participate in the policy survey