1. CSci5221: Inter-Domain Routing Convergence Issues and Improvements 1 Inter-Domain Routing Convergence Issues, Impacts and Improvements Inter-Domain Routing Convergence Issues – Path exploration, misconfiguration, timers, … Impact on Data Plane Data Forwarding – transient forwarding loops – transient loss of reachability, even during “failure recovery” Possible improvements – EPIC: limit effect of path exploration – R-BGP: pro-active “back-up” BGP paths (optional) Readings: Please do the required readings

2. CSci5221: Inter-Domain Routing Convergence Issues and Improvements: Part 1 2 Inter-Domain (BGP) Routing Convergence Issues BGP can converge very slowly –Some measurement studies show it may take up to 15 min! Many potential causes ! –Some key factors Routers use MRAI (min. route advertisement interval) to damp and ensure “routing stabilities” BGP uses path vector, route updates processed and propagated one router at a time! Routers may “change mind” when receiving new updates –This may lead to so-called the “path exploration” problem that may lead to slow convergence

3. 3 Delayed BGP Convergence and Path Exploration AS Path prevents loops..... do not help with convergence – “path exploration”, after failure, delays convergence When a link fails (or is repaired), routers “go through” a sequence of paths before selecting a “ converged ” path Inherent dependencies in advertised “path vectors” –Router’s best path is an extension of a neighbors’ best path. –Which extends a best path from one of its own neighbors. –And so on…… We will look at the path exploration problem in more detail later (slide 30 onwards); first we look at two measurement studies!

4. CSci5221: Inter-Domain Routing Convergence Issues and Improvements: Part 1 4 Classification of Routing Events Tup: a previously unavailable route is announced. –This represents a route repair Tdown: a previously unavailable route is withdraw – This represents a route failure Tshort: an active route with a longer ASPath is replaced with a new route announcement with a shoter ASPath –This represents both a route repair and failover Tlong: an active route with a shorter ASPath is replaced with a new route announcement with a longer ASPath –This represents both a route failure and failover

5. CSci5221: Inter-Domain Routing Convergence Issues and Improvements: Part 1 5 Labovitz Study: Experimental Results

6. CSci5221: Inter-Domain Routing Convergence Issues and Improvements: Part 1 6 Labovitz Study: Experimental Results (cont’d)

7. CSci5221: Inter-Domain Routing Convergence Issues and Improvements: Part 1 7 Labovitz Study: End-to-End Measurements

8. Wang et al Study [W+06]: More In-depth Analysis of Impact of BGP Dynamics on Data Plane Motivation: Real-time services have made high availability of end- to-end Internet paths of paramount importance. –low packet loss rate, low delay, high network availability, and fast reaction time Internet path failures are widespread –can last as long as 10 minutes Degraded end-to-end path performance is correlated with routing dynamics.

9. Questions to Be Answered How routing changes result in degraded end-to-end path performance? What kinds of routing dynamics cause the degraded end-to-end performance? How factors such as topological properties, or routing policies affect performance degradation?

10. Wang et al Study [W+06] Study end-to-end performance under realistic topologies. Investigate several metrics to characterize the end-to-end loss, delay, and out-of-order packets. Characterize the kinds of routing changes that impact end-to-end path performance. Analyze the impact of topology, routing policies, MRAI timer and iBGP configurations on end-to-end path performance.

11. Methodology A multi-homed prefix –BGP Beacon prefix: 192.83.230.0/24 Controlled Routing Changes –Failover events: Beacon changes from the state of having both providers to the state of having only a single provider. –Recovery events: Beacon changes from the state of having a single provider for connectivity to the state of having both providers. Provider 1 Beacon Provider 2Provider 1Provider 2 Provider 1Provider 2 Beacon Failover event Recovery event

12. Active Probing From 37 PlanetLab hosts to the Beacon host (a host within the Beacon prefix) –Back-to-back traceroutes –Back-to-back pings –UDP probing (50msec interval) Data plane performance metrics Internet Provider 2 Beacon host Provider 1 host B host A host C metrics Active probing traceroutepingUDP probing Pack lossx Delayx Out-of-orderx

13. Representativeness Connectivity of Destination Prefixes – SS: Single-homed prefixes via a single upstream link – SM: Single-homed prefixes via multiple upstream links – MS: Multi-homed prefixes via a single upstream link – MM: Multi-homed prefixes via multiple upstream links Routing tables from one tier-1 ISP on January 15, 2006 classSSSMMSMM percentage48%6%29%17%

14. Representativeness (contd.) Multi-homed destination prefixes ISP 2ISP 3 ISP 1 destination Customer link Peer link

15. Representativeness (contd.) Multi-homed destination prefixes with multi- upstream links ISP 2 ISP 1 ISP 2

16. Packet Loss Loss burst: consecutive UDP probing packets lost during a routing change event. FailoverRecovery

17. Correlating Packet Loss with Routing Failures ICMP replies –temporary loss of reachability (!N or !H) –forwarding loops (exceeded TTL) Routing failures –temporary loss of reachability and transient routing loops Correlate loss bursts with ICMP messages –time window [-1 sec, 1 sec] Underestimate the number of loss bursts due to routing failures –missing ICMP packets.

18. An Example planet02.csc.ncsu.edu experiences packet loss on July 30, 2005

19. Loss Bursts due to Routing Failures Failover events: 76% packets lost Recovery events: 26% packets lost FailoverRecovery

20. Loss Burst Length loss burst length can be as long as 480 packets for failover events, and 180 packets for recovery events Loss burst length Failover eventsRecovery events

21. Multiple Loss Bursts Multiple loss bursts after the injection of a withdrawal message or an announcement. Failover Recovery

22. Location of Lost Bursts (Failover events) Location of the first lost bursts caused by routing failures. From ISP 2’s BGP updates: –Routing failures do occur and are not visible from ICMP messages due to short duration. From another AS’s BGP updates, and Oregon RouteView –Routing failures are cascaded to other ASes. ClassISP 1ISP 2Other tier1Non tier-1 Failover 192%05%3% Failover 209%73%18%

23. Location of Lost Bursts (Recovery events) Location of the first lost bursts caused by routing failures. BGP updates from ISP 2 –12 withdrawals over 724 recovery events ClassISP 1ISP 2Other tier1Non tier-1 Failover 190%N/A0%10% Failover 2N/A0%59%41%

24. How Routing Failures Occur (Failover)? R1 Beacon R4R5 R6 R2R3 Provider 1Provider 2 Peer link 0 0 2 0 0 0 1 0 0 0 Prefer-customer routing policy: routes received from a provider’s customers are always preferred over those received from its peers. AS 0 Customer link

25. How Routing Failures Occur (Failover)? (contd.) R1 Beacon R4R5 R6 R2R3 Provider 1 Provider 2 Peer link 0 0 2 0 0 0 1 0 0 0 R7R9 Provider 3 2 0 1 0 2 0 No-valley routing policy: peers do not transit traffic from one peer to another. AS 0 Peer link R8

26. How Routing Failures Occur? (Recovery) R1 R2R4 R3 0 Beacon path (0) Path (0) Withdraw (2 0) 5. R1 regains its connection to the Beacon 1. Path 0 => R3 recovery. 2. R3 sends the path to R2 3. R2 sends a withdrawal to R1 4. R3 sends the recovery path to R1 iBGP constraint: a route received from an iBGP router cannot be transited to another iBGP router Provider 1 Provider 2 AS 0

27. Summary of Findings During failover and recovery events –Routing changes impact packet loss significantly. –Multiple loss bursts are observed in 60% of events. –Routing changes can lead to long packet round-trip delays and reordering. Loss bursts explained by routing failures last longer than those unidentified ones. Loss bursts caused by forwarding loops last longer than those caused by loop-free routing failures.

28. Conclusions During failover and recovery events –routing failures contribute to end-to-end packet loss significantly. Routing policies, iBGP configuration and MRAI timer values play a major role in causing packet loss during routing events. Degraded end-to-end performance can be experienced by a diverse set of hosts when there is a routing change. Accommodate routing redundancy may eliminate majority of identified path failures.

29. Recap: Impact of BGP Dynamics => loss of connectivity on data plane! Route changes cause up to 30% packet loss for more than 2 minutes [Labovitz00] Routing events can cause multiple loss bursts, and one loss burst can last for up to 20s [Wang06] Popular and unpopular prefixes experience losses due to BGP dynamics [Wang05] VoIP outages are highly correlated with BGP updates [Kushman06]

30. 30 Delayed BGP Convergence and Path Exploration AS Path prevents loops..... do not help with convergence – “path exploration”, after failure, delays convergence When a link fails (or is repaired), routers “go through” a sequence of paths before selecting a “ converged ” path Inherent dependencies in advertised “path vectors” –Router’s best path is an extension of a neighbors’ best path. –Which extends a best path from one of its own neighbors. –And so on……

31. 31 What is Path Exploration (cont’d) When a link fails, a set of dependent paths becomes invalid (or obsolete) Removed one by one from the system –Router selects and propagates it –Receives withdrawal –Selects next best path (possibly invalid) –Receive withdrawal, repeat till no more invalid paths –Each route update (for same prefix) delayed by MRAI timer at each router MRAI: minRouteAdvertiseInterval 1997 Merit/Labovitz Study: 15 minutes to converge!

32. 32 Path Exploration Example 01 2 3 4 5 6 7 10 310 210 4210 6310 74210 75210 76310 9 8 Network in a steady state

33. 33 Path Exploration Example (cont’d) 01 2 3 4 5 6 7 10 310 210 4210 6310 74210 75210 76310 75210 76310 W 9 8

34. 34 Path Exploration Example (cont’d) Paths 75210 and 76310 both contain the “problem edge” 10. 2 additional messages to force 7 to flush “bad paths”. Number of “spurious messages” increases with the “richness” of connectivity ….. 01 2 3 4 5 6 7 7210 74210 7510 75210 72510 …… 8 9

35. Toy Example: Worst Case Scenario 35

36. 36 Impact of Path Exploration (cont’d) Delays a router from picking valid, alternate paths –Have to first go through all invalid paths Large scale packet losses in a short duration –Core routers process millions of packets a second Minimum convergence time is Ω(Lh) –L is “longest” path of the network –L >> D: D is “diameter” of the network – h is message processing time at a node Experimental results (in 1997) show up to 15 minutes to converge! –terrible for “streaming”, network-gaming, etc

37. 37 Quantifying Impact Position in the hierarchy affects the amount of path exploration seen

38. 38 Causes for Path Exploration Invalid paths are selected, propagated, then withdrawn –Routers waste time processing “stale information” –Delay convergence to valid, perhaps less preferred, alternate paths Key Issue: How to distinguish invalid paths from valid paths –Difficult in BGP: AS Paths -- high level, abstract

39. 39 Naive Solutions Fail TAG withdrawals: When router generates withdrawal, tag it with cause/location 01 2 3 4 5 6 7 WDRAW: (2,1) failed 74210 75210 76310

40. 40 Naïve Solutions Fail (cont’d) 01 3 4 5 6 7 AS Paths do not describe (or reflect) internal AS topology. When an internal edge fails, which AS Path affected? –[10] or [210]? 2

41. 41 Naïve Solutions Fail (cont’d) Link between 3.2 and 6.1 fails. 6.1 generates a withdrawal and tags with Should 6.3 remove all paths containing ? 01 2 4 5 7 AS 3 AS 6 6.1 6.2 3.2 3.36.3

42. 42 AS PATHS: High Level Connectivity AS 81 AS 217 AS 1239 AS 3 AS 11536 AS 217 and AS 3 receive the same AS PATH [11536 1239 81] Underlying physical paths are disjoint

43. 43 EPIC --- Our Solution Exploit Path dependencies to Invalidate Paths To avoid path exploration: –When link fails, a set of dependent paths becomes invalid –All the dependent paths must be removed from the system Dependent paths cannot be described using only AS Paths AS Paths are annotated with additional information (forward edge sequence numbers - fesn) –Can capture path dependencies –Can distinguish valid and invalid paths

44. 44 Forward Edge Sequence Numbers When AS Path being advertised to an external AS neighbor, include fesn of “ forward ” external edge fesn = edge identifier + sequence number AS XAS Y Edge

45. 45 Forward Edge Sequence Numbers (cont’d) Defined per destination, for every AS-AS edge When AS X sends a route to AS Y, the fesn (X:Y, n) is attached If route already has a previously attached fesn, new fesn is prepended to it -- fesnList AS X AS Z AS Y AS W (X:Y, n) (X:Z, m) (X:Y, n)

46. 46 Identifying Invalid Paths: External Failure Invalid path-stem is (exactly) fesnList received at AS 3 1 2 3 4 5 6

47. 47 Identifying Invalid Paths: Internal Failure Invalid path-stem is (exactly) fesnList exported from AS 3 to AS 4 1 2 3 4 5 6

48. 48 fesn Management When a link fails, its fesn does not change –Same value carried in withdrawals When is repaired: –AS X increments the sequence number –Subsequent route announcements carry “updated” fesn So a larger fesn always corresponds to “newer” information

49. 49 fesnList Propagation 01 2 3 4 5 6 7 7 7 3 14 7 3 10 14 11 [4210] {(0:1, 7)(1:2, 7)(2:4, 14)(4:7, 11)} [0] {(0:1, 7)} [10] {(0:1, 7)(1:3, 3)} [10] {(0:1, 7)(1:2, 7)} Same AS Path, distinct fesnLists

50. 50 fesnList Propagation 01 2 3 4 5 6 7 4210(4:7, 11) (2:4, 14) (1:2, 7) (0:1, 7) 5210(5:7, 10) (2:5, 14) (1:2, 7) (0:1, 7) 6310(6:7, 3) (3:6, 7) (1:3, 7) (0:1, 7) After the routes are processed at all nodes Routing Table at AS 7

51. 51 Invalidating Paths upon Failure When router generates a withdrawal: –fesnList of withdrawn route (“path stem”) is attached to the withdrawal When router receives a withdrawal: 1.Invalidates all routes containing fesnList 2.Selects a new best path 3.If best path has changed, it sends new best route to its neighbors, and the withdrawal is piggybacked. 4.If no valid path, only withdrawal is forwarded.

52. 52 fesn – Key Properties Sequence number is monotonic --- new events will have higher values Imposes a partial ordering on the fesnLists. Old information can be easily detected, and discarded Allows compact, correct description of invalid paths –If path1.fesnList  path2.fesnList, then path2 “depends” on path1 –fesnList in a withdrawal captures all obsolete paths

53. 53 What about Multiple Edges? AS 81 AS 217 AS 1239AS 3 AS 11536

54. 54 Handling Multiple Edges Each edge is associated with a minor fesn –Contrast with major fesn for “logical” AS-AS edge All edges between ASes share the same major fesn, but have distinct minor fesn ’ s Minor fesn is incremented with corresponding edge –major fesn incremented only if all edges are affected

55. 55 Minor fesn’s Minor fesn ’ s are only used between adjacent ASes. All routers in AS 6 include minor fesn in route updates. When the updates exported externally (to AS 7) minor fesn is removed AS 3 AS 6 01 2 4 5 7 6.1 6.2 3.2 3.36.3 7 (11) 7 (13) common major fesn distinct minor fesn’s

56. 56 EPIC Properties No router will select an invalid path after receiving any update triggered by a single failure event No router will select an invalid path after receiving at least one update triggered by each of a set of multiple failure events Achieves optimal bounds for a path vector protocol –Routers may still explore paths –But these paths are all valid

57. 57 EPIC Performance (vs BGP) Time(L-2)Δ(D-1)h Messages(L-2)(|E| -1)|E| - 1 Time(L’+D’–1) ΔD’(h+Δ) Messages(L’+D’-1)(|E’|-1)(|E’|-1)D’(h+Δ)/Δ Fail Down Fail Over BGPEPIC

58. 58 Implementation and Deployment Can be incrementally deployed –[fesnList] implemented as a new “transitive” route attribute –Withdrawals need to carry an [fesnList] Additional memory required to store fesnLists associated with routes –  2 x |AS Path table| –Very little processing overhead Simple string matching required to process fesnLists

59. Resilient BGP (R-BGP) Main Problem to be addressed –During convergence (e.g., during path exploration), BGP can temporarily lose connectivity (no “valid” path), or create transient forwarding loops Key Ideas – Instead of solving BGP slow convergence problem, focus on preventing transient loss of connectivity or transient forwarding loops –Similar to IP FRR, proactively pre-compute a “failover” AS path Remember each AS is not single node! Also need to deal with impact of BGP dynamics within each AS (using pre-configured “tunnels” between primary and fail-over “next-hop” egress points)

60. Transient “No (Valid) Path” Problem All of Pop’s neighbors are using him to get to MIT  Nobody tells Pop an alternate path to MIT! All of Pop’s neighbors are using him to get to MIT  Nobody tells Pop an alternate path to MIT! MIT AT&T. Mom Pop Sprint.

61. Transient No-Path Problem MIT AT&T. Mom Pop Sprint. Link Down LOSS! When “link” to MIT fails, Pop knows no alternate path to MIT - Pop drops all his packets to MIT as well as those from Mom and AT&T When Pop’s BGP router is notified of the failure, Pop will announce a path withdraw to both Mom and AT&T.

62. Transient No-Path Problem MIT AT&T. Mom Pop Sprint. Link Down After receiving Pop’s withdraw, Mom and AT&T will stop sending packets to Pop - Mom has no idea why Pop withdraws his path to MIT - Mom may switch to use the alternate path via AT&T to MIT (path exploration!) -Some routers in AT&T may decide to switch to Mom! Note that AT&T is a big AS, not all routers learn the same info at same time! Packets may also be dropped/caught in transient loops due to during Mom/AT&T path exploration withdraw

63. Transient No-Path Problem MIT AT&T. Mom Pop Sprint. Link Down Transient No-Path causes temporary disconnectivity Eventually AT&T realizes that it has to use the alternate path via Sprint Announce “new” path to Mom and Pop Mom and Pop use the new path to reach MIT - packets now flow through AT&T and Sprint to MIT! announcement

64. How Do We Solve Pop’s Problem? Tell Pop a failover path before the link fails – rather than after it, as is often the case in current BGP Also note that Mom’s alternate path (Mom  AT&T  Pop  MIT) is no good –i.e., can’t protect her from Pop->MIT link failure –however, as long as AT&T knows a “fail-over” path, Mom is in fact is Okay ! AT&T does know a “good” fail-over path (AT&T  Sprint  MIT) that protects it from Pop  MIT failure How do we pro-actively select “good” fail-over path

65. No Loss ! AT&T advertises to Pop “AT&T  Sprint  MIT” as a failover path Link Fails: Pop immediately sends traffic on failover path; once PoP’s BGP learns the failure, sends a BGP withdrawal to AT&T and Mom MIT AT&T. Mom Pop Sprint. Link Down Help Pop Help You! Internally, AT&T tunnels Pop’s traffic toward Sprint -Why does AT&T need an (internal) tunnel within its network? What about Mom?

66. AT&T A B BGP/IGP convergences take time! packets from Pop to MIT may reroute by Pop to A even before A learns the path withdraw! Can’t distinguish “normal” and “fail-over” traffic packets to MIT that originate within AT&T will still be forwarded to A and then to Pop during the convergence period Pop re-routes “fail-over” traffic back to A of AT&T, which then tunnel them to B! - A (implicitly) uses “interface-specific” forwarding! Tunnel Why Tunnel within an AS?

67. Should AT&T advertise a failover path to every neighbor (i.e., Sprint & Mom)? –Not necessary Neither Sprint or Mom is currently on the “default” (i.e., best) path from AT&T to MIT AT&T already advertises its “default” (best) path (AT&T  Pop  MIT) to Sprint and Mom! –Also excessive overhead ( bandwidth, processing and memory!) Constraint/Rule: An AS advertises one and only one path to each neighbor! If a neighbor is the next-hop AS on the default (best) path, we can advertise a fail-over path (if available) ! - otherwise, it learns the best path as in (normal) BGP Constraint/Rule: An AS advertises one and only one path to each neighbor! If a neighbor is the next-hop AS on the default (best) path, we can advertise a fail-over path (if available) ! - otherwise, it learns the best path as in (normal) BGP How Do We Select “Good” Fail-over Paths?

68. R-BGP Goal: Staying Connected If Pop’s link to destination fails and After convergence, Pop will have a path to destination Pop should have a failover path to the destination when the link fails AT&T. Sprint. Destination Pop X

69. Three Key Mechanisms of R-BGP Mechanism 1 – Fail-over Paths: Each AS advertises to its next-hop AS (of default or primary path) a failover path that, among the available paths, is the one most disjoint from the primary path. Mechanism 1 – Fail-over Paths: Each AS advertises to its next-hop AS (of default or primary path) a failover path that, among the available paths, is the one most disjoint from the primary path. Mechanism 2 – Route Cause Information (RCI): Each BGP update includes RCI which can be used to invalidate other (currently available) paths that are also affected by the same failure event Mechanism 2 – Route Cause Information (RCI): Each BGP update includes RCI which can be used to invalidate other (currently available) paths that are also affected by the same failure event Hence R-BGP assumes that an EPIC-like mechanism is also incorporated and deployed in BGP !

70. Three Key Mechanisms (cont’d) Mechanism 3a – Use Old Primary Paths: When left without a usable primary path, an AS continues to forward packets along its old primary path! Mechanism 3a – Use Old Primary Paths: When left without a usable primary path, an AS continues to forward packets along its old primary path! Mechanism 3b – Ensuring Convergence: An AS stops forwarding internally originated traffic along withdrawn primary paths or failover paths when explicit withdrawals have been received from all neighbors. An AS delays sending a withdrawal to a neighbor until it is sure it will not offer this neighbor a valley-free path at convergence. Mechanism 3b – Ensuring Convergence: An AS stops forwarding internally originated traffic along withdrawn primary paths or failover paths when explicit withdrawals have been received from all neighbors. An AS delays sending a withdrawal to a neighbor until it is sure it will not offer this neighbor a valley-free path at convergence.

71. Dest AT&T. Mom Pop x Sis The most disjoint path protects against more link failures! Fail-over Path: Why Most Disjoint

72. Link Down Pop withdraws primary path from AT&T AT&T moves to second best path through Mom AT&T withdraws failover from Pop because its next-hop is Mom, and sends Pop Mom’s path Mom may do the same, switch to AT&T ! - We may end up with a transient forwarding loop between Mom and AT&T, eventually get dropped! The problem is voided if Pop’s withdrawal indicates the down link MIT AT&T. Mom Pop Sprint. Outage! Why RCI Is Needed

73. Path via AT&T is invalidated by RCI; Mom has no path! (a) Without Mechanism 3, Mom drops packets when Pop withdraws his path! Outage! Why Mechanism 3 Link Down MIT AT&T. Mom Pop Sprint. Sis (b) Mom doesn’t send Sis a withdrawal before she hears from AT&T! - otherwise Sis will have no path! (b) Mom doesn’t send Sis a withdrawal before she hears from AT&T! - otherwise Sis will have no path!

74. Correctness of R-BGP Theorem 1: If any AS using the down link will have a path after convergence, then R-BGP guarantees that the AS immediately above the down link knows a failover path when the link fails. Theorem 2: All ASs that will eventually learn a valley- free path to the destination are guaranteed no BGP- caused packet loss during convergence

75. Some Issues with R-BGP Some Implementation Details: –Use of “virtual interfaces” and “virtual connections” between two (neighboring) ASes (e.g., AT&T and Pop) to distinguish between normal and fail-over traffic Main Drawbacks: –R-BGP does not necessarily protect against node (router) failures Suppose Pop has one backbone router which connects to both AT&T and Mom What happens when this router fails (as opposed to link from Pop to MIT fails), which would cause both routes to AT&T and Mom to be invalid, thus withdrawn? -Using RCI, Mom will has no valid path; Mechanism 3 doesn’t help! -Correctness of R-BGP hinges on RCI (e.g., EPIC), which is currently not available in BGP -Fail-over traffic may use “non-valley-free” paths (e.g., Mom  Pop  AT&T  Sprint  MIT)

76. Conclusion BGP loses connectivity even when the graph is connected R-BGP solves this problem by advertising a single failover path downstream –BGP’s convergence stays unaffected Simple and powerful, but have limitations Other alternative solutions: – e.g., [Y+08] “Reliable Inter-domain Routing Through Multiple Complementary Routing Processes” Yong Liao; Lixin Gao; Roch Guerin; Zhi-Li Zhang. ReArch’08 Workshop, Madrid, 2008 Does not use RCI; ensure always use policy-compliant paths