1. An Adaptive Policy Management Approach to BGP Convergence by Selma Yilmaz PhD Examining Committee: Prof. Ibrahim Matta, First Reader (Major Advisor) Prof. John Byers, Second Reader Prof. Assaf Kfoury, Third Reader Prof. Azer Bestavros, Committee Chair Prof. Richard West

2. Border Gateway Protocol (BGP) Is the de facto inter-domain routing protocol of today’s global Internet Is a policy-based routing protocol –allows ASes to share reachability information according to policies Export policy Always share routes with AS1 Import policy Accept routes from AS2 for destination A

3. Border Gateway Protocol (BGP) BGP does not necessarily solve shortest path routing problem –Best path is the path with the highest local preference value assigned by locally defined policies AS1 AS3 AS2 AS5 AS4 May assign higher preference value to path (AS3 AS2 AS1) than path (AS4 AS1)

4. BGP Routing Process Allows to select the routes based on any desired criteria Makes it easier to realize commercial relationships between ASes Ex: Forward data only for paying customers Filter out the paths passing through AS x Apply Export Policies forward, not forward Send BGP UPDATEs to peers Update IP Forwarding Table (FIB) BGP UPDATEs from peers Apply Import Policies accept, reject, set local preferences select best path BGP Decision Process Open-ended programming

5. Problems with Policy-based Routing Collection of locally well-configured policies may cause global conflicts: –It may not be possible to satisfy conflicting policies simultaneously –Causes BGP to diverge ASes exchange routing messages indefinitely First shown by Varadhan et al. [USC Technical Report 1996] Statically checking for BGP convergence property is an NP- complete problem [Griffin et al. Sigcomm 1999] Many solutions proposed to detect and prevent policy conflicts

6. Why is this problem important? BGP is widely deployed in today’s Internet Persistent oscillations leads to –repeated advertising and withdrawing of routes higher processing load –re-running BGP decision process to select the best path –updating routing and forwarding tables endangered scalability –routers may experience severe CPU load, and memory problems makes traffic engineering through an AS very difficult Convergence of BGP must be guaranteed independent of locally selected policies

7. Thesis Contributions A generalized control theoretical framework for BGP convergence is developed The framework is instantiated for recently proposed algorithms Deficiencies of previous solutions are exposed A new dynamic algorithm called “Adaptive Policy Management Scheme (APMS)” is proposed Correctness and convergence analysis of the algorithm is presented APMS implemented in the SSF network simulator, and its performance is compared against other solutions

8. Abstract Model of BGP [Griffin Infocom 2000] Stable Paths Problem (SPP) represents the static semantics of BGP Simple Path Vector Protocol (SPVP) represents the dynamic semantics of BGP is a distributed algorithm solving SPP An SPP is called safe if SPVP always converges

9. Stable Paths Problem (SPP) Network is represented as a simple, undirected graph –Nodes represent BGP routers, edges represent BGP sessions –Node 0 represents destination For each node v, there is a set of permitted paths, P v For each node v, there is a ranking function, λ v Empty path, ε, is a permitted path at each node, and has the lowest rank Paths are simple, i.e. no repeated nodes Example of an SPP instance: 4 31 2 0 210 20 130 10 30 420 430 Most preferred Least preferred

10. A Solution to a Stable Paths Problem (SPP) A solution is an assignment of permitted paths to each node such that –node u’s assigned path is either of the following ε max (λ u ((u w)P w )) among the advertised path P w by w є peers(u) An SPP instance may have multiple solutions –SPVP may diverge no solution – SPVP diverges a unique solution –does not mean that SPVP converges to that solution solvability does not imply safety What is the sufficient condition that will guarantee safety of an SPP specification? 4 31 2 0 210 20 130 10 30 420 430 Safe

11. Dispute Wheel [GW ICNP 99] A wheel of size k For each 1 ≤ i ≤ k: R i is a path from u i to u i+1 (u 1 =u k+1 ) Q i is a permitted path u i R i Q i+1 is a permitted path at u i (Q 1 =Q k+1 ) Q i is less (or equally) preferred than R i Q i+1 at node u i Properties of dispute wheels assuming S is an SPP instance: If S has no DW, then S is safe and robust Lack of DW implies a solution Presence of DW does not imply divergence Divergence due to lack of solution implies DW Divergence due to multiple solutions implies DW RiRi d u1u1 R1R1 u2u2 uiui ukuk RkRk u i+1 Q1Q1 Q2Q2 QiQi Q i+1 QkQk spokes Route preferences of these nodes cause dispute wheel

12. Examples of Stable Paths Problem 0 1 3 2 21 210 20 120 10 0 Solutions: (10)(210) and (20)(210) 0 1 2 Safe, Not Robust No SolutionMultiple Solutions 2 3 130 10 210 20 3420 30 1 0 4 420 430 4 2 3 130 10 210 20 3420 30 1 0 4 40 420 430 Stabilizes on (130)(30)(20)(40) If the link (40) fails, new SPP has “no solution” Unique Solution, May Not Converge 4 1 2 0 1 2 0 312 4 5 6 4310 453120 43120 5310 563120 53120 6310 643120 63120 3 5 6 0 120 10 210 20 2 4 1 310 3120

13. Generalized Control Theoretical Framework Feedback Monitor: Update AdjRibIn Check path for AS loops (path vector property) Apply import policies to decide permitted paths Control Mechanism: Apply import policies to assign Local Preferences Choose best path Check for an indication of divergence If YES, change best path Update locRIB, and export to peers Control Mechanism Best Path Network Update Messages Feedback Monitor Router u

14. Details of Control Theoretical Framework Check the path P for loops: If P contains “u” Update AdjRibIn(w) Apply import policies to see if the path P is permitted at node u Control Mechanism Node u UPDATE message from peer w with path P Compute best path, best(u) Apply import policies to assign Local Preferences Check for an Indication of Divergence Update localRIB and export best(u) yes no Feedback Monitor Control Best Path Selection Change Ranking Function Restrict Usage of Some Paths Re-Compute best path

15. Related Work and Instantiations in the Control Theoretical Framework

16. Gao&Rexford Algorithm [Infocom01] Provides guidelines that guarantees safety of BGP –Use hierarchical structure of the Internet and commercial relationships between ASes to specify local policies Provider-to-customer graph should be acyclic –Paths are classified as provider/customer/peer according to next-hop AS Each AS must prefer customer paths more than provider/peer paths –Route Registry database keeps relationships and verify that guidelines are followed Disadvantages –Static solution –Requires Route Registry –Disallows many paths

17. Gao&Rexford Algorithm [Infocom01] 0 2 3 1 130 10 320 30 210 20 Ex: 0 2 3 1 Cycle involving 1,2,3 will be detected by Route Registry and ASes will be advised to use their shortest AS paths: (10),(20),(30) Assume following provider-to-customer graph:

18. Griffin&Wilfong Algorithm [Infocom00] Proposes carrying dynamically computed history of path change events with Update messages, history Path change event is computed as follows: If node u changes its current path from P old to P new –P new is more preferred than P old, e=(+, P new ) –P old is more preferred than P new, e=(-, P old ) History explains the exact sequence of events leading to the adoption of the current path Cycles in the history corresponds exactly to dispute wheels The path whose adoption creates a cycle is suppressed Disadvantages –History may get very long, may reveal preferences –Cycle in the history is necessary but not sufficient condition –Cannot distinguish temporary and persistent oscillations

19. Griffin&Wilfong with Control Theoretical Framework Check the path P for loops: If the AS path contains “u” Update AdjRibIn(w) Apply import policies to see if the path is permitted at node u Control Mechanism Node u UPDATE message from peer w with path P and history h Compute best path, best B (u), excluding the paths in bad path set Apply import policies to assign Local Preferences Re-compute best path, best B (u), excluding the paths in bad path set Update history Update localRIB and export best B (u) yes no Feedback Monitor Check for an Indication of Divergence Compute path change event, p p=(+, best B (u)) if λ u (best B (u))> λ u (current best path) p=(-, current best path) if λ u (best B (u))< λ u (current best path) Check updated history for loops Control Best Path Selection add best B (u) to bad paths set, B(u)

20. Griffin&Wilfong Periodic Reset Purge bad paths set, B(u) Update each path stored in AdjRibIn by resetting history Control Mechanism Node u Periodic Reset Update localRIB and export best B (u) Feedback Monitor Compute best path, best B (u), excluding the paths in bad path set Apply import policies to assign Local Preferences Compute Path Change Event, p p=(+, best B (u)) if λ u (best B (u))> λ u (current best path) p=(-, current best path) if λ u (best B (u))< λ u (current best path) Set history of bestB(u) to p

21. Griffin&Wilfong [Infocom00] 0 2 3 1 130 10 320 30 210 20 Stabilizes to unreachable destination for all nodes step node best path path assignment 0 1 (10) (+10) 2 (20) (+20) 3 (30) (+30) 1 1 (130) (+130)(+30) 2 (210) (+210)(+10) 3 (320) (+320)(+20) 2 1 (10) (-130)(+320)(+20) 2 (20) (-210)(+130)(+30) 3 (30) (-320)(+210)(+10) 3 1 (130) (+130)(-320)(+210)(+10) 2 (210) (+210)(-130)(+320)(+20) 3 (320) (+320)(-210)(+130)(+30) 4 1 (10) (-130)(+320)(-210)(+130)(+30) 2 (20) (-210)(+130)(-320)(+210)(+10) 3 (30) (-320) (+210)(-130)(+320)(+20) 5 1 epsilon 2 epsilon 3 epsilon May Griffin&Wilfong lead to simultaneous path eliminations? 1 0 3 2

22. Cobb&Musunuri [Globecomm04] Assigns integer costs to the nodes Monotonically increases the cost whenever the new path of a node has lower rank then its previous path If there is divergence, costs grow Costs are included in Update messages Whenever a node has option to improve its current path by choosing a better alternative path P –Checks first if the cost of the next-hop node along P is lower than a threshold –Otherwise, keeps the current path Disadvantages –Aggregates paths through the same node –May lead to simultaneous path rejections –Lowering costs are hard –Lowering costs are suggested to be done periodically without taking any precaution to prevent re-introducing the resolved conflicts

23. Cobb&Musunuri with Control Theoretical Framework Check the path P for loops: If the AS path contains “u” Update AdjRibIn(w) Apply import policies to see if the path is permitted at node u Control Mechanism Node u UPDATE message from peer w with path P and cost c Compute best path, best(u) Apply import policies to assign Local Preferences Update localRIB and export best(u) along with cost of u yes no Feedback Monitor Check for an Indication of Divergence (λ u (best(u)) > λ u ( current path)) and (cost(next(best(u))) ≥ threshold and current path is not epsilon) Update Cost of Node u if ((λ u (current path)> λ u (best(u)) if nextHop(current path)!=nextHop(best(u)) cost(u)=cost(u)+1 if nextHop(current path)==nextHop(best(u)) cost(u)=cost(nextHop(current path(u)) else cost(u)=cost(nextHop(current path(u)) Restrict Usage of Some Paths if (current path is not epsilon) and (cost(next(best(u))) ≥ threshold) do not adopt best(u)

24. Cobb&Musunuri Periodic Reset cost(u)=0 Control Mechanism Node u A command received to reset the cost of node u to 0 Compute best path, best(u) Apply import policies to assign Local Preferences Update localRIB and export best(u) along with cost of u Feedback Monitor Update Cost of Node u if ((λ u (current path)> λ u (best(u)) if nextHop(current path)!=nextHop(best(u)) cost(u)=cost(u)+1 if nextHop(current path)==nextHop(best(u)) cost(u)=cost(nextHop(current path(u)) else cost(u)=cost(nextHop(current path(u))

25. Cobb&Musunuri [Globecomm04] 0 2 3 1 130 10 320 30 210 20 All nodes stabilize to their lowest preferred paths step node count best path 0 1 0 (10) 2 0 (20) 3 0 (30) 1 1 0 (130) 2 0 (210) 3 0 (320) 2 1 1 (10) 2 1 (20) 3 1 (30) 3 1 1 (130) 2 1 (210) 3 1 (320) 4 1 2 (10) 2 2 (20) 3 2 (30) 5 1 won’t use (130) since count(3) ≥ 2 2 won’t use (210) since count(2) ≥ 2 3 won’t use (130) since count(3) ≥ 2 1 0 3 2 May lead to unnecessary path eliminations? Assume threshold=2

26. Motivation for APMS Detect persistent oscillations dynamically For better scalability –Detect paths involved in a policy conflict using only local info –Resolve conflicts locally Each node involved in a conflict observes route flaps –Constantly adopting a path and later abandoning it –Not every advertisement received is changing Safe path There must be more preferred path(s) than the safe path –Make the safe path highest ranked path to stop oscillation –Each node needs to keep local history to detect the flapping paths Count is associated with the paths in the local history –increased with every flap of the path –Distributed algorithm There may be synchronous detection and path rank change –Perform rank change probabilistically

27. Adaptive Policy Management Scheme max_threshold –Due to probabilistic adjustment of path preferences, the conflict may remain unresolved If count> max_threshold, suppress the path. min_threshold To distinguish between temporary and persistent oscillations Each node independently classifies the state of the network by comparing count values against max_threshold and min_threshold time count max_threshold min_threshold Policy conflict free phase Policy conflict avoidance phase Policy conflict control phase

28. Adaptive Policy Management Scheme State of the system: –Path ordering at each node –(Path, Count) pairs in localHistory Count value denotes how many times a path is adopted and later abandoned –Bad path set keeps suppressed paths –peerStability value associated with each peer How many times the path advertised by a peer has changed –The peers with peerStability=1 are stable peers –The paths advertised by stable peers are safe –keepAliveCount associated with each peer Used as an indication of stability If keepAliveCount ≥ ka_threshold for each peer –Node concludes that the system is stabilized –Probabilistically restore local preference values

29. APMS Feedback Monitor (When an Update is Received) peerStability(w)++ keepAliveCount=0 Check the path for loops: If the AS path contains “u” Update AdjRibIn(w) Apply import policies to see if the path is permitted at node u CONTROL MECHANISM Node u UPDATE message from peer w

30. APMS Control Mechanism (When an Update is Received) Compute best path, best B (u), excluding the paths in bad path set If best B (u) is different from the current best path, count(best B (u))++ count(best B (u))>max_threshold count(best B (u))>min_threshold Control Best Path Selection: Policy Conflict Avoidance Phase Change ranking with probability ½ rank(P safe )=1 where P safe is the most preferred safe path best B (u)=P safe reset some states: count(P) for each P in localHistory peerStability(w) for each peer w no yes Control Best Path Selection: Policy Conflict Control Phase Restrict usage of the path badPaths(u)= badPaths(u) U best B (u) re-compute best path, best B (u) reset some states: count(P) for each P in localHistory peerStability(w) for each peer w Update localRIB and export best B (u) no yes Node u Apply import policies to assign Local Preferences

31. APMS Path Rank Restoration When the system stabilizes, there may be some path rank changes that are not contributing to the current state of stability –Policies are placed for a purpose such as traffic engineering, cost, security Must keep them untouched unless they are conflicting Must adapt to every state of the network –conflict free as well as potentially conflicting When the system stabilizes, peers exchange only keepAlive messages –Nodes may use this as an indication of convergence Probe the state for improvement, i.e. restoration, in their current policies Probabilistically restore local preference values –May introduce instability back to system –Use smaller probability, 1/4

32. APMS Path Rank Restoration (When a KeepAlive is Received) keepAliveCount(w)++ Control Mechanism Node u KeepAlive message from peer w Feedback Monitor Stability Check keepAliveCount(v) ≥ ka_threshold for each peer v of u for each peer v of node u for path P in AdjRibIn(v) with probability 1/4 if P was suppressed, remove it from bad paths set if P’s preference has been changed, reset its original local preference reset some states: count(P) for path P in localHistory peerStability(v) keepAliveCount(v) Compute best path, best B (u), excluding the paths in bad path set Update localRIB and export best B (u) yes

33. Handling Transient Oscillations due to Topology Changes If there is a topology change such as link/node failure/recovery Resulting flaps may interfere with diagnosing conflicts –May lead to false positives The system before and after the change have different policy dynamics –New state may be conflict free, local states must be reset Suggest that the node next to the topology change includes extra-information in the resulting Update message – topologyChange helps to reset local state temporarily turn policy conflict detection process off –originator is a list of nodes who adapted to the new topology helps to turn policy conflict detection process on

34. Convergence Analysis of APMS Different path orderings at the nodes specify different states of the network and define different policies Goal: Show that starting with an arbitrary state of the system, the APMS converges to a stable state within a finite number of steps. Use substability property of chosen paths

35. Definitions Conflict free node is a node whose policies are not conflicting with any other node Nonflapping (stable) path P=(v,..,destination) is the best path of a conflict free node, which does not change over time Safe path (u,v)P is a permitted path at node u, where v is a peer of u, and v is a conflict free node and advertising nonflapping path P Conflicting safe alternative node is involved in a policy conflict and has a safe path Conflicting node is involved in a policy conflict, and does not have a safe path

36. Example: Conflicting safe-alternative nodes can stabilize by holding onto their safe paths –realize through rank change Convergence Analysis of APMS Node 1 is a conflicting safe-alternative node with safe path (150) Node 2 is a conflicting safe-alternative node with safe path (250) Node 3 is a conflicting safe-alternative node with safe path (350) Node 4 is a conflicting node Node 5 is a conflict free node with stable path (50) If node 2 changes its path preference to prefer (250) more than (2150): node 2 becomes conflict free node path (250) becomes stable path path (4250) becomes safe path at node 4 ….. 43 4250 4350 34250 350 5 1 2 1350 150 2150 250 0 50

37. Observable Safe Path Path P=(u,v,..,0) is an observable safe path at a conflicting safe alternative node u if none of the nodes along this path observes route flaps due to other conflicts. Innermost Conflict Convergence Analysis of APMS ukuk u k-1 u2u2 u1u1 0 uiui u i-1 u i+1 conflict free nodes involved in conflict C i Path P= may be involved in conflict C i+1 may be involved in conflict C k-1 C i is the innermost conflict along P

38. 3 conflicts with intervening safe paths Convergence Analysis of APMS 0 9 8 3 7 4 2 1 5 6 120 140 560 520 4520 40 2370 20 640 60 780 70 3140 370 970 90 890 80 1 2 3 0 4 7 7 8 9 0 4 5 6 0 2 Conflict I Conflict II Conflict III (60) is observable safe path at node 6 (370) is not an observable safe path Innermost conflict along (370) is Conflict II

39. Theorem: During the execution of the APMS, the size of the set of nodes that are conflict free increases monotonically. Proof: S=set of conflict free nodes S forms a routing tree rooted at the destination, and grows as the nodes in S advertise their chosen paths. By induction show that S grows monotonically: Basis: At the beginning, S={}. Destination is added. Hypothesis: At step k of the execution, assume the size of S is n, and up to this point S grew monotonically. Induction Step: Show that at step (k+1), the size of S will be greater than n. Convergence Analysis of APMS

40. At step (k+1): Case I: (u v)P v is not permitted, then the size of S will stay the same. v pvpv S with n nodes already stabilized to their paths 0 u v advertises P v to u 1 2 0 3 4 130 10 320 30 210 20 4210 Ex: 1 0 3 2 4 S Node 2 advertises (20) to node 4 1 3 2 0

41. Convergence Analysis of APMS At step (k+1): v pvpv S with n nodes already stabilized to their paths 0 u v advertises P v to u 1 2 0 3 4 130 10 320 30 210 20 420 Ex: 1 0 3 2 4 S Node 2 advertises (20) to node 4 1 3 2 0 Case II: u stabilizes on path (u v)P v and then added to S a)u is a conflict free node  its path to destination may have node(s) involved in conflict(s)

42. Convergence Analysis of APMS At step (k+1): v pvpv S with n nodes already stabilized to their paths 0 u v advertises P v to u 2 3 0 1 4 210 20 130 10 3420 30 420 430 Ex: 3 0 1 2 4 S Node 2 advertises (20) to node 4 1 3 2 0 Case II: u stabilizes on path (u v)P v, and then added to S b) u is a conflicting node, and path (u v)P v is node u’s most preferred path 4

43. Convergence Analysis of APMS At step (k+1): Ex: 1 0 3 2 4 S Node 2 advertises (20) to node 4 1 3 2 0 Case III: (u v)P v is permitted at u, but u does not stabilize on this path - (u v)P v is a safe path at node u - u must be conflicting safe alternative node - u performs rank change and stabilizes on (u v)P v - for each conflict there are at least 2 safe alternative nodes, this is the step where they are breaking the conflict 1 2 0 3 4 130 10 320 30 210 20 450 420 6 5 560 50 6420 60 5 6 4 0 2 v pvpv S with n nodes already stabilized to their paths 0 u v advertises P v to u

44. For cases II and III, size of S increases monotonically. What about case I? –If for each node u outside of S, the paths (u v)P v advertised by peers v in S are not permitted, then node u converges to epsilon. –Then all the nodes outside of S will converge to epsilon at this point. –APMS returns with a stable routing tree. After finite number of steps, all nodes will be in S and APMS converges. Convergence Analysis of APMS

45. Advantages of APMS over Related Work 1) Gao&Rexford Algorithm [Infocom01] 2) Griffin&Wilfong Algorithm [Infocom00] 3) Cobb&Musunuri Algorithm [Globecomm04]

46. Gao&Rexford Static solution Requires a global database to keep relationships between ASes – Global authority checks periodically for conformance with guidelines Eliminates lots of paths from the beginning –too restrictive Path elimination is the only means of resolving conflicts Stability of the system is the only goal APMS Dynamic solution Distributed computation Allows ASes to adopt to the current state: conflict free or potentially conflicting Path elimination is not the primary means of removing conflicts Paths are eliminated only during the policy conflict control phase – Helps to limit the number of paths eliminated For the stabilized system, there will be as many paths as possible – better connectivity – more flexibility in path selection Both stability and limiting the number of path eliminations are concerns of algorithm

47. Griffin&Wilfong Dynamic solution “History” carried with each Update message – Potentially very long messages – High communication overhead History may reveal preferences of other ASes Cycle in the history is necessary but not sufficient condition for divergence – There may be false positives Stability is the only goal Path elimination is the only means of resolving conflicts Eliminated paths cannot be used later under any condition Cannot differentiate between persistent and transient oscillations – Suggests observing the same loop for a number of times in history bigger values increase communication overhead even more smaller values lead to false positives Simultaneous path eliminations are possible even when single path elimination is sufficient APMS Dynamic solution No communication overhead unless there is a topology change No privacy concerns There may be false positives due to local solution – Paths are not eliminated immediately – Eliminated paths may be reused after the system stabilizes Goal is both stability and limiting the number of path eliminations Changing policies is the primary means of resolving conflicts Eliminated paths can be used later –Adapts to every state of the network –Topology change Differentiate between persistent and transient oscillations due to topology change – More effective mechanism for this purpose Helps to minimize false positives If transient oscillation is not because of topology change cannot distinguish – Uses min_treshold for this purpose bigger values lead to longer convergence smaller values lead to more rank change Simultaneous rank changes are minimized via probabilistic approach

48. Cobb&Musunuri Dynamic solution Costs are associated with nodes, not paths – Aggregates paths through the same node – One flapping path may cause all the alternatives to be rejected Costs of the nodes involved in the same conflict grows in tandem – Simultaneous path eliminations Solves conflicts through path elimination Hard to adapt to the dynamics of the system after conflicts disappear –Suggests resetting costs via diffusing computations periodically Has to keep min-hop spanning tree for each destination Cannot be done very often, expensive Blindly resetting the costs introduces the resolved conflicts back to the system –Weekly or monthly resets are suggested hoping that conflicts resolved by themselves in the meantime! Local state at each node – node count per destination – (id of the parent on the min-hop spanning tree, hop count to the destination) per dest APMS Dynamic solution Costs are associated with paths – Can exactly pinpoint the paths causing problems Leads to less preference change and/or path suppression Costs of the nodes involved in the same conflict grows in tandem – Due to probabilistic approach, nodes do not react simultaneously Leads to less preference change and/or path suppression Path elimination is the not the primary means of solving conflicts Easily adapt to the dynamics of the system: conflict free or potentially conflicting Potentially larger local state at each node

49. Simulation Results

50. Performance Metrics Average percentage of paths that are eliminated per node among the permitted paths to provide stability –Smaller values indicate better performance Eliminating permitted paths may –strain reachability –force router to use less preferred path Average percentage of the paths whose rank has been changed per node –Smaller values indicate better performance higher number of rank changes mess with the policies placed for specific purposes Average of the percentage of the preference loss per node –Preference loss of a path is the difference between its original local preference value and its current local preference value If a path is in bad path set, its preference loss is its original preference value –Helps quantify the total effect of both path elimination and rank change –Smaller values indicate better performance

51. Performance Metrics Number of Update messages exchanged between routers –Indication of stability –Smaller values show the efficiency of the protocols dealing with conflicts Number of octets carried with Update messages –Measures overhead Average extra storage used (in bytes) –For SPVP history carried and stored at the routing tables along the Updates bad path set –For APMS local history, bad path set are main contributors per peerStability, per peer keepaliveCount Throughput Number of packets received in the last 100sec is averaged over 100sec. Average delay for the packets received Delay of the packets received in the last 100sec is averaged over 100sec

52. Simulation Set I 15 independent dispute wheels with increasing size Each node has 3 permitted paths: 1.Through its clockwise neighbor; localPref(100) 2.Direct path; localPref (80) 3.Path through its counterclockwise neighbor; localPref(40) Constant data flow Unbounded buffers Periodic link failures: ASes lose connection to 0 Failures happen at 1000sec, 3000sec Failures last 1000sec APMS variations: (min_threshold=2, ka_threshold=6) 1. max_threshold=3, topology change diagnostic 2. max_threshold=3, no topology change diagnostic 3. max_threshold=10, topology change diagnostic 4. max_threshold=10, no topology change diagnostic Griffin&Wilfong: Uses path elimination after seeing the same loop twice

53. APMS with max_threshold=3, no topology change diagnostic –False positives APMS with max_threshold=3, with topology change diagnostic – Big improvement, 0.48% APMS with max_threshold=10 –Resolves conflicts by path rank change –Minimal path elimination SPVP eliminates the flapping paths to deal with conflicts, 14.4% Average percentage of the paths whose rank has been changed per node Using topology change diagnostic improves performance For max_threshold=10, metric value drops from 18% to 7% There is not a single path elimination for this case For max_threshold=3, metric value drops from 15% to 5.4% Results Average percentage of paths that are eliminated

54. Results Average of the percentage of the preference loss per node SPVP causes loss of 18%, only because of eliminated paths Performance with APMS is always better than SPVP Larger values of max_threshold along with topology change diagnosis significantly improves performance to less than 1% loss of path preferences. Number of Update messages exchanged between routers Link failure and restoration causes burst of Updates Failures: Paths to the destination are withdrawn Recovery: BGP Session is re-established, whole routing tables are exchanged Metric value for BGP4 for non-fail periods is not 0: system does not stabilize

55. Results Number of octets carried with Update messages SPVP has the highest number of octets carried APMS shows best performance APMS’s way of differentiating temporary oscillations due to topology change is more efficient than SPVP’s BGP4 has nonzero value for the metric for non-fail periods due to instability Average extra storage used (in bytes) Due to history, SPVP requires much larger storage than APMS APMS requires 10KB extra storage, SPVP requires 200KB-360KB For non-fail periods, the metric value is higher due to better reachability

56. Simulation Set II 7 dispute wheels, some intervening: {AS1, AS2, AS3}, {AS4, AS5, AS6}, {AS7, AS8, AS9}, {AS10, AS11, AS12}, {AS13, AS14, AS15}, {AS16, AS17, AS18}, {AS19, AS20, AS21} No topology change Limited buffer size, routing packets are given priority over data packets Constant data flow: –From servers located at AS0 to the clients located at the other ASes –From servers located at other ASes to the clients located at AS0

57. Topology and Path Ranks

58. Results Throughput APMS is better than SPVP –Size of Update messages are short –Does not eliminate as many paths APMS is better than BGP4 –Reaches stability quickly leads to smaller number of exchanged Updates BGP4 performs better than SPVP –Does not eliminate paths permanently –Some packets may not reach destination due to temporary stability –Update messages are shorter than SPVP Delay SPVP causes the highest packet delay due to the longest Update messages BGP4 performs better than SPVP due to shorter Update messages BGP4 performs better than APMS since APMS forces some nodes to stabilize on their longer paths

59. Conclusion Proposed new dynamic algorithm, APMS, adapting to the system dynamics while resolving policy conflicts and overcoming the shortcomings of available solutions Correctness analysis Simulation results Future Work Transient performance analysis More detailed evaluation model –include IBGP Prototype implementation

60. Other Work Class based Isolation of UDP, short lived TCP and long lived TCP flows –separate service queues at the routers –better fairness –improved predictability for all kinds of flows –better control over QoS of a particular traffic type Evaluated scalability vs performance tradeoffs in MPLS and IP Routing –per-packet routing: stateless –Widest Shortest Path: per-flow state –MIRA: per-flow state, uses ingress-egress pair matrix –PBR: per-flow state, per-class state, uses both ingress-egress pairs and traffic matrix WSP is the most scalable among per-flow algorithms, shows good performance PBR is the least scalable, most complex (time and space), performance suffers due to unsplitability of flows