Swinog-3, 19 September 2001 Fabien Berger, BGP Oscillation …the Internet routing protocol is diverging! Fabien Berger CCIE#6143 IP-Plus Backbone Engineering

Swinog-3, BGP Oscillation Fabien Berger, 2 Well known issue?  Does a BGP system always converge? –NO! –feature not a bug :)  Researchers have shown theoretical eBGP convergence issues –[Griffin]:  “bad gadget” topology diverges!  backup scenario diverges!  iBGP diverges in complex RR/confederation environment (draft- ietf-idr-route-oscillation-00.txt)

Swinog-3, BGP Oscillation Fabien Berger, 3 Goal of the presentation  make you aware of the issue (before the customer :) –troubleshooting not easy  pointer to solutions/discussions  presentation based on [NANOG] [Cisco] [IETF]

Swinog-3, BGP Oscillation Fabien Berger, 4 Convergence  convergence = “process of bringing all route tables to a state of consistency” –no loops!  does not converge -> you see (on a RR or a confed border): #show ip bgp | include best # Paths: (3 available, best #3) #show ip bgp | include best # Paths: (3 available, best #2) #show ip bgp | include best # Paths: (3 available, best #3)...

Swinog-3, BGP Oscillation Fabien Berger, 5 Cause of the Oscillation  RR/confederation hides some information –RR/confederation sends best path only –not all routers know all best paths  MED (Multi Exit Discriminator) vs IGP cost to the neighbor: A: path , igp cost 5, med 2 B: path , igp cost 50 C: path , igp cost 500, med 1  if A,B,C are known: B is best (assuming “deterministic-med” is enabled [detMED] :)  if C is hidden: A is best  A<B<C<A

Swinog-3, BGP Oscillation Fabien Berger, 6 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster = Withdrawal = Advertisement Step 1 – B selects Y0 – C selects Y1 = Route Reflector = Client B C X3 Y 12 * * Y0 AS_PATHMEDIGP 10

Swinog-3, BGP Oscillation Fabien Berger, 7 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster = Withdrawal = Advertisement Step 2 – C selects X = Route Reflector = Client B C * X3 Y 12 * Y0 AS_PATHMEDIGP Y Y 13

Swinog-3, BGP Oscillation Fabien Berger, 8 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster = Withdrawal = Advertisement Step 3 – B selects X = Route Reflector = Client B C * X3 Y 12 Y0 AS_PATHMEDIGP Y * X4

Swinog-3, BGP Oscillation Fabien Berger, 9 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster = Withdrawal = Advertisement Step 4 – C selects Y1 = Route Reflector = Client B C X3 * Y 12 Y0 AS_PATHMEDIGP 10 * X4

Swinog-3, BGP Oscillation Fabien Berger, 10 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster = Withdrawal = Advertisement Step 5 – B selects Y0 = Step 1!! = Route Reflector = Client B C X3 * Y 12 0 AS_PATHMEDIGP 10

Swinog-3, BGP Oscillation Fabien Berger, 11 How to detect an oscillation?  Observe the latest received routes: –run every minute during 5 minutes #show ip route | include ^B_.*_00:00: –prefixes that appear 60% of the time are probably oscillating –full routing table must be traversed :(  Via SNMP: –poll ipRouteAge of ipRouteTable  Observation should be made in the core (top level RR, backbone sub-AS) –eBGP within a confed applies flap damping –RR client may see only the replacement route

Swinog-3, BGP Oscillation Fabien Berger, 12 Shall we care?  MED usage –34% of the prefix we receive have the MED set –75% of our peers have > 1 prefix with MED set  Potential AS that can oscillate (AS received via > 2 peers) –60% (upper bound, as-path not taken into account!)  Oscillation not propagated to customers because of damping  Oscillation seen in our backbone :( but cured :)

Swinog-3, BGP Oscillation Fabien Berger, 13 Solutions  configure bgp deterministic-med  full iBGP mesh when you can  do not listen to the MED (or only with stub-AS) –set metric 0 on all prefixes –bgp always-compare-med  use local-pref to force decision –exit no longer chosen by peer = more work :(  allow peer to set local-pref using community  protocol improvement –RR/confederation should send more than just the best path –closer to the iBGP full mesh :(

Swinog-3, BGP Oscillation Fabien Berger, 14 Conclusion  It’s happening today :(  It is possible to detect  Solutions (fixes) exist today  Protocol improvement on the way by IETF

Swinog-3, BGP Oscillation Fabien Berger, 15 References  [Cisco]  [IETF] draft-ietf-idr-route-oscillation-00.txt,  [Nanog] NANOG 21 Atlanta February 2001,  [Griffin]  [detMED]  [bgpDecision]

Swinog-3, BGP Oscillation Fabien Berger, 16 BGP Oscillation comments? questions? experiences?

Swinog-3, BGP Oscillation Fabien Berger, 17 BGP Decision Process [bgpDecision] 1. Largest weight 2. Largest local preference 3. Locally originated 4. Shortest AS-Path length 5. Lowest origin 6. Lowest Multi Exit Discriminator (cisco default = 0 unlesss “bgp- bestpath-missing-as-worst”) 7. Prefer EBGP over IBGP (conf EBGP=IBGP) 8. Lowest IGP metric 9. Lowest BGP router ID