1. Swinog-3, 19 September 2001 Fabien Berger, berger@ip-plus..net BGP Oscillation …the Internet routing protocol is diverging! Fabien Berger CCIE#6143 IP-Plus Backbone Engineering

2. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 2 Well known issue?  Does a BGP system always converge? –NO! –feature not a bug :)  Researchers have shown theoretical eBGP convergence issues –[Griffin]:  “bad gadget” topology diverges!  backup scenario diverges!  iBGP diverges in complex RR/confederation environment (draft- ietf-idr-route-oscillation-00.txt)

3. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 3 Goal of the presentation  make you aware of the issue (before the customer :) –troubleshooting not easy  pointer to solutions/discussions  presentation based on [NANOG] [Cisco] [IETF]

4. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 4 Convergence  convergence = “process of bringing all route tables to a state of consistency” –no loops!  does not converge -> you see (on a RR or a confed border): #show ip bgp 10.0.0.0 | include best # Paths: (3 available, best #3) #show ip bgp 10.0.0.0 | include best # Paths: (3 available, best #2) #show ip bgp 10.0.0.0 | include best # Paths: (3 available, best #3)...

5. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 5 Cause of the Oscillation  RR/confederation hides some information –RR/confederation sends best path only –not all routers know all best paths  MED (Multi Exit Discriminator) vs IGP cost to the neighbor: A: path 200 100, igp cost 5, med 2 B: path 300 100, igp cost 50 C: path 200 100, igp cost 500, med 1  if A,B,C are known: B is best (assuming “deterministic-med” is enabled [detMED] :)  if C is hidden: A is best  A<B<C<A

6. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 6 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster 2 23 10 1 = Withdrawal = Advertisement Step 1 – B selects Y0 – C selects Y1 = Route Reflector = Client B C X3 Y 12 * * Y0 AS_PATHMEDIGP 10

7. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 7 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster 2 23 10 1 = Withdrawal = Advertisement Step 2 – C selects X = Route Reflector = Client B C * X3 Y 12 * Y0 AS_PATHMEDIGP Y0 10 11 Y 13

8. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 8 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster 2 23 10 1 = Withdrawal = Advertisement Step 3 – B selects X = Route Reflector = Client B C * X3 Y 12 Y0 AS_PATHMEDIGP Y0 10 11 * X4

9. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 9 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster 2 23 10 1 = Withdrawal = Advertisement Step 4 – C selects Y1 = Route Reflector = Client B C X3 * Y 12 Y0 AS_PATHMEDIGP 10 * X4

10. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 10 Oscillation A DE AS Y MED 0 AS XAS Y MED 1 B C Cluster 1Cluster 2 23 10 1 = Withdrawal = Advertisement Step 5 – B selects Y0 = Step 1!! = Route Reflector = Client B C X3 * Y 12 0 AS_PATHMEDIGP 10

11. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 11 How to detect an oscillation?  Observe the latest received routes: –run every minute during 5 minutes #show ip route | include ^B_.*_00:00: –prefixes that appear 60% of the time are probably oscillating –full routing table must be traversed :(  Via SNMP: –poll ipRouteAge of ipRouteTable  Observation should be made in the core (top level RR, backbone sub-AS) –eBGP within a confed applies flap damping –RR client may see only the replacement route

12. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 12 Shall we care?  MED usage –34% of the prefix we receive have the MED set –75% of our peers have > 1 prefix with MED set  Potential AS that can oscillate (AS received via > 2 peers) –60% (upper bound, as-path not taken into account!)  Oscillation not propagated to customers because of damping  Oscillation seen in our backbone :( but cured :)

13. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 13 Solutions  configure bgp deterministic-med  full iBGP mesh when you can  do not listen to the MED (or only with stub-AS) –set metric 0 on all prefixes –bgp always-compare-med  use local-pref to force decision –exit no longer chosen by peer = more work :(  allow peer to set local-pref using community  protocol improvement –RR/confederation should send more than just the best path –closer to the iBGP full mesh :(

14. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 14 Conclusion  It’s happening today :(  It is possible to detect  Solutions (fixes) exist today  Protocol improvement on the way by IETF

15. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 15 References  [Cisco] http://www.cisco.com/warp/customer/770/fn12942.htmlhttp://www.cisco.com/warp/customer/770/fn12942.html  [IETF] draft-ietf-idr-route-oscillation-00.txt, www.ietf.orgwww.ietf.org  [Nanog] NANOG 21 Atlanta February 2001, www.nanog.orgwww.nanog.org  [Griffin] http://www.research.att.com/~griffin/http://www.research.att.com/~griffin/  [detMED] http://www.cisco.com/warp/public/459/37.htmlhttp://www.cisco.com/warp/public/459/37.html  [bgpDecision] http://www.cisco.com/warp/public/459/25.shtmlhttp://www.cisco.com/warp/public/459/25.shtml

16. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 16 BGP Oscillation comments? questions? experiences?

17. Swinog-3, BGP Oscillation Fabien Berger, berger@ip-plus.net 17 BGP Decision Process [bgpDecision] 1. Largest weight 2. Largest local preference 3. Locally originated 4. Shortest AS-Path length 5. Lowest origin 6. Lowest Multi Exit Discriminator (cisco default = 0 unlesss “bgp- bestpath-missing-as-worst”) 7. Prefer EBGP over IBGP (conf EBGP=IBGP) 8. Lowest IGP metric 9. Lowest BGP router ID