© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-1 Optimizing BGP Scalability Improving BGP Convergence

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-2 Outline Overview BGP Convergence BGP Processes CPU Effects of BGP Processes Improving BGP Convergence PMTU Discovery Increasing Input Queue Depth BGP Scan Time BGP Advertisement Interval BGP Nonstop Forwarding Awareness Summary

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-3 BGP Convergence As the number of routes in the Internet routing table grows, the time it takes for BGP to converge increases. The Internet currently contains more than 110,000 prefixes. Network convergence times can range from 10 minutes to more than one hour. BGP is considered converged when: –All routes have been accepted. –All routes have been installed in the routing table. –The table version for all peers equals the table version of the BGP table. –The input queue and output queue for all peers is 0.

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-4 BGP Processes BGP scanner and BGP router are responsible for a large number of calculations and can lead to high CPU utilization. BGP open As BGP control packets are received BGP scanner BGP router ProcessDescriptionInterval Performs BGP peer establishment. Handles queuing and processing of BGP packets (updates and keepalives). At initialization, when establishing a TCP connection with a BGP peer BGP I/O Walks the BGP table and confirms reachability of the next hops. BGP scanner also checks conditional advertisement to determine whether or not BGP should advertise condition prefixes. Performs route dampening. Every 60 seconds Calculates the best BGP path and processes any route changes. It also sends and receives routes, establishes peers, and interacts with the routing information base (RIB). Once per second and when adding, removing, or soft- reconfiguring a BGP peer

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-5 CPU Effects of BGP Processes BGP scanner process High CPU utilization stemming from the BGP scanner process can be expected for short durations on a router carrying a large Internet routing table. While the BGP scanner runs, low-priority processes need to wait a longer time to access the CPU. BGP router process The BGP router process runs about once per second to check for work. The BGP router consumes all free CPU cycles.

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-6 Improving BGP Convergence You can reduce BGP convergence time and high CPU utilization caused by BGP processes in the following ways: Queuing to TCP peer connections –BGP now automatically queues data aggressively from the BGP output queue to the TCP socket for each peer Deploying BGP peer groups –Simplifies BGP configuration and enhances BGP scalability Enabling the path MTU feature –Improves efficiency by dynamically determining the largest MTU that you can use without creating packets that need to be fragmented Increasing interface input queues –Improves convergence by reducing dropped TCP ACKs

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-7 Improving BGP Convergence (Cont.) BGP convergence can also be improved to some extent by: Configuring a smaller interval for the BGP scanner process (scan time) Configuring a smaller advertisement interval between BGP neighbors Limitation: Not recommended in routers dealing with large BGP tables Could lead to CPU or memory exhaustion

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-8 PMTU Discovery ip tcp path-mtu-discovery [age-timer {minutes | infinite}] router(config)# This command enables the PMTU discovery feature for all new TCP connections from the router. The age timer is a time interval for how often TCP re-estimates the path MTU with a larger MSS (default age timer is 10 minutes). This feature is described in RFC 1191.

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-9 Monitoring PMTU Discovery The default MSS is 536 bytes. After enabling of the PMTU discovery feature, the MSS has been increased.

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-10 Increasing Input Queue Depth hold-queue length in router(config-if)# This command limits the size of the IP queue on an interface. The default input hold-queue limit is 75 packets, configurable from 0 to 65,535 packets. A length of 1000 will normally resolve problems caused by input queue drops of TCP ACKs.

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-11 Monitoring Input Queue Depth

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-12 BGP Scan Time bgp scan-time scanner-interval router(config-router)# This command changes the default value of BGP scanner process runs (default = 60 seconds). The BGP scanner walks the BGP table and confirms the reachability of next hops. The BGP scanner process is also responsible for advanced features such as conditional advertisement check and performing route dampening.

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-13 Monitoring BGP Scan Time Scan interval is defined per BGP router process and address family Checked with show ip bgp summary command

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-14 BGP Advertisement Interval neighbor {ip-address | peer-group-name} advertisement- interval seconds router(config-router)# This command changes the default time interval in the sending of BGP routing updates for a specific neighbor: –If lowered, can improve convergence –Can consume considerable resources in a jittery network if the value is set too low Default values: –30 seconds for EBGP neighbors –5 seconds for IBGP neighbors

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-15 Monitoring the BGP Advertisement Interval Defined per BGP neighbor and address family Manually configured minimum value stated under address family output of show ip bgp neighbors command

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-16 NSF Awareness Allows an NSF-aware router to assist NSF-capable and NSF- aware neighbors to continue forwarding packets during a switchover operation or during a well-known failure condition Minimizes the effects of the following: –Well-known failure conditions (for example, a stuck-in- active event) –Unexpected events (for example, an RP switchover operation) –Scheduled events (for example, a hitless software upgrade)

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-17 Summary Convergence is defined as the process of bringing all route tables to a consistent state. Several Cisco IOS processes (including BGP open, I/O, scanner, and router) that are executed on the router enable BGP to run. The BGP scanner and BGP router processes can significantly impact the CPU utilization of the router, causing some low-priority processes to suffer increased processing delays. You can reduce BGP convergence time and high CPU utilization caused by BGP processes in the following ways: queuing to TCP peer connections, deploying BGP peer groups, enabling the PMTU feature, and increasing interface input queues. The PMTU discovery feature, implemented through the ip tcp path- mtu-discovery global configuration command, finds the largest packet that can be sent to a destination without requiring IP fragmentation, minimizing packet overhead.

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-18 Summary (Cont.) Increasing the input queue depth is a technique that can eliminate dropped TCP ACKs, resulting in improved BGP convergence. To specify the size of the IP input or output queue on an interface, use the hold-queue command. Reducing the time between runs of the BGP scanner process (using the bgp scan-time command to configure the time interval for repetitions of the BGP scanner process) improves BGP convergence at the cost of increased CPU resource consumption. With the help of the neighbor advertisement-interval command, you can reduce the advertisement interval, causing BGP updates to be sent to neighbors more quickly and resulting in improved BGP convergence time. NSF awareness allows an NSF-aware router to assist NSF- capable and NSF-aware neighbors to continue forwarding packets during a switchover operation or during a well- known failure condition.

© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—7-19