Automating BGP Customer Configuration R: Joel Gottlieb, Albert Greenberg, Jennifer Rexford, and Jia Wang D/O: Jose Berrios, Jay Borkenhagen, Gary Flack, Brian Freeman, Rich Kwapniewski, Han Nguyen, and Janet Peng PM: Herani Brotman and Rose Klimovich

Outline Introduction –What is the Border Gateway Protocol? –Which CBB customers use BGP, and why? –How does CBB configure the customers today? ENCORE project –Configuration template with Cisco commands –Analysis of existing customer configuration –Prototype “configlet” generation from database Conclusion –Project timeline and status –Ongoing work and future directions

Many Customers Have Static Routing provider customer Nail up default routes /0 pointing to provider Nail up routes /23 pointing to customer /23

Border Gateway Protocol (BGP) Interdomain routes between Autonomous Systems ASes announce the destinations they can reach Local policies for path selection (which to use?) Local policies for route propagation (who to tell?) “I can reach /23” “I can reach /23 via network 1”

Multi-Homed Customer Speaking BGP Motivations for using BGP –Load balancing and fault tolerance –Send updates to influence how traffic enters –Apply routing policy to select outbound paths Provider A Provider B Customer Provider Customer

BGP Customers are Very Important Who are they? –Large, savvy customers with multiple providers –Retailers, enterprises, & carriers with big pipes –E.g., Amazon, Wal-Mart, Kodak, Prudential, Marriott, Merck, Charter Communications, etc. How many are there? –Around 3600 BGP sessions –Around 4600 access links How do we provision them? –Treated as “special case” & configured manually –Sometimes seen as too complicated to automate

Manual Router Configuration is Costly Slow and expensive –Extra delays in provisioning new customers –Folks living in MT during the Redwood migration –Hiring skilled engineers, and training on CBB policies –Problematic to select other vendors (e.g., Juniper) Error-prone –Fidelity cut off while provisioning a new customer –Vulnerability to DoS due to missing packet/route filters –Errors manifest themselves as costly SLA violations Inflexible –Hard to handle complexity of new features and services –Hard to change the business and configuration rules –Hard to let customers change their own configuration

Automation is Challenging Extremely difficult to outsource –Customer data: sensitive, internal information –Configuration rules: embody AT&T’s proprietary design and engineering of the network Requires an end-to-end perspective –Data: familiarity with TQ and with BIDS database –Policies: understanding of the CBB policies –Configuration: knowledge of router configuration Bootstrapping –Data for existing BGP customers not fully in database –Manually-configured policies don’t fully match template, causing chicken-or-the-egg problem for automation

ENCORE: End-to-End Approach Complete automation of provisioning of BGP customers From TQ, to database, network rules, and the router BIDS interface Serial10/1/0/12:0 description CBB Customer ip address ip access-group 666 in ! router bgp 7018 neighbor remote-as neighbor route-map CUST-FACE in neighbor route-map FULL-ROUTES out neighbor distribute-list 13 in neighbor soft-reconfiguration-inbound ! What is your AS number? What export policy do you want? Do you want a dynamic default? What are your address blocks? Do you need to receive communities? query template RULESRULES configlet Technical Questionnaire (TQ) interface description ip address ip access-group in ! router bgp 7018 neighbor remote-as neighbor route-map CUST-FACE in neighbor route-map out neighbor distribute-list in neighbor soft-reconfiguration-inbound [neighbor send-community] ! router

Abstract and Codify Documented the CBB rules –Explained the reasons for the commands –Explicit pseudocode for the “configuration rules” –Rules vetted by the key people in Tier-3 & Tier-2 – Created CBB-Specific IOS template –Access links with name, address, bandwidth, etc. –BGP session with remote-AS, IP address, etc. –Static routes associating access links with session –Packet filters, route filters, and routing policies

An Example Configlet hostname abyny31c3 ! interface Serial10/1/0/12:0 description CBB Customer ip address ip access-group 666 in ! router bgp 7018 neighbor remote-as neighbor route-map CUST-FACE in neighbor route-map FULL-ROUTES out neighbor distribute-list 13 in neighbor soft-reconfiguration-inbound ! access-list 666 permit access-list 666 permit access-list 13 permit ip host host

Analyze and Fix Analyze: validated the template –Netdb extraction of existing customer configuration –Comparison with our proposed Cisco IOS template –Matching more than 99% of the customer BGP sessions –Numbers continue to improve as configuration is fixed Fixed: identified mismatches with the template –Small number of dangling customer BGP sessions –Small number of missing packet/route filters –Other minor violations of the CBB network rules –Tier-2 fixed (and fixing) the config of these customers –Must keep fixing problems as they arise, … or automate!

Identifying CBB Customers Private peers –Inbound route-map that tags routes with 0:5000 –E.g., Sprint, UUNet, Cable & Wireless, … Siblings –AT&T-owned ASes in a hand-constructed list –Cerfnet, AGNS, AT&T Canada, … Internal ASes –Internal networks with private AS numbers –Network management, Dial Platform access routers, … Special external ASes –Hand-constructed list of ASes –Public route servers, multicast networks, etc. Customers: Everything else! –More than 3600 sessions and 4600 access links

Classifying Customers (April 15, 2002) Single-session customers –One BGP session with the CBB at one AR –1195 ASes, 1195 BGP sessions, and 1452 interfaces Multi-session customers –Two or more BGP sessions with the CBB –339 ASes, 1077 BGP sessions, and 1374 interfaces Dedicated-AS sessions –Customers using AS 2386 for their BGP sessions –1 AS, 1422 BGP sessions, and 1774 interfaces “Swamp” customers –Sessions that we cannot associate with interfaces –8 sessions (some in provisioning or decommissioning)

Mandatory Commands Nearly every (non-swamp) session is complete –Each interface has address, match, and packet filters –Each session has remote-as, distribute-list, and inbound and outbound route-maps Small number of exceptions (17 sessions) –10 sessions with interface missing a packet filter –5 sessions with static route to non-existent interface –1 session missing an outbound route-map –1 session with interface missing address/mask Route-maps –Vast majority of customers use standard route-maps –Less than 1% have unusual policy (tailored to customer)

Prototype Identified additions to TQ and database –Variables and features specific to BGP configuration –Identified customer vs. CBB-selected fields –Proposed layout of new tables in the database Generated BGP configlets –X-BIDS database populated from Netdb data –Query to extract data needed for BGP configuration –Checked that all of the necessary data are present –Populated the templates to generate the configlet –Displayed configlets on the ENCORE Web site

ENCORE Timeline: Rapid Prototyping Analyze existing BGP customersSmall set of cases plus the dangling sessions (Web site) Fix the configuration mistakesFix danglers and mismatches with the template/rules (Tier-2) Create provisioning templatesCisco IOS config template, XML representation of customer data Document the CBB policiesVetted by key players; presented in a technical report and tutorial Identify new data for BIDSX-BIDS, virtual view, DBI interface Identify new questions for TQX-TQ questions Build proof-of-concept prototypeENCORE prototype Phase into production systemsProposed “routing database” Cut-over to operationsConcept of Zero You are here

Ongoing/Future Work Routing database effort (BIDS extensions) –Team from BIDS, Tier-2, Tier-3, and Research –Identifying the necessary extensions to BIDS –Identifying “use cases” to test the data models Expanding the configuration template –All BGP sessions, including peers of the CBB –All routing technologies, including static routes –Base configuration, including BGP routing policies Data modeling –Making configuration changes through virtual view –Configuration changes/deletions for existing customers –Supporting other router vendors, such as Juniper