1. The Cutting EDGE of Router Configuration D. Caldwell, A. Gilbert, J. Gottlieb, A. Greenberg, G. Hjalmtysson, and J. Rexford AT&T Labs—Research; Florham Park, NJ Manual configuration: Type first, and ask questions later

2. Outline Motivation for automation EDGE “bottom up” approach Example functionality of the EDGE tool –Inventory and network visualization –Configuration errors and warnings –Reverse engineering of local policies Software architecture –Data feeds and configuration parsing –Extracting tables and joining them –Presentation queries and GUI Conclusions

3. Manual Configuration Dangerous –Typo in routing policy: black hole –Wrong OSPF area: no traffic on link –Missing packet filter: DoS vulnerability Expensive –Delays in deploying equipment –Hiring and training skilled engineers –Lock-in to the router vendor Disruptive –Half of network outages (Yankee Group) –BGP routing anomalies (SIGCOMM’02) –Failures of Internet services (USITS’03)

4. Why is the Situation So Bad? Networking field: emphasis on speed & features –Not manageability and simplicity –Constant change, without revisiting the design –Oodles of complex protocols and tunable parameters Router vendors: lack of abstraction –Assembly language commands –Element-level configuration –Low-level mechanisms, not intent Network administrators: melting under complexity –Learning on the job –Struggling just to keep up –The path to automation is overwhelming

5. Our Goal: Automated Configuration How to transition an existing network? How to get value as you move from here to there? Our approach: detailed analysis of configuration data DB interface Serial10/1/0/12:0 description CBB Customer ip address 12.7.35.1 255.255.255.252 ip access-group 666 in ! router bgp 7018 neighbor 12.7.35.2 remote-as 18585 neighbor 12.7.35.2 route-map CUST-FACE in neighbor 12.7.35.2 route-map FULL-ROUTES out neighbor 12.7.35.2 distribute-list 13 in neighbor 12.7.35.2 soft-reconfiguration-inbound ! What is your AS number? What export policy do you want? Do you want a dynamic default? What are your address blocks? Do you need to receive communities? query template RULESRULES configlet Technical Questions (TQ) interface description ip address ip access-group in ! router bgp 7018 neighbor remote-as neighbor route-map CUST-FACE in neighbor route-map out neighbor distribute-list in neighbor soft-reconfiguration-inbound [neighbor send-community] ! router

6. Lowering Barrier-to-Automation for Enterprises Large enterprise networks –Large stand-alone data networks –Retail, financial, health, business, etc. –Heterogeneity due to mergers and acquisitions –Stringent reliability and performance demands TodayEDGE Inventory database The network is the database Extracted from the configuration files Existing network Mistakes and inconsistencies Discord reports from analyzing config files Local config rules Poor (or no) documentation, policy violations Reverse-engineered by data mining on the configuration files

7. Enablement and Debugging of Growing Enterprises BOTTOM-UP APPROACH 1.Inventory database  Extract summary information  Bootstrap the database 2.Fixing config mistakes  Report errors & warnings  Allows immediate fixes 3.Codifying local policies  Reverse-engineer policies  Aids in fixing inconsistencies 4.Automated configuration  Explicitly enforce the rules  Avoid future config mistakes Web reports Discords Low level standard form (tables) Abstract network database polled queries Router configuration Automation fix Runs on many thousands of configs a day.

8. Network Inventory Equipment –Routers Model, OS version, available slots, protocol usage, … –Interfaces Media, speed, edge/core, protocol usage, … Address space –Network equipment and announced supernets –Network services (e.g., NTP and SNMP) –Packet/route filters and eBGP end-points Configuration commands –Histogram of configuration commands –Frequency of formats and options

9. Identifying the Topology Interfaces to links BGP end-points to sessions 12.7.35.0/30 12.7.35.1/3012.7.35.2/30 neighbor 12.7.108.3 neighbor 12.123.37.250 12.123.37.250 12.7.108.3

10. Network Visualization Goal: automatic network design diagram –Input: router configuration files –Output: picture a network designer would draw Key ingredients from configuration –Layer-3 topology (e.g., routers and links) –Routing protocols and their attributes –Route injection, filtering, & aggregation policies Visualization software –Placement based on node degree, routing, … –GUI with browsing, zooming, attributes,…

11. Automatic classification –Red: high degree (dual hubs) –Blue: degree two (spokes) –Yellow: degree one (stubs)

12. Visualization clearly shows disconnected network, or missing configuration files

13. OSPF Topology Area 0 Area 1 Area 2 2 2 1 2 1 5 3 3 2 4 3 3 3 3 3 4 4 4 4

14. OSPF Example hostname MyRouter ! interface POS7/0 ip address 12.7.35.1 255.255.255.252 ip ospf cost 50 ! router ospf 2 network 12.7.35.0 255.255.255.0 area 9 passive-interface Serial2/1/0/3.1 ! Remote end is in 12.7.35.2/30 Interface participates in OSPF

15. Routing Protocol Consistency Warnings –OSPF interface has no cost metric –Non-OSPF interface has OSPF attributes –OSPF network matches no interfaces Errors –Remote end is a non-OSPF interface –Remote end has a different OSPF area Inventory –OSPF links and edge interfaces –Routers and links by OSPF area –Interfaces by OSPF cost

16. Referential Integrity Checks Duplicate IP addresses –Multiple interfaces with the same address Items used but not defined, and vice versa –Packet filters, route filters, QoS policies, routing policies, AS path lists, etc. interface Serial10/1/0/12:0 ip address 12.7.35.1 255.255.255.252 ip access-group 666 in ! access-list 666 permit 12.34.158.0 0.0.1.255 access-list 666 permit 12.7.35.0 0.0.0.3

17. Using Data Mining to Infer Local Policies Pattern matching across routers –Equivalence: same config, same names –Synonym: same config, different names –Homonym: same name, different config Infer rules and report exceptions –“Finger daemon disabled” –“Edge interfaces have inbound packet filters” –“OSPF costs of 100 to hub1 and 200 to hub2” –“IPX interfaces have SAP encapsulation” Identifying “use cases” –Initial base configuration of each router –“Diffs” over time and across related routers –Common cases (e.g., adding link, new BGP session)

18. Data Mining Approaches Configuration files –Start with raw or lightly-parsed data –Look for patterns in the uninterpreted strings –Discover “templates” for flat parts of configuration Database tables –Start with a model of part of the data –Look for correlations between fields –Find features that tend to appear together Network graph –Start with a graph with edge/node attributes –Look for patterns in topology and attributes –Classify the graph and routing parameters

19. EDGE Software Design Principles Scalability –CVS repository for daily feed of configuration data –Automated data processing for hundreds of networks –Precomputed tables underling interactive Web GUI Extensibility –Extensible configuration parser –Incremental additions to data models –Ease of adding new queries on existing data models Ease of development –Single low-level configuration parser –SQL database for running queries –XML specification of GUI functionality

20. EDGE Software Architecture Raw router configuration data Parsed configuration data Extractors Utility queries Presentation queries Web-based GUI OSPF, EIGRP, RIP Referential integrity developer smokestacks

21. Router Configuration Data Getting the configuration data –From a backup server (ssh, wget, ftp, etc.) –Directly from the router (poller, crawler) Mapping files to networks –Regular expressions on file names –E.g., “feed-foo/hp*” maps to Hewlett Packard Storing the raw files –Committing the data by network into CVS Checking data integrity –Detecting duplicate configuration files –Identifying decommissioned routers

22. Parsing the Configuration Files Practical constraints –No public grammar exists for Cisco IOS –Too many commands to parse everything Solution: light, extensible parsing –Identify section boundaries Router, interface, router ospf, router bgp, etc. –Parse key commands Interface name, address, description, bandwidth –Retain unparsed commands Leave unparsed commands as strings –Store results in Perl hash table Support indexing and sequencing of data

23. Extractors Practical constraints –Huge breadth and depth of the data –Unstructured raw configuration data –Absence of predefined data models Solution: incremental data modeling –Extract low-level tables from the data Simple scripts using the Perl hash table or “grep” –Small number of flat tables for data mining Router, interface, BGP end-points, ACLs, names, etc. –Insert tables into an SQL database Allow other queries to build on this data

24. Example Extractor: Referential Integrity Extract relevant commands –Definitions and references, by type and name RouterR/DCommandTypeName nyc72defaccess-listACL666 nyc72refaccess-groupACL666 sfo35refneighborroute-mapBLOCK chi19defip prefix-listprefix-listPEER sea42defclass-map Silver sea42refclassclass-mapSilver

25. Utility Queries Practical constraints –Some joined data needed by multiple queries –Table joins are too slow to do on-demand Solution: precomputed utility tables –SQL queries built on extracted tables Run in advance, with results stored in database –Building blocks for the “smoke-stacks” Key constructs like link, OSPF interface, etc. –Example: constructing the “links” table Self-join on the “interface” table Interface1 address in same network as interface2

26. Presentation Queries Practical constraints –Many different reports on the same data –Customizing column names and order of rows Solution: XML specification of output –SQL query on the database Lightweight sub-selecting, summing, max, etc. “ORDER BY” to control the order of rows –Presentation niceties Query title, description, and export to Excel Renaming of columns to user-friendly terms Click on table entry to run additional query

27. Example: Remote End in Different OSPF Area interface OSPF network OSPF passive interface link OSPF interface Extracted tables active OSPF interface OSPF link with area mismatch Utility tables Presentation query result Simple SQL queries OSPF link

28. Software Status Runs daily for hundreds of networks –AT&T’s core backbone networks –Managed enterprise networks –New eVPN customers –Stand-alone enterprise customers Reports in the Web portal sites –Inventory and visualization –Routing protocols and policies –Referential integrity checks –ACL optimization and synonyms

29. Conclusion Moving beyond manual configuration –Manual configuration is bad Error-prone, expensive, and disruptive –Migrating to automation is hard No inventory DB, buggy network, and poor docs –EDGE supports migration to automation Bootstrap DB, report discords, and infer policies More innovation is needed –Data mining on existing data networks –Protocol models and best common practices –Better router configuration languages –Self-configuring protocols and mechanisms