1 Routing Design in Operational Networks: A Look from the Inside David A. Maltz, Geoffrey Xie, Jibin Zhan, Hui Zhang Carnegie Mellon University Gisli Hjalmtysson, Albert Greenberg ATT Labs Research

2 The Problem of Routing Design HostA HostB HostCHostD eBGP iBGP OSPF RIP EIGRP ACLs policy

3 Many Routing Designs Possible A B C A B C A B C A B C A B C Drop A B AS1 AS2 AS3 Multiple OSPF instances Packet filters Multiple AS’s & BGP

4 Routing Design Selecting routing protocols Configuring their boundaries Setting the policies that control their interaction Adding packet filters, other mechanisms Routing design fundamentally establishes the network’s properties Remains constant as network expands Details of protocol, path computation are second-order effects Topology doesn’t say much about reachability

5 Reachability Example Enterprise with two remote offices Only A&B should be able to talk to server C Internet A B C Permit B->C Permit A->C

6 Reachability Example Network designers add two links for robustness Configure routing protocols to use new links in failure Internet A B C Permit B->C Permit A->C

7 Reachability Example Designers apply packet filters to new links Internet A B C Permit B->C Permit A->C Permit B->C

8 Reachability Example Internet A B C Permit A->C Permit B->C

9 Reachability Example Packet from B->C dropped! Testing under normal conditions won’t find this error! Internet A B C Permit A->C Permit B->C

10 How Are Routing Designs Expressed Today? interface Ethernet0 ip address interface Serial1/0.5 point-to-point ip address ip access-group 143 in frame-relay interface-dlci 28 router ospf 64 redistribute connected subnets redistribute bgp metric 1 subnets network area 0 router bgp redistribute ospf 64 match route-map 8aTzlvBrbaW neighbor remote-as neighbor distribute-list 4 in access-list 143 deny /16 access-list 143 permit any route-map 8aTzlvBrbaW deny 10 match ip address 4 route-map 8aTzlvBrbaW permit 20 match ip address 7 ip route /

11 Lots of Configuration Files Router ID 8810 Lines in config file

12 Problems with State of the Art No good way to visualize or describe routing design Impossible to establish linkage between configurations and routing design Only a few “textbook” routing designs are widely known

13 Approaches? Option 1: High-level design compiled down to configuration commands Feasible? What are the constructs? How to capture design intents? Our starting point: Bottom-up white-box approach Start with router configuration files Reverse-engineer the routing design

14 Contributions Abstractions for modeling routing design Routing Process Graph Routing Instance Graph Reverse-engineering methodology Anonymization of configuration files Tools to convert configurations into model Study of 31 production networks using model Verified some bits of common wisdom Found counter examples for other bits

15 Router Model OSPFBGPOSPF Route Selection Route Table Router 1

16 Route Redistribution OSPFBGPOSPF Route Selection Route Table Router 1 Routing policy 1Routing policy 2

17 Routing Protocol Adjacencies OSPFBGPOSPF Route Selection Route Table OSPF RS Route Table Router 1Router 2 Routing policy 1Routing policy 2

18 Reverse-Engineering Overview Configuration files Find links Find adjacent routing processes Construct Routing Process Graph Condense adjacent routing processes Construct Routing Instance Graph Construct Layer 3 Topology OSPF #1OSPF #2BGP AS1 AS2

19 Reconstruct the Layer 3 Topology interface Serial1/0.5 ip address …. Router 1 Config interface Serial2/1.5 ip address …. Router 2 Config Internet

20 Construct the Routing Process Graph OSPFBGPOSPF Route TableRT OSPF RT OSPF RT OSPF RT OSPF EBGP Policy1Policy2 Internet

21 Abstract to a Routing Instance Graph Pick an unassigned Routing Process Flood fill along process adjacencies, labeling processes Repeat until all processes assigned to an Instance OSPF #1OSPF #2BGP AS1 EBGP AS2 Policy1Policy2 OSPFBGPOSPF Route Table RT OSPF RT OSPF RT OSPF RT OSPF

22 Abstract to a Routing Instance Graph Router1Router2 OSPFBGPOSPF Route Table RT OSPF RT OSPF RT OSPF RT OSPF Router2Router1 OSPF #1OSPF #2BGP AS1 EBGP AS2 Policy1Policy2

23 A Study of Operational Production Networks Obtained anonymized configuration files for 31 active networks (>8,000 configuration files) Networks include: 6 Tier-1 and Tier-2 Internet backbone networks 25 enterprise networks Sizes between 10 and 1,200 routers 4 enterprise networks significantly larger than the backbone networks Networks created by diverse set of designers and companies

24 Textbook Routing Design for Enterprise Networks Border routers speak eBGP to external peers BGP selects a few key external routes to redistribute into OSPF 7 of 25 enterprise networks follow this pattern OSPF BGP AS #1 EBGP AS2 AS3

25 Reality: A Diversity of Unusual Routing Designs Network broken up into compartments, each with only 1 to 4 routers Each compartment has its own AS number Hub and spoke logical topology Why? Lots of control over how spokes communicate BGP AS #1 BGP AS #4 BGP AS #2 BGP AS #3 BGP AS #5 EBGP Rest of the World

26 Routing Design for 900 Router Network

27 Reality: A Diversity of Unusual Routing Designs Network broken up into many compartments, each running EIGRP, some with 400+ routers BGP used to filter routes passed between compartments Compartments themselves pass information between BGP speakers Why? Little need for IBGP; few routers speak BGP; Lots of control over how packets move between compartments BGP AS #1 EBGP Rest of the World EIGRP BGP AS #2 EIGRP BGP AS #3 BGP AS #4 Rest of the World EBGP

28 Myth: Policy Enforced at Edge of Network Conventional wisdom: Place packet filters on the edge to defend infrastructure Routing policy applied where networks touch

29 Reality: Policy Exists Throughout Networks Packet filters commonly used on internal links Protect routers from attack Implement reachability matrix –Prevent some hosts from communicating with others –Localize traffic, particularly multicast

30 Summary Developed abstractions to model routing design Routing Instance – abstracts away details Reverse-engineer routing design from configs We presented our extracted design to designers They agreed we captured their design intent Focusing on individual protocols is not enough Understanding composition is equally important First step towards turning routing design from an art into a science

31 Applications of Routing Design Analysis Enables static analysis of network properties Reachability/security analysis –Route leaks? Reachability violations? Robustness analysis –How sensitive is the network to external events such as route announcements? Resource usage analysis –Will a particular configuration cause the routing table of a router to overflow?

32 The Value of Investigating Routing Design: Next Steps Found many different designs in use Do we need so many designs? Framework to ask and answer questions of scalability, completeness, optimality Do we have the right abstractions? Is this the right way to program routers? Suggest improvements to protocols and configuration languages Can the network be run using abstractions?

33 Questions?

34 Textbook Routing Design for Backbone Network Border routers speak eBGP to external peers All routers speak iBGP with each other All routers participate in both BGP and OSPF (learning infrastructure routes from OSPF, external routes from BGP) OSPF BGP AS #1 EBGP IGBP MESH AS2 AS3 ASn …

35 Real Routing Designs for Backbone Networks All 6 backbone networks used basic OSPF/BGP pattern OSPF BGP AS #1 AS2 AS3 ASn

36 Real Routing Designs for Backbone Networks All 6 backbone networks used basic OSPF/BGP pattern 3 of 6 include many additional routing instances Used to exchange routes with customers OSPF BGP AS #1 AS2 AS3 ASn RIP EIGRP Customer EIGRP

37 BGP Used an IGP

38 What do Designers do Today? Network designers balance many goals Scalability Resiliency to failure Make it easy to expand network Many “rules of thumb” in use Instability results from overloaded routers Too much routing state is bad Use routing boundaries to control spread of change Routing Design is currently an art – can we add more science?

39 Approaches? Need deeper understanding than network topology Need broader study than backbone networks Interviewing network designers isn’t enough No language/visualization exists for communicating about routing design Documentation is out-of-date or non-existent Our approach: Bottom-up white-box Start with router configuration files Reverse-engineer the routing design

40 Potential Approaches Top-down design problem How should networks be designed? First must understand what happens in real networks Bottom-up black-box approach Send probe traffic to explore network properties Very successful at recovering topology [RocketFuel] [Skitter] [Mercator] Measured topology a result of a routing design --- it does not expose the routing design itself Our approach: Bottom-up white-box Start with router configuration files Reverse-engineer the routing design

41 Router Configuration Files

42 Lots of Configuration Files