Efficient Network Reachability Analysis using a Succinct Control Plane Representation Seyed K. Fayaz, Tushar Sharma, Ari Fogel Ratul Mahajan, Todd Millstein, Vyas Sekar George Varghese

Network configuration is hard ??? Network operator Reachability policy: A can talk to B Reality What the network does R3 R4 R1 A R2 B network Does the network do what we want it to do?

State of the art in network verification Data plane verification Prior work: HSA, NSDI’12 ATPG, CoNext’12 NOD, NSDI’15 … DP3 traffic DP4 DP1 B A DP2 Network operator ✔ Can A talk to B? Reachability policy: A can talk to B Data plane (Forwarding table) ✔ DP3 DP4 R3 DP1 R4 R1 DP2 A R2 B Are we done?

The data plane keeps changing! DP3 DP3 DP3 DP4 DP4 DP4 … DP1 DP1 DP1 DP2 DP2 DP2 Can A talk to B? Can A talk to B? Can A talk to B? time Time = t1 Time = t2 Time = t3 traffic from A to B Network operator Reachability policy: A can talk to B DP3 DP3 DP3 R3 DP3 DP3 DP4 DP3 DP3 DP1 R4 R1 DP3 DP2 DP3 A R2 B

Motivating example: Reachability bug triggered by a BGP announcement Before the incident After the incident culprit W W A B A B DCA DCA DCB DCB ✔ ✔ services in 10.10.0.0/16 DCB/16 New service 10.10.1.160/28 DCB/28 ✗ DCB/16 10.10.0.0/16 Root cause: Router B’s config. had a aggregate route 10.10.0.0/16 pointing to DCB The /28 advertisement activated the aggregate route! How can we proactively find such latent reachability bugs?

A data plane is just the current incarnation of the control plane! Router configuration Prior work on control plane verification rcc, NSDI’05 Bagpipe, OOPSLA’16 ARC, SIGCOMM’16 Batfish, NSDI’15 … Route advertisement 1 (e.g., BGP advertisement) Control plane (implementation of BGP, OSPF, etc.) Route advertisement 2 (e.g., OSPF advertisement) advertisements to neighbors Route advertisement 3 (e.g., RIP advertisement) Limitations: Incomplete: Focus on just one routing protocol Unscalable: Detailed modeling of message passing Data Plane 1 <prefix P,port1> … Data Plane 3 <prefix P,port3> … Data Plane 2 <prefix P,port2> … … … Router To find latent reachability bugs, we should focus on the control plane!

Our contributions ERA: A tool for finding latent router configuration bugs in seconds based on control plane analysis Expressive-yet-tractable control plane model Scalable exploration of control plane model Implementation as an open source tool

ERA: System overview ERA Pass Fail Operator Pass router configurations reachability policies Fail ERA ERA: A tool to find latent reachability bugs due to router misconfiguration. Scope: Reachability bugs occurring in the steady state

Outline Background and motivation Design of ERA Implementation and evaluation

Challenges in control plane analysis Operator router configurations reachability policies Pass Challenge 1: Expressive and tractable model? Fail control plane model Challenge 2: Scalable exploration model exploration ERA

Challenge 1: Expressive and tractable control plane model Operator router configurations reachability policies Pass Challenge 1: Expressive and tractable model? Fail control plane model Challenge 2: Scalable exploration model exploration ERA

✗ ✗ ? ? ✔ ✔ ✔ ✔ A route as a succinct bit-vector Router control plane Control plane I/ O model Expressive Tractable Actual protocol’s messages (e.g., Batfish, NSDI’15) ✗ ✔ Protocol agnostic I/O model (e.g., ARC, SIGCOMM’16) ✗ ✔ ✔ ✔ Route as a compact bit vector Dst IP (32 bits) Dst mask (5 bits) Administrative distance (4 bits) Protocol attributes (87 bits) A route as a succinct and unifying control plane I/O unit

? ? Control plane as a fast pipeline of boolean operators: Intuition _ Router control plane ? Why not actual router’s code?  Hard to explore Router as a fast route processing pipeline X3 X2 X1 X0 protocol attribute An example: prefix admin. distance (RIP=1, BGP=0) _ _ router config. ? X1X0 Router control plane   X3X1X0∨X2X1X0 input set RIP attr. to 1 RIP static 10 _ _ _ _ _ _ _ _ _ _ _ X1X0 X1X0 X1 = X1X0 X3X2 X1X0 = X3X1X0∨X2X1X0 X3X1X0∨X2X1X0

Control plane as a fast pipeline of boolean operators: Complete pipeline BDD of input routes AND with supported protocols 1 2 3 OR with routes originated by router Apply input filters AND with NEG. of static routes 6 OR with aggregate routes 5 OR with redistributed routes 4 Select best route per dst prefix 7 8 BDD of output routes Apply output filters Compact representation of a collection of routes using Binary Decision Diagrams (BDDs) The pipeline captures key control plane behaviors that are source of many bugs.

Challenge 2: Scalable control plane exploration Operator router configurations reachability policies Pass Challenge 1: Expressive and tractable model? Fail control plane model Challenge 2: Scalable exploration model exploration ERA

Reachability analysis by exploring the control plane model Intuition: To see what traffic can reach from A to B, just find out what route prefixes advertised by B can reach A! True ? True ? R3 R4 route advertisements (represented as BDDs) RA R1 A R2 RB B traffic R5 Network Environment True ? Prepare for the worst! Optimizations to scale control plane exploration: Equivalence classes of routes Fast AVX2 instructions to implement conjunction/disjunction

Outline Background and motivation Design of ERA Implementation and evaluation

Implementation Fail Pass https://github.com/Network- verification/ERA Router config. parser from Batfish (NSDI’15) parsing Cisco and Juniper Control plane model (Custom Java code and BDD library) Network topology (Custom format) Operator Reachability policies (e.g., AB, valley-free, blackhole) Model Exploration (Java and Intel AVX2 optimizations) Environment assumptions (default: “all routes”) Pass Fail https://github.com/Network- verification/ERA

Evaluation ERA is effective in finding latent reachability bugs Found known and new bugs in synthetic scenarios Found known and new bugs in real scenarios These bugs were caused by router misconfiguration wrt Incorrect route redistribution Incorrect route aggregation Unintended cross-protocol effects Interaction between SDN and traditional routing protocols … ERA is fast and scalable ERA analyzes networks with over 1,600 routers in < 7 seconds Finding a latent bug using state of the art data plane analysis techniques in a 2-router network would take up to 1022 days!

Conclusions Problem: How to find latent network reachability bugs? Data plane verification is fundamentally limited Current control plane analysis tools are incomplete or unscalable ERA: A fast control plane analysis tool: Modeling control plane’s I/O as compact BDDs Modeling control plane processing logic using fast boolean arithmetic ERA can help find latent bugs and is scalable