1. Nick Feamster Interdomain Routing Correctness and Stability

2. 2 Is correctness really that important?

3. 3 The Internet is increasingly becoming part of the mission-critical Infrastructure (a public utility!). Big problem: Very poor understanding of how to manage it.

4. 4 Why does routing go wrong? Complex policies –Competing / cooperating networks –Each with only limited visibility Large scale –Tens of thousands networks –…each with hundreds of routers –…each routing to hundreds of thousands of IP prefixes

5. 5 What can go wrong? Two-thirds of the problems are caused by configuration of the routing protocol Some things are out of the hands of networking research But…

6. 6 Review: Simple operation… Route Advertisement Autonomous Systems (ASes) Session DestinationNext-hopAS Path 18.0.0.0/8 192.5.89.89 66.250.252.44 1 3356 3 174 3 MIT

7. 7 …but complex configuration! Which neighboring networks can send traffic Where traffic enters and leaves the network How routers within the network learn routes to external destinations Flexibility for realizing goals in complex business landscape FlexibilityComplexity Traffic Route No Route

8. 8 Configuration Semantics Ranking: route selection Dissemination: internal route advertisement Filtering: route advertisement Customer Competitor Primary Backup

9. 9 What types of problems does configuration cause? Persistent oscillation (today’s reading) Forwarding loops Partitions “Blackholes” Route instability …

10. 10 These problems are real “…a glitch at a small ISP… triggered a major outage in Internet access across the country. The problem started when MAI Network Services...passed bad router information from one of its customers onto Sprint.” -- news.com, April 25, 1997 UUNet Florida Internet Barn Sprint

11. 11 These problems are real “…a glitch at a small ISP… triggered a major outage in Internet access across the country. The problem started when MAI Network Services...passed bad router information from one of its customers onto Sprint.” -- news.com, April 25, 1997 “Microsoft's websites were offline for up to 23 hours...because of a [router] misconfiguration…it took nearly a day to determine what was wrong and undo the changes.” -- wired.com, January 25, 2001 “WorldCom Inc…suffered a widespread outage on its Internet backbone that affected roughly 20 percent of its U.S. customer base. The network problems…affected millions of computer users worldwide. A spokeswoman attributed the outage to "a route table issue." -- cnn.com, October 3, 2002 "A number of Covad customers went out from 5pm today due to, supposedly, a DDOS (distributed denial of service attack) on a key Level3 data center, which later was described as a route leak (misconfiguration).” -- dslreports.com, February 23, 2004

12. 12 Several “Big” Problems a Week

13. 13 Why is routing hard to get right? Defining correctness is hard Interactions cause unintended consequences –Each network independently configured –Unintended policy interactions Operators make mistakes –Configuration is difficult –Complex policies, distributed configuration

14. 14 Correctness Specification Safety The protocol converges to a stable path assignment for every possible initial state and message ordering The protocol does not oscillate

15. 15 Safety: No Persistent Oscillation 1 23 1 3 0 1 0 3 2 0 3 0 2 1 0 2 0 0 Varadhan, Govindan, & Estrin, “Persistent Route Oscillations in Interdomain Routing”, 1996

16. 16 Strawman: Global Policy Check Require each AS to publish its policies Detect and resolve conflicts Problems: ASes typically unwilling to reveal policies Checking for convergence is NP-complete Failures may still cause oscillations

17. 17 Think Globally, Act Locally Key features of a good solution –Safety: guaranteed convergence –Expressiveness: allow diverse policies for each AS –Autonomy: do not require revelation/coordination –Backwards-compatibility: no changes to BGP Local restrictions on configuration semantics –Ranking –Filtering

18. 18 Main Idea of Today’s Paper Permit only two business arrangements –Customer-provider –Peering Constrain both filtering and ranking based on these arrangements to guarantee safety Surprising result: these arrangements correspond to today’s (common) behavior Gao & Rexford, “Stable Internet Routing without Global Coordination”, IEEE/ACM ToN, 2001

19. 19 Relationship #1: Customer-Provider Filtering –Routes from customer: to everyone –Routes from provider: only to customers providers customer From the customer To other destinations advertisements traffic From other destinations To the customer customer providers

20. 20 Relationship #2: Peering Filtering –Routes from peer: only to customers –No routes from other peers or providers advertisements traffic customer peer

21. 21 Rankings Routes from customers over routes from peers Routes from peers over routes from providers provider peer customer

22. 22 Additional Assumption: Hierarchy Disallowed!

23. 23 Safety: Proof Sketch System state: the current route at each AS Activation sequence: revisit some router’s selection based on those of neighboring ASes

24. 24 Activation Sequence: Intuition Activation: emulates a message ordering –Activated router has received and processed all messages corresponding to the system state “Fair” activation: all routers receive and process outstanding messages

25. 25 Safety: Proof Sketch State: the current route at each AS Activation sequence: revisit some router’s selection based on those of neighboring ASes Goal: find an activation sequence that leads to a stable state Safety: satisfied if that activation sequence is contained within any “fair” activation sequence

26. 26 Proof, Step 1: Customer Routes Activate ASes from customer to provider –AS picks a customer route if one exists –Decision of one AS cannot cause an earlier AS to change its mind An AS picks a customer route when one exists

27. 27 Proof, Step 2: Peer & Provider Routes Activate remaining ASes from provider to customer –Decision of one Step-2 AS cannot cause an earlier Step- 2 AS to change its mind –Decision of Step-2 AS cannot affect a Step-1 AS AS picks a peer or provider route when no customer route is available

28. 28 Ranking and Filtering Interactions Allowing more flexibility in ranking –Allow same preference for peer and customer routes –Never choose a peer route over a shorter customer route … at the expense of stricter AS graph assumptions –Hierarchical provider-customer relationship (as before) –No private peering with (direct or indirect) providers Peering

29. 29 Some problems Requires acyclic hierarchy (global condition) Cannot express many business relationships Abovenet Verio PSINet Sprint Customer Question: Can we relax the constraints on filtering? What happens to rankings?

30. 30 Other Possible Local Rankings Accept only next-hop rankings –Captures most routing policies –Generalizes customer/peer/provider –Problem: system not safe Accept only shortest hop count rankings –Guarantees safety under filtering –Problem: not expressive 1 3 2 3*,2*,0* 2*,1*,0*1*, 3*, 0* Feamster, Johari, & Balakrishnan, “Implications of Autonomy for the Expressiveness of Policy Routing”, SIGCOMM 2005

31. 31 What Rankings Violate Safety? Theorem. Permitting paths of length n+2 over paths of length n will violate safety under filtering. Theorem. Permitting paths of length n+1 over paths of length n will result in a dispute wheel. Feamster, Johari, & Balakrishnan, “Implications of Autonomy for the Expressiveness of Policy Routing”, SIGCOMM 2005

32. 32 What about properties of resulting paths, after the protocol has converged? We need additional correctness properties.

33. 33 Correctness Specification Safety The protocol converges to a stable path assignment for every possible initial state and message ordering The protocol does not oscillate Path Visibility Every destination with a usable path has a route advertisement Route Validity Every route advertisement corresponds to a usable path Example violation: Network partition Example violation: Routing loop If there exists a path, then there exists a route If there exists a route, then there exists a path

34. 34 Path Visibility: Internal BGP (iBGP) “iBGP” Default: “Full mesh” iBGP. Doesn’t scale. Large ASes use “Route reflection” Route reflector: non-client routes over client sessions; client routes over all sessions Client: don’t re-advertise iBGP routes.

35. 35 iBGP Signaling: Static Check Theorem. Suppose the iBGP reflector-client relationship graph contains no cycles. Then, path visibility is satisfied if, and only if, the set of routers that are not route reflector clients forms a clique. Condition is easy to check with static analysis.

36. 36 How do we guarantee these additional properties in practice?

37. 37 Today: Reactive Operation Problems cause downtime Problems often not immediately apparent What happens if I tweak this policy…? ConfigureObserve Wait for Next Problem Desired Effect? Revert No Yes

38. 38 Goal: Proactive Operation Idea: Analyze configuration before deployment Configure Detect Faults Deploy rcc Many faults can be detected with static analysis. Feamster & Balakrishnan, “Detecting BGP Configuration Faults with Static Analysis”, NSDI 2005

39. 39 “rcc” rcc Overview Normalized Representation Correctness Specification Constraints Faults Analyzing complex, distributed configuration Defining a correctness specification Mapping specification to constraints Challenges Distributed router configurations (Single AS) Feamster & Balakrishnan, “Detecting BGP Configuration Faults with Static Analysis”, NSDI 2005

40. 40 rcc Implementation PreprocessorParser Verifier Distributed router configurations Relational Database (mySQL) Constraints Faults (Cisco, Avici, Juniper, Procket, etc.) Feamster & Balakrishnan, “Detecting BGP Configuration Faults with Static Analysis”, NSDI 2005

41. 41 Summary: Faults across 17 ASes Route ValidityPath Visibility Every AS had faults, regardless of network size Most faults can be attributed to distributed configuration

42. 42 rcc: Take-home lessons Static configuration analysis uncovers many errors Major causes of error: –Distributed configuration –Intra-AS dissemination is too complex –Mechanistic expression of policy http://nms.csail.mit.edu/rcc/ About 100 downloads (70 network operators)

43. 43 Two Philosophies This lecture: Accept the Internet as is. Devise “band-aids”. Another direction: Redesign Internet routing to guarantee safety, route validity, and path visibility

44. 44 Preventing Errors in the First Place iBGP RCP After: RCP gets “best” iBGP routes (and IGP topology) iBGP eBGP Before: conventional iBGP Feamster et al., “The Case for Separating Routing from Routers”, SIGCOMM FDNA, 2004 Caesar et al., “Design and Implementation of a Routing Control Platform”, NSDI, 2005

45. 45

46. 46 Ranking Configuration Syntax (Example) Dissemination router bgp 7018 neighbor 192.0.2.10 remote-as 65000 neighbor 192.0.2.10 route-map IMPORT in neighbor 192.0.2.20 remote-as 7018 neighbor 192.0.2.20 route-reflector-client ! route-map IMPORT permit 1 match ip address 199 set local-preference 80 ! route-map IMPORT permit 2 match as-path 99 set local-preference 110 ! route-map IMPORT permit 3 set community 7018:1000 ! ip as-path access-list 99 permit ^65000$ access-list 199 permit ip host 192.0.2.0 host 255.255.255.0 access-list 199 permit ip host 10.0.0.0 host 255.0.0.0

47. 47 Why is Routing Hard to Get Right? Defining correctness is hard Operators make mistakes –Configuration is difficult –Complex policies, distributed configuration Interactions cause unintended consequences –Each network independently configured –Unintended policy interactions

48. 48 Which faults does rcc detect? Faults found by rcc Latent faults Potentially active faults End-to-end failures

49. 49 Normalizing Router Configuration Hundreds of routers distributed across an AS Thousands to tens of thousands of lines per router Many ways to express identical policy Challenge: Solution: Express configuration with centralized tables Check constraints by issuing queries on tables Sessions Filters Rankings

50. 50 Route Validity: Consistent Export Malice/deception iBGP signaling partition Inconsistent export policy neighbor 10.1.2.3 route-map PEER permit 10 set prepend 123 neighbor 10.4.5.6 route-map PEER permit 10 set prepend 123 123 Possible Causes 10.1.2.3 4561 10.4.5.6 4562 Neighbor AS Export 11 123 ExportClause Prepend 21 123 123 Policy normalization makes comparison easy.

51. 51 Inconsistent Export Observed at AT&T Feamster et al., “BorderGuard: Detecting Cold Potatoes from Peers”. ACM IMC, October 2004. 15% of destinations inconsistent for >4 days Percentage of destinations with inconsistent routes Percentage of time

52. 52 Example: “Bogon” Routes Feamster et al., “An Empirical Study of ‘Bogon’ Route Advertisements”. ACM CCR, January 2005.

53. 53 rcc Interface

54. 54 Parsing Configuration

55. 55 List of Faults