2. Exterior Routing 201 Howard C. Berkowitz hberkowi@nortelnetworks.com hcb@clark.net (703)998-5819 ESN 451-5819

3. NANOG 21 Exterior Routing tutorial 2/17/2001 2 Agenda What's the problem? —Formal and informal clue —ISP service offerings Quirks, Defnitions, and Issues ISP External Scenarios POP and other infrastructure Router requirements Playing in the Club Turning it On If there's time...full employment for consultants: path selection

4. What is the Problem to be Solved?

5. NANOG 21 Exterior Routing tutorial 2/17/2001 4 Good little boys and girls read RFC1771 and live happily ever after

6. NANOG 21 Exterior Routing tutorial 2/17/2001 5 Noah. (yawn) MMMmmmmhp? Noah. Yeahh? Build an ISP.

7. NANOG 21 Exterior Routing tutorial 2/17/2001 6 ISPs Facing End User Entry —Basic Internet Access —Hosting —Availability and QoS —Dealing with specialized access providers (DSL, CATV, etc.) —Dealing with content providers —Voice services? Improvement for Users –Improving capacity –Improving availability –Adding services –Perceptions of end-to-end SLA

8. NANOG 21 Exterior Routing tutorial 2/17/2001 7 Before the Animals Uplinks Routers User Hosts Downlinks Management Facilities HVACStaff

9. NANOG 21 Exterior Routing tutorial 2/17/2001 8 Load the Ark Policies Traffic From Downstreams Policies Traffic From Upstreams From Users Traffic AAA Traffic From Virtual Hosts

10. Quirks, Definitions and Issues

11. NANOG 21 Exterior Routing tutorial 2/17/2001 10 I said "peer," not "peer" Peer relationship 1 —Basic BGP session Peer relationship 2 —Mulual benefit customers reach one another —No monetary exchange —Each advertises customer routes Transit Provider relationship —Customer pays for service —Full routes available to customer

12. NANOG 21 Exterior Routing tutorial 2/17/2001 11 (C) O'Leary Museum and Library Association Ltd. Inc.

13. NANOG 21 Exterior Routing tutorial 2/17/2001 12 Closest Exit Routing Hot potato Paths are not optimized end-to-end Paths are optimized for each AS src dest

14. NANOG 21 Exterior Routing tutorial 2/17/2001 13 Asymmetrical Routing No guarantee that traffic leaving your AS at one point Will return at the same point Remember —Each AS in both directions makes decisions on its information

15. ISP Scenarios

16. NANOG 21 Exterior Routing tutorial 2/17/2001 15 Basic Internet Access ISP POP2POP3 Core POP1 Hosted Servers Internal Servers ISP #2ISP #1 /18 /16 8x/23 To 70-90% of customers Default route To 5-10% of customers Partial routes To 10% of customers Full routes From customers Few # public routes ??? VPN

17. NANOG 21 Exterior Routing tutorial 2/17/2001 16 Bilateral Peering Big ISP 1 Big ISP 2 eBGP Relationship Exchange of customer routes only Some aggregation No infrastructure routes Highest bandwidth requirement "Tier 1 Provider" Does not buy transit service from anyone Has default-free routers Gets all routes from bilateral/multilateral peering Total RIB size of 1.3-1.5 * DefaultFreeZone (D)

18. NANOG 21 Exterior Routing tutorial 2/17/2001 17 Large Content Provider 10/100 Server L4 distribution GE ports Firewall, etc. Provider Server L7 Distribution L3 Path Determination Sometimes bandwidth limited Provider may be default free Often high touch processing limited Possible SLA and VPN agreements May participate in content distribution, caching

19. NANOG 21 Exterior Routing tutorial 2/17/2001 18 Layer 2 Fabric ISP 1 ISP 2 ISP 3 ISP 4 ISP 5 ISP 6 Route Server Multilateral Peering eBGP Relationships Depending on exchange rules Exchange of customer routes only Most common case Some aggregation No infrastructure routes Some ISPs buy transit services Can receive full routes Private peerings Largest carriers tend to avoid due to congestion ISPs can peer with route server rather than a mesh of ISPs May be done to reduce BGP peers Or simply for statistics collection

20. NANOG 21 Exterior Routing tutorial 2/17/2001 19 Special Case: Local Exchanges Entry —Who's in charge? —Connectivity —Facilities —Allow content providers? —Allow end users? —Peering model? —Supplementary services? Improvements Layer 2? 3? Fabric ISP 1 ISP 2 ISP 3 ISP 4 ISP 5 ISP 6 Route Server

21. POP and Other Internal Design

22. NANOG 21 Exterior Routing tutorial 2/17/2001 21 Typical Basic POP Implementation Gigabit Ethernet Frame Interfaces ATM Interfaces PSTN LAN Switch Management Servers Access Server Dialup Customers Router Fabric ISP Core Router 1 ISP Core Router 2 Dedicated Customers Customer Site Routers Frame DS3 2x/25 32x/30 Dedicated Customers Customer Site Router Full DS3 1 per POP25 per POP 450 users per POP /18

23. NANOG 21 Exterior Routing tutorial 2/17/2001 22 Transit Provider POP, Intra-POP Design Alternatives 1. POP is a route reflector cluster Core is higher-level cluster 2. Each POP is a private or public AS Full mesh iBGP or route reflectors inside POP Confederation between POPs 3. IGP within POP Controlled redistribution inside POP to BGP Prefer intra-POP of same metric Access Router Access Router POP Router POP Router

24. NANOG 21 Exterior Routing tutorial 2/17/2001 23 Public AS POP Confederations POP Confederations POP AS65000POP AS65111POP AS65222

25. NANOG 21 Exterior Routing tutorial 2/17/2001 24 Public AS POP Reflectors POP Reflectors POP AS65000POP AS65111POP AS65222

26. NANOG 21 Exterior Routing tutorial 2/17/2001 25 Open Access/Specialized Access Layer 1/2 Fabric Subscribers ISP 1 ISP 2 ISP 3 Internal Routed Network Tunnel Server Content Servers

27. NANOG 21 Exterior Routing tutorial 2/17/2001 26 Internal Routing & Switching Tunneled Addressing CLE Data Provider 1 Voice Provider 1 Enterprise VPN NAS CLE Access Gateway Data Provider 2 L2TP, Differv High VoIP Access OAM address space PPPoE or GRE ISP address space Data 1 Data 2 VPN DHCP DNS

28. Router Requirements Big part of the solution...but not all.

29. NANOG 21 Exterior Routing tutorial 2/17/2001 28 Routing Paradigms Number of Routes Forwarding Bandwidth Hello Processing Number of Interfaces Policy Analysis QoS Awareness LowHighMediumLowHighMedium LowMediumHigh MediumHighMedium End to EndEtE & PHBPHB LowHighLow L4/7 Processing MediumHighLow EnterpriseEdgeCore

30. NANOG 21 Exterior Routing tutorial 2/17/2001 29 Observations on Routing Table Size Global default-free table continues to grow exponentially —96509 routes as of Tony Bates' CIDR report 2/11/2001 —Let the default routing table size be D Large provider often has 1.3 to 1.5 D active routes —additional routes are more-specific customer & internal —may also have substantial numbers of inactive routes

31. NANOG 21 Exterior Routing tutorial 2/17/2001 30 Growth in Global Routing Table Size 184K 368K Sep01Sep01 Sep02Sep02 Sep03Sep03 Sep04Sep04 736K 85K public

32. NANOG 21 Exterior Routing tutorial 2/17/2001 31 Growth in Typical Tier 1 Routing Table Size (external + customer, not infrastructure) 276 552 Sep01Sep01 Sep02Sep02 Sep03Sep03 Sep04Sep04 1104 85K public 42K internal

33. NANOG 21 Exterior Routing tutorial 2/17/2001 32 Observation: More than Routes Customer routes Paths per route Route validity

34. NANOG 21 Exterior Routing tutorial 2/17/2001 33 Convergence Global routing system Intra-AS Single Router

35. NANOG 21 Exterior Routing tutorial 2/17/2001 34 Single Router Convergence Initialization —Time to add new route —Time to add better route —Time to withdraw route —Time to withdraw and replace route Parameters —Matrix: number of peers versus –Routes advertised –Routes accepted Performance Modifiers —Route filtering —Route flapping —Packet vs. route filtering draft-berkowitz-bgpcon-0x.txt

36. NANOG 21 Exterior Routing tutorial 2/17/2001 35 Distinguish among cases Failover of link or router between customer and provider Rerouting to intranet/adjacent provider resources Rerouting to arbitrary internet destnation More multihoming in next tutorial

37. S-T-R-E-T-C-H

38. Joining the Club

39. NANOG 21 Exterior Routing tutorial 2/17/2001 38 More than Just Addresses, Protocol... Routing Registry Route objects AS objects Maintainer objects Routing System Configs Customer DNS NAT Hosts Specify Policy RouteTrackService ISP with Prefixes Allocate Directories SWIP Reverse DNS Address Registry Route Registry

40. NANOG 21 Exterior Routing tutorial 2/17/2001 39 Complexity BGP itself is fairly simple Additional attributes it carries are more complex Policy actions taken inside router (BGP sender or receiver) far more complex than the protocol itself

41. NANOG 21 Exterior Routing tutorial 2/17/2001 40 "BGP Transmits Policies"

42. NANOG 21 Exterior Routing tutorial 2/17/2001 41 Operational Relationships 1 Addresses and Delegation Address authority Address delegation Prefixes Hosts DNS Reverse DNS

43. NANOG 21 Exterior Routing tutorial 2/17/2001 42 Obtain routable address space Apply to registry —RIPE, APNIC, ARIN —If immediate need for /19 or /20* Obtain addresses from upstream ISP —If /19 or /20 cannot be justified Registry needs —Network design —Justification for address space

44. NANOG 21 Exterior Routing tutorial 2/17/2001 43 Origination vs. Advertising 128.0.0.0/20 /23 POP Dialups /23 Internal /23 Customers /23 Customers /25 32 * /30 32 * /28 /24 /25 AS 65000 192.0.0.0/16 AS64444 an AS65000 Customer AS65000 128.0.0.0/19 AS65000 192.0.0.0/16 AS64444

45. NANOG 21 Exterior Routing tutorial 2/17/2001 44 Aggregating your Own Traffic 128.0.0.0/20 /23 POP Dialups /23 Internal /23 Customers /23 Customers /25 32 * /30 32 * /28 /24 /25 AS65000 128.0.0.0/19 Suppress more specific routes unless required by multihoming

46. NANOG 21 Exterior Routing tutorial 2/17/2001 45 Advertising with NO-EXPORT AS63333 64.0.0.0/12 Assigns 64.0.0.0/22 Assigns 64.0.4.0/22 AS62222 AS61111 AS61000 96.1.0.0/16 Advertises 64.0.0.0/22 NO-EXPORT Advertises 64.0.4.0/22 NO-EXPORT 96.1.0.0/16

47. NANOG 21 Exterior Routing tutorial 2/17/2001 46 Aggregation is better than Aggravation Blackhole routes for your blocks — Avoid more-specifics — Use NO-EXPORT when controlling load to upstream Encourage customers to aggregate — Proxy aggregation hard to administer Understand which blocks you can advertise — And do ingress/egress filtering

48. NANOG 21 Exterior Routing tutorial 2/17/2001 47 Preparing for Address Request (1) Address requirements of services are you offering Dynamic addressing —Dialup —Residential broadband Private addressing —Enterprises homed only to you —Dialup/broadband not offering servers Globally addressable

49. NANOG 21 Exterior Routing tutorial 2/17/2001 48 Prepare for Address Request (2) An ISP Topology POP1 1 internal LAN 100 Dial Ports 8 small LANs 1 med. LAN POP2 1 internal LAN 100 Dial Ports 8 small LANs 1 med. LAN POP3 1 internal LAN 100 Dial Ports 8 small LANs 1 med. LAN POP4 1 internal LAN 100 Dial Ports 8 small LANs 1 med. LAN Core Router 1 Core Router 2 Hosting Farm 1Hosting Farm 2 Infrastructure Servers Switch

50. NANOG 21 Exterior Routing tutorial 2/17/2001 49 Establishing an AS (1) AS Number Request In request to AS number registry —Administrative and technical contacts —Autonomous system name —Router description —Deployment schedule —Networks (by name) connected by the router(s) —Internet addresses of the routers

51. NANOG 21 Exterior Routing tutorial 2/17/2001 50 Establishing an AS (2) Registering in Routing Registry Minimum requirements —Maintainer object —AS object —Route object (s)

52. NANOG 21 Exterior Routing tutorial 2/17/2001 51 Establishing an AS (3) Operational deployment Build configuration —Policy implementation —Ingress/egress filtering Establish security procedures Start BGP connections

53. NANOG 21 Exterior Routing tutorial 2/17/2001 52 Routing Registry Objects Basic —AS —Route —Maintainer Additional —Inter-AS Network —Community —Router Refinements

54. NANOG 21 Exterior Routing tutorial 2/17/2001 53 Operational Relationships 3: Registries, Domains, etc. Address authority Address delegation Prefixes Hosts Route objects AS DNS Reverse DNS

55. NANOG 21 Exterior Routing tutorial 2/17/2001 54 Autonomous System Basis of exterior routing AS originate routes for some prefixes they want to be visible AS advertise routes to one another —Advertisement may not contain all addresses —Not all advertisements need be accepted

56. NANOG 21 Exterior Routing tutorial 2/17/2001 55 Current AS Definition RFC 1930 Connected group of IP CIDR blocks Run by one or more network operators Single routing policy —announced to the general Internet —announced with BGP-4

57. NANOG 21 Exterior Routing tutorial 2/17/2001 56 AS Number 16 bit number —32 bit under discussion Numbers assigned by registries —Routing policy should be stored in registry —ISPs can mirror routing registry -- place for sensitive data Private ASNs —64512 through 65535 —Private AS stripping, confederations

58. NANOG 21 Exterior Routing tutorial 2/17/2001 57 Operational Relationships 2: Addesses and Autonomous Systems Address authority Address delegation Prefixes Hosts AS DNS Reverse DNS

59. NANOG 21 Exterior Routing tutorial 2/17/2001 58 Full Employment for Consultants: Policies are inside Routers Advertising Policies —Outbound to other AS —BGP advertisement sources —Outbound route filters —Route must be in internal routing table Acceptance Policies —Inbound AS filters —Inbound route filters

60. NANOG 21 Exterior Routing tutorial 2/17/2001 59 Stop! What are you going to Advertise? Routes Assigned/Allocated to You Routes Assigned/Allocated to Customers Routes for which you provide Transit

61. NANOG 21 Exterior Routing tutorial 2/17/2001 60 Advertising Affects The way the world sees you/sends to you Binary —Routes to which you provide routing Quantitative Preferences —Multi-Exit Discriminators to your Neighbors —AS Path Manipulation to all

62. NANOG 21 Exterior Routing tutorial 2/17/2001 61 Routes Eligible to Advertise Are reachable by your IGPor static routes Unless they are black holes —Which conceptiually are reachable Do not advertise —Spoofed source addresses —Your internal addresses —RFC1918 space —Known rogues? –RBL?

63. NANOG 21 Exterior Routing tutorial 2/17/2001 62 Stop! What are you going to Accept? It depends Only those routes you will do something about Otherwise default

64. NANOG 21 Exterior Routing tutorial 2/17/2001 63 Do Not Accept RFC1918 source or destination Unexpected sources not assigned/allocated to peers Your internal addresses from peers

65. Turning it On

66. NANOG 21 Exterior Routing tutorial 2/17/2001 65 BGP Configuration Overview Plans and policies first! Define system of BGP speakers Specific BGP speaker configuration —Identifier —BGP process —Neighbors —NLRI to advertise —Filters and other policy mechanisms Cisco commands used as examples

67. NANOG 21 Exterior Routing tutorial 2/17/2001 66 Policy Implementation Flow Main BGP RIB Adj RIB Out Outgo- ing Adj RIB In Incom- ing Main RIB/ FIB IGPs Static & HW Info

68. NANOG 21 Exterior Routing tutorial 2/17/2001 67 AS1 R1 AS21 R1 AS1 R1 AS21 R1 AS1 R1 AS21 R1 AS1 R2 AS21 R2 All equivalent from a policy standpoint! Policy vs. Protocol Flow

69. NANOG 21 Exterior Routing tutorial 2/17/2001 68 BGP Configurations Know global information (AS, policies, etc.) Establish router ID Create BGP process Identify internal and external peers

70. NANOG 21 Exterior Routing tutorial 2/17/2001 69 Router ID and loopback interface interface loopback 0 ip address 192.168.0.1 255.255.255.0

71. Refining the Configuration Single and Multiple Links to a Single Provider

72. NANOG 21 Exterior Routing tutorial 2/17/2001 71 The BGP Tunnel Serial 0 Serial 1 Loop 0 ebgp-multihop needed when neighbor is not on same subnet

73. NANOG 21 Exterior Routing tutorial 2/17/2001 72 Load Balancing 1: IP Level to Single Provider Router Serial 0 Serial 1 Loop 0 Customer AS Provider AS

74. NANOG 21 Exterior Routing tutorial 2/17/2001 73 Load Balancing 1: Multiple Routers Customer AS Provider AS

75. NANOG 21 Exterior Routing tutorial 2/17/2001 74 Another Non-BGP Alternative OSPF Routing Domain Default Route (0.0.0.0/0) Metric Type 1 Equal Metrics Static routes D1-A0 ASBR1 D1-A0 ASBR2 ISP 1

76. NANOG 21 Exterior Routing tutorial 2/17/2001 75 Multiple OSPF Defaults ISP 1 POP ISP 2 POP Default Route (0.0.0.0/0) Metric Type 2 Higher Metric to ISP 2 (Backup) Static routes D1-A0 ASBR1 D1-A0 ASBR2

77. NANOG 21 Exterior Routing tutorial 2/17/2001 76 Blackhole Route Establish static route to your block(s) ip route 1.2.3.4 255.255.240.0 null0 Redistribute/import into BGP Suppress more-specific prefix advertising

78. NANOG 21 Exterior Routing tutorial 2/17/2001 77 Effects of Blackholing No route flapping outside your AS —If your internal routes go up or down Incoming traffic for specific routes that are down —Doesn’t match any internal route —Automatically discarded without concerning anyone else

79. BGP Path Selection

80. NANOG 21 Exterior Routing tutorial 2/17/2001 79 Next Hop Access R2 R1 X Advertised route via R1 Advertised route via R2

81. NANOG 21 Exterior Routing tutorial 2/17/2001 80 Scope: MED vs. Local Preference vs. Weight Weight Local Preference Weight AS1 AS2 MED

82. NANOG 21 Exterior Routing tutorial 2/17/2001 81 Administrative Weight (Cisco extension) Advertised route via R1 Advertised route via R2 Rules in this router set R1 weight to 100, R2 weight to 500 R2 R1 X

83. NANOG 21 Exterior Routing tutorial 2/17/2001 82 Weight example for load sharing Primary ISP Default local preference 500 All routes ^ AS_Backup + local preference 100 Backup ISP Default local preference 200

84. NANOG 21 Exterior Routing tutorial 2/17/2001 83 Tiebreaker for Equal Weight: Local Preference R2 R1 Advertised route via R1, local preference 100 Advertised route via R2, local preference 500

85. NANOG 21 Exterior Routing tutorial 2/17/2001 84 Local Preference example for load sharing Primary ISP Default local preference 500 All routes ^ AS_Backup + local preference 100 Backup ISP Default local preference 200

86. NANOG 21 Exterior Routing tutorial 2/17/2001 85 Prefer locally originated routes R2 R1 Advertised route via R1 Locally defined via R2

87. NANOG 21 Exterior Routing tutorial 2/17/2001 86 AS Path 1 1 12 24 124 5 1245

88. NANOG 21 Exterior Routing tutorial 2/17/2001 87 Shortest AS Path (Cisco extension) R2 R1 AS Route AS Route

89. NANOG 21 Exterior Routing tutorial 2/17/2001 88 Full Employment For Consultants: Interpreting AS Path Default assumption: local preference set based on AS_PATH Cisco considers it as part of the algorithm

90. NANOG 21 Exterior Routing tutorial 2/17/2001 89 AS Path Prepending Applies to routes you advertise Makes them less attractive to others Increases AS_PATH length —your AS put in the path twice

91. NANOG 21 Exterior Routing tutorial 2/17/2001 90 Limitations of Prepending 6 1 11 1 12 2 3 113 4 124 5 1245

92. NANOG 21 Exterior Routing tutorial 2/17/2001 91 Route Learned from eBGP Route Learned from iBGP R2 R1 External Paths Preferred

93. NANOG 21 Exterior Routing tutorial 2/17/2001 92 Remote AS MED=100 MED=500 R2 R1 Lowest MED

94. NANOG 21 Exterior Routing tutorial 2/17/2001 93 Full Employment For Consultants: Weight, Local Preference & MED HIGHER value wins —Weight —Local preference LOWER value wins —MED —Cisco default: route with no MED preferred —IETF: route with no MED least preferred

95. NANOG 21 Exterior Routing tutorial 2/17/2001 94 Full Employment For Consultants: Scope of MED Default assumption: —MEDs only compared between exits to the same adjacent AS Alternate: always-compare-MED —Useful at exchange points, possibly private peerings —Cisco knob

96. NANOG 21 Exterior Routing tutorial 2/17/2001 95 Closest Neighbor IGP metric to R1=100 IGP metric to R1=500 R2 R1

97. NANOG 21 Exterior Routing tutorial 2/17/2001 96 Lowest BGP router ID R2 2.2.2.2 R1 1.1.1.1