Exterior Routing 201 Howard C. Berkowitz (703) ESN

NANOG 21 Exterior Routing tutorial 2/17/ Agenda What's the problem? —Formal and informal clue —ISP service offerings Quirks, Defnitions, and Issues ISP External Scenarios POP and other infrastructure Router requirements Playing in the Club Turning it On If there's time...full employment for consultants: path selection

What is the Problem to be Solved?

NANOG 21 Exterior Routing tutorial 2/17/ Good little boys and girls read RFC1771 and live happily ever after

NANOG 21 Exterior Routing tutorial 2/17/ Noah. (yawn) MMMmmmmhp? Noah. Yeahh? Build an ISP.

NANOG 21 Exterior Routing tutorial 2/17/ ISPs Facing End User Entry —Basic Internet Access —Hosting —Availability and QoS —Dealing with specialized access providers (DSL, CATV, etc.) —Dealing with content providers —Voice services? Improvement for Users –Improving capacity –Improving availability –Adding services –Perceptions of end-to-end SLA

NANOG 21 Exterior Routing tutorial 2/17/ Before the Animals Uplinks Routers User Hosts Downlinks Management Facilities HVACStaff

NANOG 21 Exterior Routing tutorial 2/17/ Load the Ark Policies Traffic From Downstreams Policies Traffic From Upstreams From Users Traffic AAA Traffic From Virtual Hosts

Quirks, Definitions and Issues

NANOG 21 Exterior Routing tutorial 2/17/ I said "peer," not "peer" Peer relationship 1 —Basic BGP session Peer relationship 2 —Mulual benefit customers reach one another —No monetary exchange —Each advertises customer routes Transit Provider relationship —Customer pays for service —Full routes available to customer

NANOG 21 Exterior Routing tutorial 2/17/ (C) O'Leary Museum and Library Association Ltd. Inc.

NANOG 21 Exterior Routing tutorial 2/17/ Closest Exit Routing Hot potato Paths are not optimized end-to-end Paths are optimized for each AS src dest

NANOG 21 Exterior Routing tutorial 2/17/ Asymmetrical Routing No guarantee that traffic leaving your AS at one point Will return at the same point Remember —Each AS in both directions makes decisions on its information

ISP Scenarios

NANOG 21 Exterior Routing tutorial 2/17/ Basic Internet Access ISP POP2POP3 Core POP1 Hosted Servers Internal Servers ISP #2ISP #1 /18 /16 8x/23 To 70-90% of customers Default route To 5-10% of customers Partial routes To 10% of customers Full routes From customers Few # public routes ??? VPN

NANOG 21 Exterior Routing tutorial 2/17/ Bilateral Peering Big ISP 1 Big ISP 2 eBGP Relationship Exchange of customer routes only Some aggregation No infrastructure routes Highest bandwidth requirement "Tier 1 Provider" Does not buy transit service from anyone Has default-free routers Gets all routes from bilateral/multilateral peering Total RIB size of * DefaultFreeZone (D)

NANOG 21 Exterior Routing tutorial 2/17/ Large Content Provider 10/100 Server L4 distribution GE ports Firewall, etc. Provider Server L7 Distribution L3 Path Determination Sometimes bandwidth limited Provider may be default free Often high touch processing limited Possible SLA and VPN agreements May participate in content distribution, caching

NANOG 21 Exterior Routing tutorial 2/17/ Layer 2 Fabric ISP 1 ISP 2 ISP 3 ISP 4 ISP 5 ISP 6 Route Server Multilateral Peering eBGP Relationships Depending on exchange rules Exchange of customer routes only Most common case Some aggregation No infrastructure routes Some ISPs buy transit services Can receive full routes Private peerings Largest carriers tend to avoid due to congestion ISPs can peer with route server rather than a mesh of ISPs May be done to reduce BGP peers Or simply for statistics collection

NANOG 21 Exterior Routing tutorial 2/17/ Special Case: Local Exchanges Entry —Who's in charge? —Connectivity —Facilities —Allow content providers? —Allow end users? —Peering model? —Supplementary services? Improvements Layer 2? 3? Fabric ISP 1 ISP 2 ISP 3 ISP 4 ISP 5 ISP 6 Route Server

POP and Other Internal Design

NANOG 21 Exterior Routing tutorial 2/17/ Typical Basic POP Implementation Gigabit Ethernet Frame Interfaces ATM Interfaces PSTN LAN Switch Management Servers Access Server Dialup Customers Router Fabric ISP Core Router 1 ISP Core Router 2 Dedicated Customers Customer Site Routers Frame DS3 2x/25 32x/30 Dedicated Customers Customer Site Router Full DS3 1 per POP25 per POP 450 users per POP /18

NANOG 21 Exterior Routing tutorial 2/17/ Transit Provider POP, Intra-POP Design Alternatives 1. POP is a route reflector cluster Core is higher-level cluster 2. Each POP is a private or public AS Full mesh iBGP or route reflectors inside POP Confederation between POPs 3. IGP within POP Controlled redistribution inside POP to BGP Prefer intra-POP of same metric Access Router Access Router POP Router POP Router

NANOG 21 Exterior Routing tutorial 2/17/ Public AS POP Confederations POP Confederations POP AS65000POP AS65111POP AS65222

NANOG 21 Exterior Routing tutorial 2/17/ Public AS POP Reflectors POP Reflectors POP AS65000POP AS65111POP AS65222

NANOG 21 Exterior Routing tutorial 2/17/ Open Access/Specialized Access Layer 1/2 Fabric Subscribers ISP 1 ISP 2 ISP 3 Internal Routed Network Tunnel Server Content Servers

NANOG 21 Exterior Routing tutorial 2/17/ Internal Routing & Switching Tunneled Addressing CLE Data Provider 1 Voice Provider 1 Enterprise VPN NAS CLE Access Gateway Data Provider 2 L2TP, Differv High VoIP Access OAM address space PPPoE or GRE ISP address space Data 1 Data 2 VPN DHCP DNS

Router Requirements Big part of the solution...but not all.

NANOG 21 Exterior Routing tutorial 2/17/ Routing Paradigms Number of Routes Forwarding Bandwidth Hello Processing Number of Interfaces Policy Analysis QoS Awareness LowHighMediumLowHighMedium LowMediumHigh MediumHighMedium End to EndEtE & PHBPHB LowHighLow L4/7 Processing MediumHighLow EnterpriseEdgeCore

NANOG 21 Exterior Routing tutorial 2/17/ Observations on Routing Table Size Global default-free table continues to grow exponentially —96509 routes as of Tony Bates' CIDR report 2/11/2001 —Let the default routing table size be D Large provider often has 1.3 to 1.5 D active routes —additional routes are more-specific customer & internal —may also have substantial numbers of inactive routes

NANOG 21 Exterior Routing tutorial 2/17/ Growth in Global Routing Table Size 184K 368K Sep01Sep01 Sep02Sep02 Sep03Sep03 Sep04Sep04 736K 85K public

NANOG 21 Exterior Routing tutorial 2/17/ Growth in Typical Tier 1 Routing Table Size (external + customer, not infrastructure) Sep01Sep01 Sep02Sep02 Sep03Sep03 Sep04Sep K public 42K internal

NANOG 21 Exterior Routing tutorial 2/17/ Observation: More than Routes Customer routes Paths per route Route validity

NANOG 21 Exterior Routing tutorial 2/17/ Convergence Global routing system Intra-AS Single Router

NANOG 21 Exterior Routing tutorial 2/17/ Single Router Convergence Initialization —Time to add new route —Time to add better route —Time to withdraw route —Time to withdraw and replace route Parameters —Matrix: number of peers versus –Routes advertised –Routes accepted Performance Modifiers —Route filtering —Route flapping —Packet vs. route filtering draft-berkowitz-bgpcon-0x.txt

NANOG 21 Exterior Routing tutorial 2/17/ Distinguish among cases Failover of link or router between customer and provider Rerouting to intranet/adjacent provider resources Rerouting to arbitrary internet destnation More multihoming in next tutorial

S-T-R-E-T-C-H

Joining the Club

NANOG 21 Exterior Routing tutorial 2/17/ More than Just Addresses, Protocol... Routing Registry Route objects AS objects Maintainer objects Routing System Configs Customer DNS NAT Hosts Specify Policy RouteTrackService ISP with Prefixes Allocate Directories SWIP Reverse DNS Address Registry Route Registry

NANOG 21 Exterior Routing tutorial 2/17/ Complexity BGP itself is fairly simple Additional attributes it carries are more complex Policy actions taken inside router (BGP sender or receiver) far more complex than the protocol itself

NANOG 21 Exterior Routing tutorial 2/17/ "BGP Transmits Policies"

NANOG 21 Exterior Routing tutorial 2/17/ Operational Relationships 1 Addresses and Delegation Address authority Address delegation Prefixes Hosts DNS Reverse DNS

NANOG 21 Exterior Routing tutorial 2/17/ Obtain routable address space Apply to registry —RIPE, APNIC, ARIN —If immediate need for /19 or /20* Obtain addresses from upstream ISP —If /19 or /20 cannot be justified Registry needs —Network design —Justification for address space

NANOG 21 Exterior Routing tutorial 2/17/ Origination vs. Advertising /20 /23 POP Dialups /23 Internal /23 Customers /23 Customers /25 32 * /30 32 * /28 /24 /25 AS /16 AS64444 an AS65000 Customer AS /19 AS /16 AS64444

NANOG 21 Exterior Routing tutorial 2/17/ Aggregating your Own Traffic /20 /23 POP Dialups /23 Internal /23 Customers /23 Customers /25 32 * /30 32 * /28 /24 /25 AS /19 Suppress more specific routes unless required by multihoming

NANOG 21 Exterior Routing tutorial 2/17/ Advertising with NO-EXPORT AS /12 Assigns /22 Assigns /22 AS62222 AS61111 AS /16 Advertises /22 NO-EXPORT Advertises /22 NO-EXPORT /16

NANOG 21 Exterior Routing tutorial 2/17/ Aggregation is better than Aggravation Blackhole routes for your blocks — Avoid more-specifics — Use NO-EXPORT when controlling load to upstream Encourage customers to aggregate — Proxy aggregation hard to administer Understand which blocks you can advertise — And do ingress/egress filtering

NANOG 21 Exterior Routing tutorial 2/17/ Preparing for Address Request (1) Address requirements of services are you offering Dynamic addressing —Dialup —Residential broadband Private addressing —Enterprises homed only to you —Dialup/broadband not offering servers Globally addressable

NANOG 21 Exterior Routing tutorial 2/17/ Prepare for Address Request (2) An ISP Topology POP1 1 internal LAN 100 Dial Ports 8 small LANs 1 med. LAN POP2 1 internal LAN 100 Dial Ports 8 small LANs 1 med. LAN POP3 1 internal LAN 100 Dial Ports 8 small LANs 1 med. LAN POP4 1 internal LAN 100 Dial Ports 8 small LANs 1 med. LAN Core Router 1 Core Router 2 Hosting Farm 1Hosting Farm 2 Infrastructure Servers Switch

NANOG 21 Exterior Routing tutorial 2/17/ Establishing an AS (1) AS Number Request In request to AS number registry —Administrative and technical contacts —Autonomous system name —Router description —Deployment schedule —Networks (by name) connected by the router(s) —Internet addresses of the routers

NANOG 21 Exterior Routing tutorial 2/17/ Establishing an AS (2) Registering in Routing Registry Minimum requirements —Maintainer object —AS object —Route object (s)

NANOG 21 Exterior Routing tutorial 2/17/ Establishing an AS (3) Operational deployment Build configuration —Policy implementation —Ingress/egress filtering Establish security procedures Start BGP connections

NANOG 21 Exterior Routing tutorial 2/17/ Routing Registry Objects Basic —AS —Route —Maintainer Additional —Inter-AS Network —Community —Router Refinements

NANOG 21 Exterior Routing tutorial 2/17/ Operational Relationships 3: Registries, Domains, etc. Address authority Address delegation Prefixes Hosts Route objects AS DNS Reverse DNS

NANOG 21 Exterior Routing tutorial 2/17/ Autonomous System Basis of exterior routing AS originate routes for some prefixes they want to be visible AS advertise routes to one another —Advertisement may not contain all addresses —Not all advertisements need be accepted

NANOG 21 Exterior Routing tutorial 2/17/ Current AS Definition RFC 1930 Connected group of IP CIDR blocks Run by one or more network operators Single routing policy —announced to the general Internet —announced with BGP-4

NANOG 21 Exterior Routing tutorial 2/17/ AS Number 16 bit number —32 bit under discussion Numbers assigned by registries —Routing policy should be stored in registry —ISPs can mirror routing registry -- place for sensitive data Private ASNs —64512 through —Private AS stripping, confederations

NANOG 21 Exterior Routing tutorial 2/17/ Operational Relationships 2: Addesses and Autonomous Systems Address authority Address delegation Prefixes Hosts AS DNS Reverse DNS

NANOG 21 Exterior Routing tutorial 2/17/ Full Employment for Consultants: Policies are inside Routers Advertising Policies —Outbound to other AS —BGP advertisement sources —Outbound route filters —Route must be in internal routing table Acceptance Policies —Inbound AS filters —Inbound route filters

NANOG 21 Exterior Routing tutorial 2/17/ Stop! What are you going to Advertise? Routes Assigned/Allocated to You Routes Assigned/Allocated to Customers Routes for which you provide Transit

NANOG 21 Exterior Routing tutorial 2/17/ Advertising Affects The way the world sees you/sends to you Binary —Routes to which you provide routing Quantitative Preferences —Multi-Exit Discriminators to your Neighbors —AS Path Manipulation to all

NANOG 21 Exterior Routing tutorial 2/17/ Routes Eligible to Advertise Are reachable by your IGPor static routes Unless they are black holes —Which conceptiually are reachable Do not advertise —Spoofed source addresses —Your internal addresses —RFC1918 space —Known rogues? –RBL?

NANOG 21 Exterior Routing tutorial 2/17/ Stop! What are you going to Accept? It depends Only those routes you will do something about Otherwise default

NANOG 21 Exterior Routing tutorial 2/17/ Do Not Accept RFC1918 source or destination Unexpected sources not assigned/allocated to peers Your internal addresses from peers

Turning it On

NANOG 21 Exterior Routing tutorial 2/17/ BGP Configuration Overview Plans and policies first! Define system of BGP speakers Specific BGP speaker configuration —Identifier —BGP process —Neighbors —NLRI to advertise —Filters and other policy mechanisms Cisco commands used as examples

NANOG 21 Exterior Routing tutorial 2/17/ Policy Implementation Flow Main BGP RIB Adj RIB Out Outgo- ing Adj RIB In Incom- ing Main RIB/ FIB IGPs Static & HW Info

NANOG 21 Exterior Routing tutorial 2/17/ AS1 R1 AS21 R1 AS1 R1 AS21 R1 AS1 R1 AS21 R1 AS1 R2 AS21 R2 All equivalent from a policy standpoint! Policy vs. Protocol Flow

NANOG 21 Exterior Routing tutorial 2/17/ BGP Configurations Know global information (AS, policies, etc.) Establish router ID Create BGP process Identify internal and external peers

NANOG 21 Exterior Routing tutorial 2/17/ Router ID and loopback interface interface loopback 0 ip address

Refining the Configuration Single and Multiple Links to a Single Provider

NANOG 21 Exterior Routing tutorial 2/17/ The BGP Tunnel Serial 0 Serial 1 Loop 0 ebgp-multihop needed when neighbor is not on same subnet

NANOG 21 Exterior Routing tutorial 2/17/ Load Balancing 1: IP Level to Single Provider Router Serial 0 Serial 1 Loop 0 Customer AS Provider AS

NANOG 21 Exterior Routing tutorial 2/17/ Load Balancing 1: Multiple Routers Customer AS Provider AS

NANOG 21 Exterior Routing tutorial 2/17/ Another Non-BGP Alternative OSPF Routing Domain Default Route ( /0) Metric Type 1 Equal Metrics Static routes D1-A0 ASBR1 D1-A0 ASBR2 ISP 1

NANOG 21 Exterior Routing tutorial 2/17/ Multiple OSPF Defaults ISP 1 POP ISP 2 POP Default Route ( /0) Metric Type 2 Higher Metric to ISP 2 (Backup) Static routes D1-A0 ASBR1 D1-A0 ASBR2

NANOG 21 Exterior Routing tutorial 2/17/ Blackhole Route Establish static route to your block(s) ip route null0 Redistribute/import into BGP Suppress more-specific prefix advertising

NANOG 21 Exterior Routing tutorial 2/17/ Effects of Blackholing No route flapping outside your AS —If your internal routes go up or down Incoming traffic for specific routes that are down —Doesn’t match any internal route —Automatically discarded without concerning anyone else

BGP Path Selection

NANOG 21 Exterior Routing tutorial 2/17/ Next Hop Access R2 R1 X Advertised route via R1 Advertised route via R2

NANOG 21 Exterior Routing tutorial 2/17/ Scope: MED vs. Local Preference vs. Weight Weight Local Preference Weight AS1 AS2 MED

NANOG 21 Exterior Routing tutorial 2/17/ Administrative Weight (Cisco extension) Advertised route via R1 Advertised route via R2 Rules in this router set R1 weight to 100, R2 weight to 500 R2 R1 X

NANOG 21 Exterior Routing tutorial 2/17/ Weight example for load sharing Primary ISP Default local preference 500 All routes ^ AS_Backup + local preference 100 Backup ISP Default local preference 200

NANOG 21 Exterior Routing tutorial 2/17/ Tiebreaker for Equal Weight: Local Preference R2 R1 Advertised route via R1, local preference 100 Advertised route via R2, local preference 500

NANOG 21 Exterior Routing tutorial 2/17/ Local Preference example for load sharing Primary ISP Default local preference 500 All routes ^ AS_Backup + local preference 100 Backup ISP Default local preference 200

NANOG 21 Exterior Routing tutorial 2/17/ Prefer locally originated routes R2 R1 Advertised route via R1 Locally defined via R2

NANOG 21 Exterior Routing tutorial 2/17/ AS Path

NANOG 21 Exterior Routing tutorial 2/17/ Shortest AS Path (Cisco extension) R2 R1 AS Route AS Route

NANOG 21 Exterior Routing tutorial 2/17/ Full Employment For Consultants: Interpreting AS Path Default assumption: local preference set based on AS_PATH Cisco considers it as part of the algorithm

NANOG 21 Exterior Routing tutorial 2/17/ AS Path Prepending Applies to routes you advertise Makes them less attractive to others Increases AS_PATH length —your AS put in the path twice

NANOG 21 Exterior Routing tutorial 2/17/ Limitations of Prepending

NANOG 21 Exterior Routing tutorial 2/17/ Route Learned from eBGP Route Learned from iBGP R2 R1 External Paths Preferred

NANOG 21 Exterior Routing tutorial 2/17/ Remote AS MED=100 MED=500 R2 R1 Lowest MED

NANOG 21 Exterior Routing tutorial 2/17/ Full Employment For Consultants: Weight, Local Preference & MED HIGHER value wins —Weight —Local preference LOWER value wins —MED —Cisco default: route with no MED preferred —IETF: route with no MED least preferred

NANOG 21 Exterior Routing tutorial 2/17/ Full Employment For Consultants: Scope of MED Default assumption: —MEDs only compared between exits to the same adjacent AS Alternate: always-compare-MED —Useful at exchange points, possibly private peerings —Cisco knob

NANOG 21 Exterior Routing tutorial 2/17/ Closest Neighbor IGP metric to R1=100 IGP metric to R1=500 R2 R1

NANOG 21 Exterior Routing tutorial 2/17/ Lowest BGP router ID R R