1. Use of BGP and MPLS VPNs: A Case Study
Fred P. Baker
CCIE#3555

2. Contents Current Network The MPLS VPN project Routing Objectives
What we did
How we tested

3. Current Network

4. Current Environment Hub and spoke to 4 data centers
Sites do not in general connect to 2 data centers due to cost and OSPF issues
Generally place servers by geography
You servers are in the data center your links are in
Mostly Frame Relay to ATM interworking with some private lines
70 of some 350 remote sites have 2 links
ATM PVC dual mesh between the data centers
12000 agent location network done by MCI with combination of DSL and Fractional T1

5. Address Space /8
Mostly inside
Some BP
/16
Used all over
/12
Extranet
/16
Public address space
Used mostly by extranet
Some legacy inside

6. Core ATM PVCs 2 10meg between each pair of data centers
2 routers on the core
So 2 meshes

7. Allstate Core

8. 10.0.0.0 address allocation /11 for core 1 per data center

9. Allstate Data Center

10. Routing Protocol Single OSPF AS Cisco and OS/390 based routers only
Firewalls now static routed
Peer authentication soon

11. Remote sites AT&T frame relay at the site ATM into the data center
Some ISDN backup
A remote site is connected to a single data center (for now)
Servers and applications tend to have geographic affinity

12. Remote Site

13. Remote Site Switch Layer

14. Agent Broadband 10,000 locations Connected via IPSEC VPN
WorldCom managed routers
NO split tunneling
IPSec Transport with GRE tunnel to Dallas and Hudson
Agent PCs are 10.*.*.*
Agent access is via Allstate Internet Proxy

15. Overview

16. Agent Broadband in Data Center

17. Agent office

18. Internet/Extranet We do not use the default route
There are 3 data center with ISP connections
We code static routes to the firewalls (we don’t trust firewalls running dynamic routing protocols) and redist to OSPF

19. The project

20. The project We use a single data network provider
This is a single point of failure of that providers ATM/Frame networks
Add a second data provider
Initially to use for the dual attached sites
Then convert 1 of the core ATM meshes to the second provider

21. Layer 2 vs Layer 3 provider
Frame Relay is layer 2 connectivity
The routers have a direct peering relationship
Many providers are offering Layer 3
Costs are the same or even less
MPLS VPN is the data transport
Many providers are using MPLS to move even layer 2 networks
You have a routing relationships with the provider not with yourself
So More complex to configure and fix
Not a simple OSPF network anymore

22. Which one we picked Layer 3…
DR becomes free do not need to run more PVCs to a DR data center
The data center placement of servers assumption is changing
Apps are being put to 1 DC
Also there is more site to site traffic than we expect
So we can reduce traffic on the ATM core
And increase response time
Do dual homed sites first convert 1 link to L3
Single homed late

23. MPLS VPN VPN A/Site 1 VPN A/Site 2 VPN A/Site 3 VPN B/Site 2
CEA1
CEB3
CEA3
CEB2
CEA2
CE1B1
CE2B1
PE1
PE2
PE3 P1 P2 P3
10.1/16
10.2/16
10.3/16
10.4/16

24. Route types CE customer Edge PE provider Edge P providers your router
run BGP to provider
Knows nothing about other customers or provider routes
PE provider Edge
Knows about all local customer VPNS
Has multiple routing tables
P providers
Transport only
No customer routes

25. Routing objectives Support load share from the home DC
Remote site goes direct to non home DC over L3
Remote site directly to remote site
Reduce transit of the core
Support a L3 provider in the core replacing 1 ATM mesh
Do not use remote sites to transit traffic

26. Technical Objectives Limit the number of bgp attributes used
Keep the remote site configuration simple
Do not inject the default route unless you must
How to inject the Internet routes

27. Routing protocol design

28. Don’t forget the 3 rules of routing
Longest subnet mask
Lowest distance
Best metric

29. BGP features we used As path Path length filters No export Backdoor
If AS Paths are equal then router uses eBGP route

30. How to route Must look at the routes going BOTH ways
Routes to
Routes from
The routes you advertise drags traffic to you
The routes you take in is how you route back
We load share by having each router use a different path, then send equal cost into IGP

31. Result Use MPLS VPN based L3 provider Remote sites 2nd link to L3
Each data center connects to L3
Will not use L3 to route between DCs due to QoS concerns

32. Routing Use BGP at remote sites EACH data center is 1 AS
Can use OSPF with SOME providers but not all
BGP works much better
Each site is 1 AS
EACH data center is 1 AS
This allows us to put an L3 provider in later
BGP routes BETWEEN ASes
Address ASes from private space
This is ok because provider is a VPN

33. Route injection to/from BGP
Allstate Data Center
Explicit network statements to BGP
Redist BGP to OSPF
Remote site routes
Redist from OSPF
Decided that using network statements to complex
BGP routers send just default route to any switches
We will accept the extra LAN transit
Internet routes
Redist static

34. Internet routes
There will be non BGP L3 switches between Inet and allstate core
Redist static into OSPF already
So just redist into BGP also
Put internet router in same AS as datacenter (have to as no direct path)
Use sync
Send to L3 provider and to sites over L3

35. BGP to L3 provider (and then remote sites
Data center side
Send data center /11s
Send internet routes
Take routes from L3 provider
Do not forward other eBGP learned routes
Remote site side
Send all local routes
do not forward other learned eBGP routes
Remember the no export to kill transit
Receive all routes
Want to take L3 when I can

36. DC to Remote site FR Send all bgp derived routes
Do as prepend of the data center AS
This makes AS path =2 for DC on FR and L3 paths
This makes AS Path=3 for DC to DC via ATM core so site to remote DC traffic over L3

37. Remote site to DC on FR Do as prepend of 1 AS at remote end
Need this so FR and L3 paths have AS Path=2 so we load share
Filter routes with AS Path >1
I only want to send the local site routes up the FR link
Do not want DC to send transit traffic to site

38. IBGP in the remote site Set next hop self
Routers must have a shared Enet
No redist of BGP to OSPF
So cant use sync so cant transit a L3 switch
Do not forward routes I learn via FR
Do not want a transit from L3 up the FR link
Do not want a transit to L3 from FR link
Set no export attribute on routes from DC over the FR link
This prevents site from passing them to L3
Cannot AS path filter on IBGP because I want to pass the DC route via iBGP
Why I use no export

39. Results

40. DC to DC Each site learns over ATM network with AS Path = 1
Cannot route over L3 provider

41. Remote site to non home dc
Non home DC sent via L3 AS Path = 2
Home data sends via FR AS Path = 3 due to prepend
Use if L3 down

42. non home dc to remote site
Non Home DC learns remote site routes from L3
Home data center sends only the /11 summary
so longest match says L3

43. home dc to remote site Load share Routes from L3 have AS Path = 2
Routes from FR have AS Path = 2 due to prepend
So each router uses eBGP route

44. remote site to home dc Don’t care as much about load share
Routes from L3 have AS Path = 2
Routes from FR have AS Path = 2 due to prepend
So each router uses eBGP route

45. remote site to remote site
Use L3 network
Learn site specific routes directly from site
Learn /11 summaries from DCs

46. Agent routes Only dual DC connected things that don’t use BGP
Many routes summarized as /19s
I get these from MCI as OSPF externals
Have not decided how to inject them
They go to two data centers for redundancy
So I need to send them via BGP
So a router will get an OSPF external from the local MCI connection and the other data center via BGP
eBGP < OSPF so BOOM
Use backdoor on core routers to set distance on the agent routes to > than OSPF
So if local MCI connection up use it, else transit core

47. Testing

48. Local Testing Use 7 routers 1 remote site OSPF route not shown Paths
iBGP at remote L3
FR to home DC
Inter DC

49. CPOC Cisco Proof Of Concept In Raleigh and San Jose
Lab use is free (if you are big enough)
Send in specific test plan
Your SE goes in a week ahead of time
Lab is all setup when you arrive

50. Testing Test migrations Test routing Measure convergence
based on our policies
failovers
Measure convergence
Test a migration of a core ATM mesh to L3
Get some data and experience on the MPLS side
Try multicast over MPLS/VPN

51. CPOC Network Diagram

52. CPOC Learnings
Inject all links both ATM core and L3 into BGP as they will source pings
Turn sync off due to code defect
You must explicitly code send community in iBGP
If you reference a non-existent as-path statement NO ROUTES
OSPF LSAs stay in the data base up to 90 minutes due to timer jitter
This is a migration issue
Do lots of clear routes/clear ip bgp in the migration
Need to change the BGP timers as default convergence is 3 minutes
iBGP only sends the best route

53. Going forward Already run BGP to some remote sites
Migrate the core to bgp first
Do a dress rehearsal
Will be a big scary change so plan well
Examine tools
May not be able to assume we will get traps
May have to watch the BGP tables for changes
Get a test connection in place