1. Operating Wide-Area Ethernet Networks
Matt Davy
Global NOC

2. Outline
Overview of Networks
Configuration
Troubleshooting/Monitoring

3. Overview of Networks NLR FrameNet nationwide ethernet over dwdm
18 Cisco 6509 switches
10GbE backbone
p2p and multipoint vlans
dedicated and best effort

5. Overview of Networks I-Light Indiana’s statewide higher ed network
statewide ethernet over dwdm
19 Cisco 6509 switches (layer2 & layer3)
10GbE backbone with p2p vlans

7. Overview of Networks MANLAN ethernet exchange in new york city
Cisco 6513 switch
1GbE and 10GbE connections over dwdm, sonet, direct fiber - even one over mpls l2 vpn
local and wide-area connections

8. Overview of Networks Indiana University campus network
large layer-2 infrastructure from edge into core (capable of plumbing vlans between buildings and even between campuses)
Cisco 6500’s and HP Procurve
very interesting stp design
~1,500 total switches

9. Configuration Issues configuration of vlans
very manual and time intensive (manual = error prone)
need to automate this process
various control plane projects are one option, but could use something more lightweight
could use vtp ?

10. Configuration Issues VLAN ID Assignment
big problem when interconnecting multiple layer2 domains
does Q-in-Q solve this ?
does vlan id translation solve this ?

11. Configuration Issues Q-in-Q sounds good, but not flexible enough
want to map some .1q tags to outside vlan and want other .1q tags to get switched normally
customer A wants to trunk vlans to customer B, but also wants vlans to customers C, D and E who don’t want Q-in- Q.
also not implemented in all switches

12. Configuration Issues VLAN ID Translation
could help, but limitations in currently implementation
each port needs it’s own translation table
on 6500, translation table is shared across multiple ports
greatly confuses cross-domain troubleshooting

13. Configuration Issues loops and spanning tree fun
spanning-tree is often not well understood
some people opt to leave it disabled or leave the default config - since they don’t plan to build loops in their topology
often does not help anyway when multiple layer2 domains are interconnected

14. Configuration Issues things that might help some:
enable spanning-tree within your domain
filter bpdus at the edge of your domain
limit total broadcast traffic on every port
make sure config has enough granularity for port speed (1% of 10G is still too much)

15. Loops Outside of Your Domain#1 #2

16. Configuration Issues
why will a loop outside your domain hose your switch ?
not 100% clear
one possibility is mac address learning overload
switch flooded with packets for which it has to learn source mac addresses
mac addresses quickly flip-flop between ports

17. Configuration Issues how could this be avoided ?
turn off mac address learning
for p2p vlans, could leave mac learning off and just flood all packets - they only have 1 direction to go anyway
could also have out-of-band mechanism to statically configure mac forwarding tables
will this entirely protect you ? don’t know

18. Troubleshooting/Monitoring
how can you tell when a vlan is down ?
hint: think break in the middle of the topology
on vlan trunks, can’t see how much traffic is associated with each vlan
CoS hack on the 6500’s for this
lack of netflow data - can get sflow on some platforms, but analysis tools for sflow lacking

19. Troubleshooting/Monitoring
tools to trace current vlan path across the network
IU has developed a spanning-tree mapping tool that helps with this
“turn-around interfaces” useful for debugging performance problems

20. Thank You