Operating Wide-Area Ethernet Networks Matt Davy Global NOC

Outline Overview of Networks Configuration Troubleshooting/Monitoring

Overview of Networks NLR FrameNet nationwide ethernet over dwdm 18 Cisco 6509 switches 10GbE backbone p2p and multipoint vlans dedicated and best effort

Overview of Networks I-Light Indiana’s statewide higher ed network statewide ethernet over dwdm 19 Cisco 6509 switches (layer2 & layer3) 10GbE backbone with p2p vlans

Overview of Networks MANLAN ethernet exchange in new york city Cisco 6513 switch 1GbE and 10GbE connections over dwdm, sonet, direct fiber - even one over mpls l2 vpn local and wide-area connections

Overview of Networks Indiana University campus network large layer-2 infrastructure from edge into core (capable of plumbing vlans between buildings and even between campuses) Cisco 6500’s and HP Procurve very interesting stp design ~1,500 total switches

Configuration Issues configuration of vlans very manual and time intensive (manual = error prone) need to automate this process various control plane projects are one option, but could use something more lightweight could use vtp ?

Configuration Issues VLAN ID Assignment big problem when interconnecting multiple layer2 domains does Q-in-Q solve this ? does vlan id translation solve this ?

Configuration Issues Q-in-Q sounds good, but not flexible enough want to map some .1q tags to outside vlan and want other .1q tags to get switched normally customer A wants to trunk vlans to customer B, but also wants vlans to customers C, D and E who don’t want Q-in- Q. also not implemented in all switches

Configuration Issues VLAN ID Translation could help, but limitations in currently implementation each port needs it’s own translation table on 6500, translation table is shared across multiple ports greatly confuses cross-domain troubleshooting

Configuration Issues loops and spanning tree fun spanning-tree is often not well understood some people opt to leave it disabled or leave the default config - since they don’t plan to build loops in their topology often does not help anyway when multiple layer2 domains are interconnected

Configuration Issues things that might help some: enable spanning-tree within your domain filter bpdus at the edge of your domain limit total broadcast traffic on every port make sure config has enough granularity for port speed (1% of 10G is still too much)

Loops Outside of Your Domain #1 #2

Configuration Issues why will a loop outside your domain hose your switch ? not 100% clear one possibility is mac address learning overload switch flooded with packets for which it has to learn source mac addresses mac addresses quickly flip-flop between ports

Configuration Issues how could this be avoided ? turn off mac address learning for p2p vlans, could leave mac learning off and just flood all packets - they only have 1 direction to go anyway could also have out-of-band mechanism to statically configure mac forwarding tables will this entirely protect you ? don’t know

Troubleshooting/Monitoring how can you tell when a vlan is down ? hint: think break in the middle of the topology on vlan trunks, can’t see how much traffic is associated with each vlan CoS hack on the 6500’s for this lack of netflow data - can get sflow on some platforms, but analysis tools for sflow lacking

Troubleshooting/Monitoring tools to trace current vlan path across the network IU has developed a spanning-tree mapping tool that helps with this “turn-around interfaces” useful for debugging performance problems

Thank You