December 2007 Neil Francis Network Administrator SWERN University of Bath JANET Optical Event SWERN rebuilt using Ethernet 50 VLANS and 3 Telcos

2 SWERN2 - Outline SWERN One of the RNOs Main University connections Bath, Bath Spa, Bristol, UWE, Exeter, Plymouth, Falmouth Approx. 40 Further Education Institutions A dozen or so other institutions About 450,000 users

3 Swern2 - Some Background SWERN is in the South West of England Merger of SWAN and BWEMAN Essentially 2 networks connected together Piecemeal build – varied active equipment Some resilience (always a worry) Mixture of ATM, POS, Ethernet etc Limited HR Two part time engineers, network managers, some Ops support etc Geographical issues – long and thin, rural(ish) 13 POPs – some remote The usual stuff.....

5 SWERN2 - Opportunities For SWERN 2 we had an opportunity to tidy things up The usual design factors Homogeneous, Resilient, Cost Effective?, Engineering, HR. The South West is long and thin We had POPs in the large conurbations Cheltenham Swindon Bath Bristol Taunton Exeter Plymouth

8 SWERN2 - Options Options Dark fibre, managed solutions etc All considered but not only cost ruled these out. Long distance ethernet circuits becoming more comfortable for us Concerns were things like Reliability Resilience Scalability (bandwidth) QOS Security Understanding exactly what we were getting

9 SWERN2 - Ethernet Ethernet – but not quite Types of connection: Point-point, point-multipoint, multipoint-multipoint, etc. Our equipment: Routers Switches Architecture: VPWS - Virtual Private Wires VPLS - Virtual Private LANs Ethernet over Optical etc... Service Characteristics: VLAN Multiplexing VLAN Transparency L2 PDU Transparency

10 SWERN2 - Ethernet Telcos had solutions available We already had some metro ethernet ccts Tail circuits only Ethernet on the backbone meant... Bandwidth - fast, cheap, tunable Standard simple equipment Standard circuit presentations (fibre and copper) Layer 2 links – easier to manage Just use point to point vlans We do layer 3 It looked OK

11 SWERN2 – Ethernet Backbone Connecting these all together Connect to enable loops Two POPs in each city Connect these together Fairly close together so relatively easy Connect each POP back to an RNEP Use Metro Ethernet circuits Connect the RNEPs together Fairly easy End up with something like this....

12 SWERN2 – Ethernet Circuits Essentially 6 loops here Connect to standard ethernet equipment...

13 SWERN 2 – Standard Equipment Cisco 7609 OSR Routers Sup. Engine 720/1Gbyte RAM Located at RNEPs Cisco 7206VXR Routers NPE-G1 Processors Cisco Catalyst Port 10/100/1000 Ethernet Switch

14 SWERN 2 – Loops & VLANS SJ5 RNEPsPOPs End Sites Allow only specific VLANS across the trunks Trunks

15 SWERN2 - VLANS

16 SWERN2 - Tender Tender issued late 2005 Solutions based on mixture of WDM SDH/SONET/EPL LES/WES MPLS But all presented to us as ethernet Carrier Ethernet Metro Ethernet Evolved Ethernet National Ethernet etc......

17 SWERN2 - Bandwidths All bandwidths uncontended

18 SWERN2 - Engineering With 6 loops 8 VLANS per loop Plus some test VLANS Over 50 VLANS – all terminate on our RNEP routers All traffic flows through these boxes All point to point VLANS Plus a management VLAN We do the layer 3 stuff (OSPF, BGP) We still have to manage, monitor, administer, troubleshoot etc.

19 SWERN 2 - Routing

20 SWERN 2 – Routing and Resilience OSPF Single small backbone (Area 0) Stub areas from the POP areas Timers tuned low Also looked at BFD for fast link failure detection Best case scenario – as good as POS! Worked between our POP routers and RNEPs Needed 12.4.x IOS on our 7200s Experimented with timers but were finding even a flash disk insertion was causing BFD to baulk We currently rely on fast converging OSPF

21 SWERN2 - Observations We know we have a mixture of the following: Simple LES/WES circuits DWDM/SDH/EPL circuits EoMPLS circuits We run point to point VLANS across all this technology and it works Seemed too easy in many ways Just 2-3 people doing all this

22 SWERN2 - Observations Management VLAN Spanning tree should be turned on Have had more than one occurrence of looping Somewhere - BPDUs are not being passed Taking longer to fix than it should Witnessed CDP neighbor inconsistencies Tracked to provider switches being at different OS levels. A bug here but related to above. Lost our.ad/.ah port configuration 3 VLANS down Provider insistent all was working Fixed at 3 rd level support – too long

23 SWERN2 – EPL Latency

24 SWERN2 – MPLS Latency

25 SWERN2 – Next stages 10 Gbps links across Bristol Connecting RNEPS Dark Fibre and 10GE over Optical Still presented as Ethernet Terminate on 10GE switches Probably do some routing on same devices Direct connection to RNEP routers 10G POS to Janet Core Everything else the same

26 SWERN2 - Possibilities The concept of SWERN2 is all very easy Identical POPs and equipment Identical circuit connections Ideally we would want less POPs Possibility to use Telco infrastructure to backhaul everything to our RNEPS Tail circuits to institutions not an issue any more Resilience QoS

28 SWERN3 - Maybe Can't keep up with the service offerings Ethernet over SONET MPLS/VPLS T-MPLS (ITU G.8110) 802.1ad, 802.1ah, 812.1Qay 802.3ah – Ethernet in the first mile Etc... Management 802.1ag – Connectivity Fault Management 802.1ag – End to End OAM/CFM Management - is this the important bit?

29 SWERN2 - Observations

30 Any Comments What we had What we did How we did it Would anyone else do it this way? What do other people do? Observations