2003 © SWITCH SWITCHlambda - Experiences with national dark SWITCH Willi Huber / Urs Eppenberger /

2003 © SWITCH 2 The SWITCHlambda Project Project goals: Provide fore front network services to the Swiss Universities at low recurring cost Network scalable to multiples of 10 Gbps with low additional cost Long living infrastructure, no need to change provider every couple of years Independence of carrier market

2003 © SWITCH 3 SWITCHlambda Project Basic decisions: Replacement of the STM-1 links by GigabitEthernet connections based on dark fiber Long term lease of dark fiber, indefeasible right of use (IRU) for at least 10 years Use of DWDM on main trunk lines Redundant wide area links Simple network design (no SDH, no ATM): Use of Ethernet-Interface all over the place

2003 © SWITCH 4 SWITCHlambda Network Design Hierarchy of technologies –DWDM on main trunk lines (16  possible today, expandable if later needed) –CWDM or single GE to smaller sites off the backbone Transparent optical path between Zurich and Geneva –No regeneration between the two main sites of SWITCH Optical path on main trunks designed for 10 Gbps / –Chromatic dispersion compensated –Polarization mode dispersion under control Bi-directional use of fibers –lower fiber costs

2003 © SWITCH 5 SWITCHlambda Network end of 2003 dark fiber based links (approx km) leased GE link 350 km Zurich Bern Basel Geneva Lausanne

2003 © SWITCH 6 Bi-directional Use of Fiber Only one fiber used for the DWDM system Second fiber free to connect smaller sites inbetween Universities with cheap GE technology P2P GigabitEthernet- connections n * 1 GE / 10 GE implemented with DWDM Router One pair of fiber DWDM node Router

2003 © SWITCH 7 Bi-directional transmission mode: i.e. a single fiber is used for signals in both directions, leaving the second fiber free for future use. Typical SWITCHlambda node

2003 © SWITCH 8 Sorrento Gigamux

2003 © SWITCH 9 Sorrento CWDM JumpStart-400 CWDM System –8 wavelengths (4 receive, 4 transmit) –Bi-directional operation –Optional backup path SWITCH uses currently 1 pair of Jumpstarts for two p2p GE connections over on single fiber.

2003 © SWITCH 10 Bi-directional GigabitEthernet Cheap solution for p2p GE connection over one single fiber with colored GBICs and passive optical devices Splitter OADM Coloured GBIC Switch / Router Splitter OADM Coloured GBIC Switch / Router Signal reflection (e.g. bad connector)

2003 © SWITCH 11 Testing 10 Gbps/channel capability Goals verify equipment capability under real life conditions verify optical design (dispersion compensation) verify fiber quality for 10 Gbps application (PMD and other adverse characteristics) Realization DWDM interface cards on loan from Sorrento (10GE WAN-PHY) IP equipment from Force10 (2002) and Cisco (2003) traffic generation with Linux based workstations and the free software “mgen”

2003 © SWITCH Gbps tests 10 Gbps/, 360 km, bi-directional transmission 1.Test with signal regeneration optical - electrical - optical in the middle (at Bern) 2.Test with transparent optical signal Zurich - Geneva 3.Same as 2., but with additional attenuation of 3 dB of optical signal at the sender Each test: 8 Gbps of traffic during 35 hours, 0 bit errors  BER < 8 *  SWITCHlambda ready for 10 Gbps !

2003 © SWITCH 13 Pros and Cons: Technical View Stable, long living infrastructure High future potential This new stuff is fun none

2003 © SWITCH 14 Is this reliable? Stable infrastructure with occasionally long interruptions –Interruptions mainly due to planned maintenance –True fiber cuts are rare, but last for hours if not days (e.g. fiber on high voltage power lines, fiber along railway) –Experience of a big Swiss carrier: Fiber cuts happen mostly in the local loop (city) Our experiences so far –Several interruptions due to planned maintenance (lasting up to 1 day) –Broken optical amplifier (interruption of 4 hours) »Redundant links are mandatory for a good sleep!

2003 © SWITCH 15 No connection, what now? 24/7 monitoring by Ascom Netwatch –ping, SNMP –alarm SWITCH staff –basic analysis SWITCH NOC for analysis and action decision University IT staff –power problems –physical equipment check Onsite support by Ascom Netwatch –localize cable problems with OTDR (optical time domain reflectometer) –Cisco hardware Onsite support by Deltanet for Sorrento DWDM

2003 © SWITCH 16 Pros and Cons: Operational View we are the bossBig effort for negotiations with a lot of contract partners no single provider to blame in case of connection loss need a car to fix problems instead of telnet

2003 © SWITCH 17 Network Cost, general Remarks Cost for wide area links –Well known in advance Local loop cost –Vary a lot from place to place –Good (say cheap) solutions need a lot of planning and patience –Building own infrastructure is economically better than renting, but not possible in all cities –In some cities there is a de facto monopole of the city’s industrial works –The railways are an excellent partner for wide area links: low local loop cost because the schools are mostly located near railway stations

2003 © SWITCH 18 Network Cost, general Remarks (2) Cost for optical equipment –DWDM is expensive –Long 10 Gbps/ DWDM trunks are very expensive (dispersion compensation, amplification) –CWDM stuff is getting cheap Routers / Switches –Ethernet interfaces are cheaper by factors than SONET ones

2003 © SWITCH 19 Cost Structure Total yearly cost:2.05 MEUR Cost include:Investment years Gbps between the major sites maintenance & support wide area fiber local loop fiber DWDM equipment CWDM equipment maintenance & support Depreciation period used for the calculations: Fiber optic infrastructure (wide area & local loop):10 years Fiber optic equipment DWDM & CWDM:5 years 31 % 12 % 6 % 44 % 7 %

2003 © SWITCH 20 Comparison with predecessor SWITCHngSWITCHlambda Years covered: Technology:ATM/STM-1dark fiber, GE & 10 GE Sites connected: 13 universities21 universities Yearly cost:2.95 MEUR2.05 MEUR

2003 © SWITCH 21 Isn’t it cheapter to buy it? We have not done a serious market review lately or even negotiated for capacity to get the real prices But we have a figure we assume is the best current offer on the market for high capacity links: 100 kEUR –limited to 300 Mbit/s –local loop must exist For 21 sites: 2.1 MEUR  Similar cost as SWITCHlambda, but less flexibility and performance.

2003 © SWITCH 22 Pros and Cons: Economic View Low recurring cost Low marginal cost for additional bandwidth High investment Uncertainity about development of bandwidth prices