Josef Vojtěch, Stanislav Šíma, Jan Radil, Lada Altmannová Dark Fibre Facilities for Research and Experimentation

7 Apr th TRIDENTCOM, Washington DC 2 Authors participate on: CESNET research program ( GÉANT2/3 project ( ), Phosphorus project ( Presented content do not necessarily reflect an official opinion of any institution or project. Dark Fibre Facilities for Research and Experimentation

7 Apr th TRIDENTCOM, Washington DC 3 Dark Fibre Facilities for Research and Experimentation Outline Introduction Experimentation on Physical Layer Concepts Used Dark Fiber Nothing in Line Open Photonic Devices CESNET2 Experimental Facility + Applications Connecting Dispersed Users to EFs Conclusions

7 Apr th TRIDENTCOM, Washington DC 4 Dark Fibre Facilities for Research and Experimentation The Czech Republic Area: km 2 = sq mi (South Carolina) Population: (Michigan) source:

7 Apr th TRIDENTCOM, Washington DC 5 Dark Fibre Facilities for Research and Experimentation CESNET CESNET - Czech Educational and Scientific Network Established as Association of Legal Entities (z.s.p.o.) in 1996 – not for profit All public universities + Czech Academy of Sciences 52 staff members in Prague + over 150 part-time staff working on projects Funded by Ministry of Education, Youth and Sports of the Czech Republic and association members under research plan: „Optical High Speed National Research Network and its New Applications“ ( )

7 Apr th TRIDENTCOM, Washington DC 6 Understanding of physical layer is crucial: There are no applications without physical layer. Technology used in physical layer limits overlying applications by transmission speed, latency, non- determinism etc. There is electronic processing speed limitation (about 100 Gb/s per port), optical processing speed limitation will be about 1 Tb/s per port. Optical processing has significantly lower energy consumption. See for example presentation Tetsuya Miyazaki: “Node and Link Technologies for New Generation Networks”, NICT, fireworks.eu/fileadmin/events/9- 10_June_2008_Brussels/Presentations/ /5B- Photonic_Networks/15-TETSUYA_MIYAZAKI_- _Photonic_Networks.pdf fireworks.eu/fileadmin/events/9- 10_June_2008_Brussels/Presentations/ /5B- Photonic_Networks/15-TETSUYA_MIYAZAKI_- _Photonic_Networks.pdf Dark Fibre Facilities for Research and Experimentation Experimentation on Physical Layer

7 Apr th TRIDENTCOM, Washington DC 7 Should be based on technology advances. Many possibilities to enhance applications can be find in physical layer. In many cases we can gauge before network users, what new physical layer possibilities will bring improvements in future. In Research and Development driven by experiments, it is important to verify and evaluate real possibilities, create new offer of services for users, and find early adopters. Applications can be enhancement by connecting dispersed users to experimental facilities. Dark Fibre Facilities for Research and Experimentation Experimentation based on physical layer

7 Apr th TRIDENTCOM, Washington DC 8 Concepts used for building: Operational National Research and Educational Network (NREN) - CESNET2 Experimental Facility (EF) – CzechLight, part of Global Lambda Integrated Facility (GLIF), see Dark Fiber (DF) + Customer Empowered Fiber (CEF) Networks Nothing in Line Approach (NIL) Cross Border Fibers (CBF) Family of open photonic devices – CzechLight Family Dark Fibre Facilities for Research and Experimentation Concepts used

7 Apr th TRIDENTCOM, Washington DC 9 Migration from telco lambda services to dark fibers brings: Much more capacity Freedom in network design DFs really needed for E2E services  First mile issues with fibers in many places  Wireless can be used too but limitations may be rather significant (no WDM, 10G predicted, but higher speeds?, reliability and availability...) First DF in CESNET in 2000, 2.5 Gb/s PoS Approximately 300km (186mi), 3 regenerators More DFs have been acquired since then First Erbium Doped Fiber Amplifiers (EDFAs) operated DF in CESNET in 2002, 1 GE. Approximately 190km (118mi), NIL concept DF line in service till upgrade to 10 Gbps O-DWDM system in 2008 Ethernet-based E2E services, shared capacity, VLANs Transition of backbone to DFs was finished in 2004 Dark Fibre Facilities for Research and Experimentation Dark Fibres, Customer Empowered Fibres

7 Apr th TRIDENTCOM, Washington DC 10 Nothing in Line (NIL), ‘hut skipping’ Original approach, hut each 80km (50mi) - not necessary always and everywhere Distances can be extended, even with commercially available equipment Reduction in CAPEX+OPEX (less HW, housing, power) Examples – NIL operational lines  235km (146mi) EDFA only originally 1GE, upgraded to 10Gbps O-DWDM  308km (191mi) EDFA + Raman Dark Fibre Facilities for Research and Experimentation Nothing in Line

7 Apr th TRIDENTCOM, Washington DC 11 Many projects and traffic with your neighbor? Acquire/rent a DF - latency will be low. In CESNET: CBF: SANET Slovakia 1GE in 2003, upgrade to Open photonic DWDM (O-DWDM) in 2006 CBF: PIONIER Poland 1GE in 2004, upgrade to DWDM in 2007 CBF: ACONET Austria O-DWDM, in 2006 O-DWDM Dark Fibre Facilities for Research and Experimentation Cross Border Fibres

7 Apr th TRIDENTCOM, Washington DC 12 CESNET is developing family of open photonic devices (CL family), enabling early adoption of leading edge photonic technology Devices support network research and experimentation on dark fibre level, fast testing and prototyping, support experimental applications requested by users or field experiments on dark fibre lines, verifying feasibility of network improvements. Proved very useful for CESNET2 development, Experimental Facility development, CBF lighting, GLIF applications development (especially for dispersed end users), improvement of interoperability on the physical layer, as well as remote monitoring and control, low latency and deterministic multicast (will be addressed further). Main advantages: offer freedom of design, ability to meet user needs, easy to modify if change necessary, avoiding delays in innovation (low needs to save investments), photonic transmission and processing speeds, low cost, saving energy, and space. Dark Fibre Facilities for Research and Experimentation Open photonic devices for network research and experimentation

7 Apr th TRIDENTCOM, Washington DC 13 First Open Photonic devices in CESNET2 in 2004 Family of open photonic devices (DWDM and other photonic tasks) EDFAs (terminal, inline applications, CATV) Raman amplifiers including TDM (amplification in long NIL) Tunable CD compensators (long NIL, 40G) Reconfigurable Optical Add/Drop Multiplexers (ROADMs) Variable Mux/Demuxes (VMUX) Wavelength selective switches (WSS) * Optical channel monitors (OCM) All optical wavelength converters * Photonic path switches (both mechanically and non mech. based) Photonic path switches with multicast option (mech. + non mech.) Manufactured by FTTx companies/vendors. Dark Fibre Facilities for Research and Experimentation Family of Open Photonics Devices * in experimental regime

7 Apr th TRIDENTCOM, Washington DC 14 Evolution of the CESNET network 1999 – 2007 (not with all details). Dark Fibre Facilities for Research and Experimentation Progress in CESNET networks : first optical amplifiers (EDFA)2004: first CLA

7 Apr th TRIDENTCOM, Washington DC 15 Dark Fibre Facilities for Research and Experimentation Up-to-date situation in CESNET2 CESNET2 network km (2920mi) of DFs Incl. 980km (609mi) single fiber DF links Mixed architecture – coexistence of commercial and open DWDM systems at speeds up to 10Gbps

7 Apr th TRIDENTCOM, Washington DC lines Cisco MSTP n x 10 Gb/s DWDM transmission system (with ROADM). 10 lines deployed with open photonic n x 10 Gb/s DWDM transmission systems, including 2 international CBF connections to Bratislava and Vienna: No bottleneck on international links anymore. 6 single fibre lines deployed with open photonic DWDM Other 4 links will be lit in 2009 (including 3 single fibre lines). Dark Fibre Facilities for Research and Experimentation Up-to-date situation in CESNET: WAN

7 Apr th TRIDENTCOM, Washington DC 17 Dark Fibre Facilities for Research and Experimentation Experimental facility: WAN

7 Apr th TRIDENTCOM, Washington DC 18 PIONIER Cieszyn Poland Cisco6506 Masaryk University BRNO E300 GOLE CzechLight Praha ONS15454 DWDM 40 x 10 Gb/s OC-192c N x 1GE over SONET GOLE NetherLight Amsterdam OC-192c Ethernet VLANs GOLE StarLight Chicago ONS15454 N x 1 GE CESNET2 DWDM backbone N x 1&10 Gb/s SANET Bratislava Slovakia CBF 4 x 10 Gb/s CBF 8 x 10 Gb/s BigIron CBF 8 x 10 Gb/s ACOnet Wien Austria Cisco7609 Dark Fibre Facilities for Research and Experimentation Experimental facility - connections

7 Apr th TRIDENTCOM, Washington DC 19 Dark Fibre Facilities for Research and Experimentation GLIF

7 Apr th TRIDENTCOM, Washington DC 20 Preferred approach is based on dark fibres connecting end users and EF via photonic (all-optical) lightpaths (i.e. implemented without OEO conversions). Tunneling through CWDM was successfully demonstrated (but not yet fully published) for example in GLIF2007 workshop in Prague as connection between Charles University and EF, used for all demos. Photonic lightpath connected to EF are available in CESNET2 NREN and in CESNET Experimental Facility, including CBFs. Dark Fibre Facilities for Research and Experimentation Connecting dispersed users to EFs

7 Apr th TRIDENTCOM, Washington DC 21 Dark Fibre Facilities for Research and Experimentation Experimental facility

7 Apr th TRIDENTCOM, Washington DC 22 BP CLA PB01 PASNET Dark Fibre, G.652, 13 dB (DWDM over CWDM) Demos PCs with XFP Switches Carolinum λ1λ2λ3λ4λ1λ2λ3λ4 λ1λ2λ3λ4λ1λ2λ3λ4 GOLE StarLight Chicago GOLE NetherLight Amsterdam E300 GOLE CzechLight Praha λ1λ2λ3λ4λ1λ2λ3λ4 λ1λ2λ3λ4λ1λ2λ3λ4 DWDM MUX DWDM DEMUX CWDM MUX CWDM DEMUX CWDM MUX DWDM DEMUX DWDM MUX CESNET2 OC-192c N x 10GE Combination of 10G international lambdas, CWDM, DWDM Open and ‚big‘ vendor transmission systems Dark Fibre Facilities for Research and Experimentation GLIF 2007 demos solution

7 Apr th TRIDENTCOM, Washington DC 23 Multicasting device demonstration CLM (Multicast Switch) has photonic part multicasting (replicating) optical signal bands electronic part for remote switching control via web interface Multicast by CLM is transmission speed agnostic, deterministic without jitter and delay, i.e. without OEO conversion, without store and forward processing and without loading of switch by multiple tasks Operation demonstrated at GLIF 2007 at Prague Operation demonstrated outside CESNET on University of Washington (8th Annual GLIF Workshop on 1 October 2008 at Seattle), CLM was located at StarLight and managed remotely Dark Fibre Facilities for Research and Experimentation

7 Apr th TRIDENTCOM, Washington DC 24 Dark Fibre Facilities for Research and Experimentation Photonic multicast with CLM

7 Apr th TRIDENTCOM, Washington DC 25 Dark Fibre Facilities for Research and Experimentation UCSD Calit2 PNWGP CW 6506 GLIF Mog HD RX 7609 CW 6506 HD RXHD TX CW e1200 HD RX StarLight CzechLight Brno C6506 Praha E300 CAVEwave (2155) C-Wave (2155) C-Wave (2155) (440)(441) CESNET (trunk 440,441) (2155) EVL e600 CESNET CLM (441) I1 O1 O2 O3 O4 (445) (2155) IP: MAC: 10:10:10:10:10:10 IP: MAC: 10:10:10:10:10:10 IP: MAC: 10:10:10:10:10:10 ‚TX only‘ RX only Tx Rx 6/2 6/66/106/14 GLIF 2008 CLM Demonstration in Cinegrid demos Combination of 10G international lambdas, DWDM Open and ‚big‘ vendor transmission systems

7 Apr th TRIDENTCOM, Washington DC 26 GLIF demo on 8th Annual Global LambdaGrid Workshop 2008, Seattle, USA. HD video with real optical multicast. GLIF demos on 7th Annual Global LambdaGrid Workshop 2007, Praha, Czech Republic. HD, 4k video transmissions. Multiple 10 Gb/s links to a medieval building with open DWDM transmission system. HEP - data access and processing for ATLAS and ALICE on LHC, D0 on TEVATRON, STAR on RHIC. First VINI sites in Europe (Praha, Plzeň). Intercontinental Remote Education on High Performance Computing between Masaryk University Brno and Louisiana State University. Dark Fibre Facilities for Research and Experimentation Applications over EF

7 Apr th TRIDENTCOM, Washington DC 27 Dark Fibre Facilities for Research and Experimentation GLIF Lightpaths (E2E) to HEP projects participants in the Czech Republic Nuclear Physics Institute (AS) Bulovka, Praha Institute of Experimental and Applied Physics (CTU) Horsk á, Praha Faculty of Mathematics and Physics (CU), Tr ó ja, Praha Nuclear Physics Institute (AS) Řež Faculty of Nuclear Sciences and Physical Engineering (CTU) Břehová, Praha TAIPEI BNLFNAL Access 1GE Trunk 10GE E300 GOLE CzechLight Praha ONS15454 Cisco6506 CWDM Institute of Physics (IoP AS), Na Slovance, Praha 10GE GOLE NetherLight Amsterdam OC-192c n x 1GE over SONET DF CWDM AS – The Academy of Sciences of the Czech Republic CTU – Czech Technical University, Praha CU – Charles University, Praha Combination of 10G international lambdas, CWDM Open and ‚big‘ vendor transmission systems

7 Apr th TRIDENTCOM, Washington DC 28 Dark Fibre Facilities for Research and Experimentation GOLE CzechLight Praha E300 vini1.cesnet.cz vinix.cesnet.cz 1GE CESNET ONS15454 CESNET2 ONS15454 Plzeň viniy.cesnet.czvini2.cesnet.cz 1GE GOLE NetherLight Amsterdam GOLE StarLight Chicago CESNET2 n x 10Gb/s CESNET EF n x 10Gb/s OC-192c vini network in the USA CESNET2 PoP Praha CESNET2 ONS15454 viniz.cesnet.cz 1GE ONS GE over SONET simple extension possibilities Lightpath Chicago/Amsterdam/Praha for VINI Combination of 10G international lambdas, DWDM Open and ‚big‘ vendor transmission systems Brno

7 Apr th TRIDENTCOM, Washington DC 29 Attention to photonic transmission and processing should be increased Speed limits of electrical processing, above certain limit the only possibility Ability to make savings on energy consumption, housing space. Should be addressed by the physical layer network design: the possibility of addressing them on the higher layers is very limited Design of E2E lightpaths – multidomain task Necessity of interoperability at physical layer Theoretical understanding and experimental verification (in DF experimental facilities) necessary to verify feasibility of deployment in the production network Cooperation with FTTx vendors and operators Open photonic systems Can enable early adoption of the new photonic technology in the production Enable and support experimental applications requested by users or field experiments on dark fiber lines Can offer beneficial cost advantages Dark Fibre Facilities for Research and Experimentation Conclusions I

7 Apr th TRIDENTCOM, Washington DC 30 The development and interconnection (federalization) of experimental facilities (open testbeds) enabling field technology testing and user participation should be supported Give researchers an experimental environment for validating innovative – and potentially disruptive – architectures and technologies including physical layer. Many issues are only discovered when technology, devices or systems are deployed in "real-life" situations. Dark Fibre Facilities for Research and Experimentation Conclusions II

7 Apr th TRIDENTCOM, Washington DC 31 Interconnection of (dark fiber) experimental facilities, enabling researchers access to these Experimental verification and deployment of cutting edge solutions and technology Dark Fibre Facilities for Research and Experimentation Call for cooperation

7 Apr th TRIDENTCOM, Washington DC 32 Dark Fibre Facilities for Research and Experimentation Kyoto Price 4K transmission (via CESNET) “The Kyoto Prize” is an international award to honor those who have contributed significantly to the scientific, cultural, and spiritual betterment of mankind. 4K uncompressed live streaming (6 Gbit/s) Kyoto to Stockholm, L2-10GbE over 21,000km. 4K compressed multicast streaming (500 Mbit/s) Multicast by hardware packet replicator World’s First Trans-Pacific and Trans- Atlantic (21,000km) Real Time Switching and Streaming Transmission of Uncompressed 4K Motion Pictures, Nov 10-11, 2007 from Kyoto to Stockholm via Chicago and Prague K.html

7 Apr th TRIDENTCOM, Washington DC 33 Dark Fibre Facilities for Research and Experimentation References [1] Petr Holub, Josef Vojtech, Jan Radil, et. al., „Pure Optical (Photonic) Multicast“, GLIF 2007 Demo, Prague, [2] Jan Radil, Stanislav Šíma, „ Customized Approaches to Fibre-based E2E Services“, TERENA 1st E2E Workshop, Amsterdam, [3] Stanislav Šíma, et. al., „ LTTx: Lightpaths to the application, From GOLEs to dispersed end users “, GLIF 2008 Workshop, Seattle, [4] Josef Vojtěch, Jan Radil, „Transparent all optical switching devices in CESNET“, 25th APAN meeting, Honolulu, 2008.

7 Apr th TRIDENTCOM, Washington DC 34 Jan Gruntorád, Miloslav Hůla, Jiří Navrátil, Jan Nejman, Václav Novák Dark Fibre Facilities for Research and Experimentation Acknowedgement

7 Apr th TRIDENTCOM, Washington DC 35 Dark Fibre Facilities for Research and Experimentation Thank you for your attention. Questions?

7 Apr th TRIDENTCOM, Washington DC 36 Dark Fibre Facilities for Research and Experimentation Invitation to Future Internet Conference May 2009, CLARION CONGRESS HOTEL Prague Organized by the EC and CESNET during the Czech Republic presidency

7 Apr th TRIDENTCOM, Washington DC 37 Dark Fibre Facilities for Research and Experimentation Invitation to 5th Customer Empowered Fiber Network Workshop May 14 – 15, 2009 Praha, Organized by CESNET Universitas Carolina, founded Previous workshops

7 Apr th TRIDENTCOM, Washington DC 38 Dark Fibre Facilities for Research and Experimentation GLIF 2007 photonic multicast with CLM