CERN external networking update & DataTAG project Presented at the Internet2 Spring member meeting 6th May 2002, Arlington Olivier H. Martin CERN - IT Division

Main Internet connections at CERN Mission Oriented & World Health Org. Swiss National Research Network IN2P3 General purpose A&R and commodity Internet connections (Europe/USA/World) WHO 2.5Gbps SWITCH 155Mbps 45Mbps Europe GEANT (1.25/2.5Gbps) 1Gbps USLIC 622Mbps USA 1Gbps CERN CIXP 2.5Gbps Commercial DataTAG NetherLight Network Research

CERN’s Distributed Internet Exchange Point (CIXP) Telecom Operators & dark fibre providers: Cablecom, COLT, France Telecom, Global Crossing, GTS/EBONE, KPNQwest, LDCom(*), Deutsche Telekom/Multilink, MCI/Worldcom, SIG, Sunrise/diAx, Swisscom (Switzerland), Swisscom (France), SWITCH (**), Thermelec. Internet Service Providers include: 3GMobile (*), Infonet, AT&T Global Network Services (formerly IBM), Cablecom, Callahan, Carrier1, Colt, DFI, Deckpoint, Deutsche Telekom, Easynet, GTS/Ebone, Eunet/KPNQwest, France Telecom/OpenTransit, Global-One, InterNeXt, IS Internet Services (ISION), IS-Productions, Nexlink, PSI Networks (IProlink), MCI/Worldcom, Petrel, Renater, Sita/Equant(*), Sunrise, Swisscom IP-Plus, SWITCH, GEANT, VTX/Smartphone, UUnet. isp isp Telecom operators c i x p isp isp isp isp isp isp CERN firewall Telecom operators Cern Internal Network

The CIXP is a combination of Starlight & AADS/StarTAP Multiple Telecom Operators, Internet Service Providers & dark fiber providers Neutral location 1 Gigabit Ethernet based, 10GBE soon. Service extension to: Telehouse Geneva (2*GBE/dark fiber) Future prospects: Close 2.5 Gbps Amsterdam-Chicago-Geneva triangle in time for iGRID’2002 Metropolitan optical testbed for demonstration at Telecom’2003 (Geneva) isp isp Telecom operators c i x p isp isp isp isp isp isp ISPs CERN firewall Telecom operators Cern Internal Network

The DataTAG Project http://www.datatag.org

Funding agencies Cooperating Networks

EU partners

Associated US partners

The project CERN is the project coordinator. European partners: INFN (IT), PPARC (UK), University of Amsterdam (NL) and INRIA (FR) soon (June/July2002). ESA/ESRIN (IT) will provide Earth Observation demos together with NASA. Budget: 3.98 MEUR Start date: January, 1, 2002 Duration: 2 years (aligned on DataGrid) Funded manpower: ~ 15 persons/year 11/12/2018 The DataTAG Project

US Funding & collaborations US NSF support through the existing collaborative agreement with CERN (Eurolink award). US DoE support through the CERN-USA line consortium. Significant contributions to the DataTAG workplan have been made by Andy Adamson (University of Michigan), Jason Leigh (EVL@University of Illinois), Joel Mambretti (Northwestern University), Brian Tierney (LBNL). Strong collaborations already in place with ANL, Caltech, FNAL, SLAC, University of Michigan, as well as Internet2 and ESnet. 11/12/2018 The DataTAG Project

In a nutshell Two main focus: Grid related network research (WP2, WP3) Interoperability between European and US Grids (WP4) 2.5 Gbps transatlantic lambda between CERN (Geneva) and StarLight (Chicago) around July 2002 (WP1). Dedicated to research (no production traffic) Fairly unique multi-vendor testbed with layer2 and layer 3 capabilities In principle open to other EU Grid projects as well as ESA for demonstrations 11/12/2018 The DataTAG Project

Multi-vendor testbed with layer3 & layer2 capabilities INFN (Bologna) STARLIGHT (Chicago) Abilene CERN (Geneva) GEANT ESnet 1.25Gbps Juniper Juniper Research2.5Gbps Cisco 6509 M M Alcatel Alcatel Starlight GBE Production 622Mbps Cisco Cisco 11/12/2018 M= Layer 2 Mux The DataTAG Project

« Distributed layer2 exchange point » G Eth switch G Eth switch 2.5Gb (STM-16 2.5Gb (STM-16 1670SM 1670SM Operator Trans-Atlantic Link (KPNQwest) CERN Chicago DWDM “POP” DWDM “POP” Alcatel 7770 Alcatel 7770 = equivalent GE GE G Eth switch GE G Eth switch Starlight CERN « Distributed layer2 exchange point » 7770 7770

Goals End to end Gigabit Ethernet performance using innovative high performance transport protocols. Assess & experiment inter-domain QoS and bandwidth reservation techniques. Interoperability between some major GRID projects in Europe and North America DataGrid, possibly other EU funded Grid projects PPDG, GriPhyN, Teragrid, iVDGL (USA) 11/12/2018 The DataTAG Project

Testbed Sites (>40) HEP sites ESA sites Dubna Moscow RAL Lund Lisboa Santander Madrid Valencia Barcelona Paris Berlin Lyon Grenoble Marseille Brno Prague Torino Milano BO-CNAF PD-LNL Pisa Roma Catania ESRIN CERN HEP sites ESA sites IPSL Estec KNMI (>40) Francois.Etienne@in2p3.fr - Antonia.Ghiselli@cnaf.infn.it

DataTAG/WP4 framework and relationships Grid projects: DataGrid PPDG Griphyn LCG Globus Condor ….. input feedback ….. Grid Interoperability Activities: DataTAG/WP4 iVDGL HICB/HIJTB GGF Integration stardardization Proposals ….. Applications: LHC experiments CDF Babar ESA 11/12/2018 The DataTAG Project

Major 2.5 Gbps circuits between Europe & USA DataTAG project NewYork Abilene UK SuperJANET4 IT GARR-B STAR-LIGHT ESNET GEANT CERN MREN NL SURFnet STAR-TAP Major 2.5 Gbps circuits between Europe & USA

Project positioning Why yet another 2.5 Gbps transatlantic circuit? Most existing or planned 2.5 Gbps transatlantic circuits are for production, i.e. not suitable for advanced networking experiments requiring operational flexibility, e.g.: deploying new equipment (routers, G-MPLS capable multiplexers), activating new functionality (QoS, MPLS, distributed VLAN) The only known exception to date is the Surfnet circuit between Amsterdam & Chicago (Starlight) Concerns: How far beyond Starlight can DataTAG extend? 11/12/2018 The DataTAG Project

Major Grid networking issues (1) QoS (Quality of Service) still largely unresolved on a wide scale because of complexity of deployment End to end performance in the presence of firewalls There is a lack of high performance firewalls, can we rely on products becoming available or should a new architecture be evolved? Evolution of LAN infrastructure to 1Gbps then 10Gbps Uniform end to end performance 11/12/2018 The DataTAG Project

CERN’s new firewall architecture Regular flow Gbit Ethernet Backup Bypass CERNH2 (Cisco OSR 7603) Gbit Ethernet Fast Ethernet FastEthernet Dxmon FE and FDDI+bridge Cisco PIX Cisco RSP7000 FastEthernet 100/1000 Ethernet Fast Ethernet Cabletron SSR Security monitor Gbit Ethernet

Major Grid networking issues (2) TCP/IP performance over high bandwidth, long distance networks The loss of a single packet will affect a 10Gbps stream with 200ms RTT (round trip time) for 5 hours. During that time the average throughput will be 7.5 Gbps. On the 2.5Gbps DataTAG circuit with 100ms RTT, this translates to 38 minutes recovery time, during that time the average throughput will be 1.875Gbps. Line Error rates A 2.5 Gbps circuit can absorb 0.2 Million 1500 Bytes packets/second Bit error rates of 10E-9 means one packet loss every 250 milliseconds Bit error rates of 10E-11 means one packet loss every 25 seconds 11/12/2018 The DataTAG Project

Single stream vs Multiple streams effect of a single packet loss (e. g Single stream vs Multiple streams effect of a single packet loss (e.g. link error, buffer overflow) Streams/Throughput 10 5 1 7.5 4.375 2 9.375 10 Avg. 7.5 Gbps Throughput Gbps 7 5 Avg. 6.25 Gbps Avg. 4.375 Gbps 5 2.5 Avg. 3.75 Gbps T = 2.37 hours! (RTT=200msec, MSS=1500B) T T T Time T 11/12/2018 The DataTAG Project

Concluding remarks The dream of abundant bandwith has now become a hopefully lasting reality! Major transport protocol issues still need to be resolved. Large scale deployment of bandwidth greedy applications still remain to be done, Proof of concept has yet to be made. 11/12/2018 The DataTAG Project