The International Grid Testbed: a 10 Gigabit Ethernet success story in memoriam Bob Dobinson GNEW 2004, Geneva Catalin Meirosu on behalf of the IGT collaboration

International Testbed GNEW Contents www: who, why, what is IGT Native Ethernet over lightpaths: from 1 Gbps to 10 Gbps First demonstration of transatlantic native 10 GE over legacy OC-192 infrastructure Future plans Conclusions

International Testbed GNEW Who’s who in IGT Universities and Research Institutes –Canada Carleton University, Ottawa University of Alberta University of Victoria TRIUMF, Vancouver Université de Montreal McGill University, Montreal University of Toronto –The Netherlands University of Amsterdam –Switzerland CERN, Geneva Connectivity provided by CANARIE SURFnetORION Netera BCnet RISQ Sources of funding: IGT (Canarie Directed Research Grant), ESTA (IST ) Collaboration: CERN Openlab, the EU DataTAG project

International Testbed GNEW Current IGT topology

International Testbed GNEW Motivation IGT is a research network for next- generation e-science –Illustration of the application empowered network concept (see Bill St. Arnaud’s panel at SC2003) –Investigate emerging technologies in a demanding real life environment (see Wade Hong’s presentation at the CA*Net 4 Design Meeting May 26-27, 2003 ) –Real Time Farms for the ATLAS experiment at CERN (see Bryan Caron’s presentation at RISQ2003)

International Testbed GNEW The ATLAS Experiment at CERN Large collaboration of about 2000 scientists for an experiment operational from 2007 Main goals: the discovery of new particles and exploring physics beyond the Standard Model Challenging data collection and analysis systems –Online: 20 Gbps to the event filter –Offline: 4 TB/day to storage

International Testbed GNEW Ethernet: from local to global Ethernet, a brand name for LAN technology –Original Ethernet: shared media, half-duplex, distance limited by protocol –Modern Ethernet: point-to-point, full-duplex, switched, distance limited by the optical components Cost effective !

International Testbed GNEW Why native Ethernet long haul? More than 90% of the Internet traffic originates on an Ethernet LAN Data traffic on the LAN increases due to new applications Ethernet services with incremental bandwidth offer new business opportunities to carriers –See IEEE Communications Magazine, Vol. 42, No. 3, March 2004, on additional benefits for both the enterprise and the service providers Why not native Ethernet ? –Scalability, reliability, service guarantees … All of the above are active research areas Native Ethernet long haul connections can be used today as a complement to the routed networks, not as a replacement

International Testbed GNEW GE: a new Ethernet 10 GE – designed from the beginning for access to long haul networks –40 km maximum distance specified by the standard … –1550nm lasers: optical amplifiers can be used to increase distance over dark fibre State of the art: 250 km demonstrated in Denmark by the EU ESTA project [see Mikkel Olesen’s presentation at NORDunet2003] –what happens when you have to regenerate the signal ? no signal-agnostic regenerators deployed

International Testbed GNEW The 10 GE WAN PHY 10GE introduces a gateway from LAN to the WAN by means of the WAN PHY –Compatible with existing WAN infrastructure Transmission rate Encapsulation – Partial use of the management bits of the SONET/SDH frame Today’s WAN PHY modules use SONET- compliant optical components WAN PHY LTE 3R OC192 Router 10GE switch/router LTE Router OC192 WAN PHY 10GE switch/router LTE traditional novel

International Testbed GNEW Demo during ITU Telecom World '03 Cisco ONS Force10 E 600 Force10 E 600 HP Itanium-2 HP Itanium-2 Cisco ONS Cisco ONS Cisco ONS Cisco ONS Ixia 400T Intel Itanium-2 Intel Xeon Ixia 400T 10GE WAN PHY 10GE LAN PHY OC192c OttawaTorontoChicagoAmsterdam Geneva 10 GE WAN PHY over an OC-192c circuit using lightpaths provided by SURFnet and CANARIE 9.24 Gbps using traffic generators 5.65 Gbps using TCP on PCs 6 Gbps using UDP on PCs

International Testbed GNEW Results on the transatlantic 10 GE Single stream UDP throughputSingle stream TCP throughput Data rates are limited by the PC, even for our memory-to-memory tests UDP uses less resources than TCP on high bandwidth-delay product networks

International Testbed GNEW WAN PHY over DWDM Direct lambda access from the provider is required The DWDM transceiver card as “LTE” HP Itanium GE WAN DWDM AmsterdamGeneva Force10 E600 10GE LAN HP Itanium-2 10 GE WAN 10GE LAN Ixia 400T Ixia 400T Force10 E600 HP Itanium- 2 Intel Xeon

International Testbed GNEW What next ? More 10 Gigabit Ethernet experiments –Extend the reach of our WAN PHY connection Adds more latency into the testbed, hence increased pressure on protocols Might allow us to identify technical showstoppers –Disk-to-disk transfers –Comparative study of data transfer protocols for 10 Gbps networks

International Testbed GNEW What next ? Support for real-time access to remote computer farms during the ATLAS testbeam run in 2004 –1 Gbps connectivity is sufficient as a proof of concept –Sites in Copenhagen, Cracow and Edmonton will process in real time data acquired at the CERN testbeam This is part of the feasibility study for remote real time processing in ATLAS

International Testbed GNEW Conclusions IGT demonstrated native 10 Gigabit Ethernet over lightpaths 10GE WAN PHY is the technology that enables inter-continental native Ethernet IGT will support remote real time applications, with emphasis on data collection and analysis

International Testbed GNEW References Bill St. Arnaud – contribution to the “Strategies for Application- Empowered Networks” panel at SC2003 Bryan Caron – “The GARDEN Project: A Testbed for High Bandwidth Real-Time Applications”, canw2003/ppt/23PM_Bryan_Caron.pdfhttp:// canw2003/ppt/23PM_Bryan_Caron.pdf Wade Hong – “CA*net 4 International Grid Testbed ”, Mikkel Olesen – “The use of dark fibre in Forskningsnttet”, the 21 st NORUnet Conference,