1. Greedy Applications Ethernet Everywhere Real Time Networks for the Atlas Experiment Brian Martin CERN In memory of Bob Dobinson And on behalf of a large collaboration.

2. TNC 2004 Rhodes Greece 2 The Large Hadron Collider (LHC) LHC is being constructed underground inside a 27km tunnel. Head on collisions of very high energy protons. World wide collaboration. Experiments in 2007. Big science.

3. TNC 2004 Rhodes Greece 3 The ATLAS experiment at LHC Atlas Experiment 1700 People 33 countries 7000 ton detector 80 Terabytes/s data output

4. TNC 2004 Rhodes Greece 4 Event Visualisation

5. TNC 2004 Rhodes Greece 5 Trigger And Event Data Flow ROB L2PU EF Back End Network ~ 25 ROBS have information for one Level 2 CPU Subsequent events go to a free CPU Total traffic is ~20Kbytes @ 100Khz Data Collection Network EF ROB SFO ~500 Level 2 dual CPU’s ~3K Level 3 dual CPU’s ‘Good’ events are collected from the ROB’s by the SFI’s and sent to level 3 CPU’s Total Traffic ~2 Mbytes @ 3.5 KHz After processing the accepted event is fetched from Level 3 and sent to storage. Total traffic ~2 Mbytes @ 200 Hz to storage ROB SFI ~ 100Sub Farm Interfaces 1600 ROB buffers

6. TNC 2004 Rhodes Greece 6 Parameterised Switch Model Switch & Network Performance It’s an issue Measure and model switches input and output buffer depth max throughput module to backplane max throughput across backplane max throughput backplane to module intra-module throughput fixed part of intra-module latency floating part (depends on packet size) of intra-module latency fixed part of inter-module latency floating part (depends on packet size) of inter-module latency

7. TNC 2004 Rhodes Greece 7 Switch Testing: Data Source Programmable Ethernet Frame source Common clock, (+GPS option) Outgoing we control: Destination address Frame Size Time of dispatch Incoming we measure: Packet loss per source Frame Size Latency Test consists of Defining traffic profile for dispatch Histogramming latency & packet loss per source/destination pair

8. TNC 2004 Rhodes Greece 8 e.g. Buffer size measurement ABC D Send A->D 50% –Measure: Tad0 Send A->D 5% B,C->D 100% –Measure: Tad1 Send A,B,C->D 100% –Measure: Tad2 Size of the queues (Tpp=time per packet): –Output queue = (Tad1-Tad0)/Tpp –Input queue = (Tad2-Tad1)/Tpp Switch R

9. TNC 2004 Rhodes Greece 9 Modeling versus measurement Using the calibrated switch model Build a system model And check it against measurements

10. TNC 2004 Rhodes Greece 10 Application of results Can’t use TCP in the data collection network –Latency too large –Requires low loss raw Ethernet Gathering full event is a bottleneck 1600 ROB’s->1 CPU –Credit based solution to random sources. –Link between number of credits and buffer depth. –More credits better use of bandwidth –But increases requirement for buffer depth Concentrating edge switches –Concatenates underused Gb/s links –Fewer but faster ports at the core –More smaller switches at the edge –Random congestion between switches Modeling can give the answer And defines acceptable switches

11. TNC 2004 Rhodes Greece 11 Must all the CPU’s be on site? ROB L2PU SFI PF Local Event Processing Farms PF Remote Event Processing Farms Remote Event Processing Farms Dedicated light paths 10 GE Packet switched WAN Data Collection Network Back End Network ATLAS Detectors

12. TNC 2004 Rhodes Greece 12 Ethernet, how you’ve grown! Shared copper media Limited distance Limited speed Half duplex IEEE802.3ae Point to point Full duplex Switched architecture 100meters over copper 40 kms over fiber 10Gbit/second WAN Access Carrier Class product

13. TNC 2004 Rhodes Greece 13 LAN at 10Gbit/s Ethernet LAN at 10Gbit/s 40 Km is the standard, –based on commercially available lasers –and installed fiber If you have access to dark fiber Then you can amplify the light One amplifier every 40 – 100- kms DCF also needed Amplifiers also amplify the noise 3R regeneration every 400 – 600 kms

14. TNC 2004 Rhodes Greece 14 10 GE over dark fibre - ESTA 250 km of DARENET’s dark fibre in July 2003 4 optical amplifiers in each direction error-free transmission for 66 hours

15. TNC 2004 Rhodes Greece 15 525 km Long Haul LAN Further tests done Paper accepted for COIN2004 in Yokohama

16. TNC 2004 Rhodes Greece 16 WAN 10 Gig E to the WAN: Theory WAN PHY LTE 3R OC192Router OC192 10GE Switch WAN PHY 10GE Switch ELTE Router OC192

17. TNC 2004 Rhodes Greece 17 In practice however: Wan Phy designed to connect to Sonet through an ELTE –Same bitrate, Same frame structure –A defined ELTE specification (But nobody built one) Not guaranteed to operate with existing LTE (OC192 port) –Unused / fixed-value bits in the management overhead –Optical signal has relaxed jitter and clock specs, (cheaper optics) Nobody built the optics either, everyone uses Sonet optics Tee Shirts versus the Suits Where to plug the Wan-Phy fiber? Into a OC192 socket, perhaps?

18. TNC 2004 Rhodes Greece 18 Plug and Play at Canarie ONS 15454 E-600 IXIA 10GE 1310nm 1550nm 1310nm 1 2 Latency @ 91.3% of the 10 GE LAN PHY line speed

19. TNC 2004 Rhodes Greece 19 WAN PHY over SONET Trials between Geneva and Amsterdam Error-free transmission Field validation of our previous lab experiments ONS 15454 IXIA 10 GE IXIA 10 GE DWDM CERN Geneva SURFnet Amsterdam OC192

20. TNC 2004 Rhodes Greece 20 Raw Ethernet

21. TNC 2004 Rhodes Greece 21 TCP Using –2.4.21, web100 TCP patch, iperf 1.7.0, kernel

22. TNC 2004 Rhodes Greece 22 TCP: Geneva to Amsterdam CERN – UvA: Average of 5448 Mbps for almost 15 hours

23. TNC 2004 Rhodes Greece 23 WAN PHY over DWDM No packet loss for 91 hours and 365 TB of continuous streaming (equivalent to a BER better then of 0.3*10^-15) IXIA 10 GE IXIA 10 GE DWDM CERN Geneva SURFnet Amsterdam ONS 15454 IXIA 10 GE IXIA 10 GE DWDM OC192

24. TNC 2004 Rhodes Greece 24 10 GE WAN PHY demo ITU World Telecom’03 First native transatlantic 10 GE experiment In June 2003, 700 GBytes of data transferred to Ottawa in 6.5 hours 9.24 Gbps with traffic generators 6 Gbps with UDP 5.24 Gbps with TCP Article describing the experiments accepted at the IEEE HSMNC’04 conference, Toulouse, June 28 - July 2 Cisco ONS 15454 Force10 E 600 Force10 E 600 Cisco ONS 15454 Cisco ONS 15454 Cisco ONS 15454 Cisco ONS 15454 Intel Itanium-2 Intel Xeon Ixia 400T 10GE WAN PHY 10GE LAN PHY OC192c OttawaTorontoChicagoAmsterdam Geneva Oct. 6 th, 2003 HP Itanium-2 HP Itanium-2 Ixia 400T

25. TNC 2004 Rhodes Greece 25 Native Ethernet Cern to Ottawa Single stream UDP throughputSingle stream TCP throughput Data rates are limited by the PC, even for our memory-to-memory tests UDP uses less resources than TCP on high bandwidth-delay product networks

26. TNC 2004 Rhodes Greece 26 Thanks to: Bob Dobinson, Piotr Golonka, Andreas Hirstius, Mihai Ivanovici, Kris Korcyl, Olivier Martin, Catalin Meirosu, Stefan Stancu, Mikkel Olesen, Stan Cannon (CERN) Lars Dittmann, Martin Petersen (COM) Cees de Laat, Antony Antony, Freek Dijkstraa (University of Amsterdam) Wade Hong (University of Carleton) Bill St. Arnaud, Rene Hatem(CANARIE ) Ray Belleville (Cortex Networks, Ottawa) Erik Radius (SURFnet ) Pieter de Boer(SARA )

27. TNC 2004 Rhodes Greece 27..and to our many partners ESTA, IST-2001-33182

28. TNC 2004 Rhodes Greece 28 Conclusions 10 GE works over dark fiber up to <600Km using LAN PHY 10 GE works over legacy trans / intercontinental with WAN PHY Commercial switches already available with all needed features The bandwidth is there to run off-site ATLAS farms in real time Further work to do on managing traffic flow Further work to do on understanding the cost model ANY QUESTIONS